Vision-based experiment wiring auxiliary method and system

Abstract

The invention discloses an experiment wiring auxiliary method and system based on vision, and relates to the field of cross application of computer vision and artificial intelligence, and the method comprises the steps: S1, obtaining a schematic diagram picture and an experiment device picture; s2, identifying the schematic diagram picture by adopting a deep learning-based target detection algorithm, a Canny edge detection algorithm, a line detection algorithm and a graph theory analysis method to obtain related information corresponding to component symbols; s3, dividing the experiment device picture by adopting an image segmentation technology, and identifying divided regions by adopting a deep learning classifier to obtain related information corresponding to each region; s4, converting the schematic diagram picture into a wiring view according to the S2, and generating a corresponding wiring path planning table; s5, according to the steps S3 and S4, the starting point and the ending point of the wire and the connection path of the wire are mapped into an experiment device picture through an image registration technology; the experiment wiring accuracy and efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the cross - application field of computer vision and artificial intelligence, and particularly to a vision - based experimental wiring assistance method and system. Background Technique

[0002] In the current electrical experiment teaching in colleges and universities, experimental wiring is an important link in cultivating students' engineering practice ability. For many students who enter the laboratory for the first time, completing experimental wiring on the experimental device according to the circuit schematic diagram has a certain learning threshold. Without relevant training, it is difficult for students to accurately correspond the symbols in the circuit schematic diagram with the modules on the experimental device, resulting in a large amount of time and energy being consumed in reading and understanding the circuit schematic diagram and familiarizing with the experimental device environment, thus reducing the experimental efficiency. In addition, the problem of non - standard wiring is relatively common during the experiment. According to the requirements, each loop in the circuit schematic diagram should be connected with wires of the same color for subsequent inspection and troubleshooting, but many students often ignore this point, resulting in chaotic wiring and increasing the difficulty of troubleshooting. Especially as the number of wires increases, it is difficult for students to check each circuit loop one by one, consuming a large amount of time and being prone to situations such as missed inspection and mis - inspection. Once the error is not discovered in time, serious consequences may occur after power - on, ranging from circuit short - circuit and damage to experimental equipment to personal injury accidents.

[0003] Therefore, there is an urgent need for an intelligent auxiliary tool that can help students quickly get started with experiments and provide efficient guidance for experimental wiring to ensure the safety and reliability of experiments.

[0004] In related technologies, most of the auxiliary devices for power system wiring and the wiring auxiliary devices applied to distribution cabinets are physical structures. Among them, the distribution cabinet wiring auxiliary device assists in connecting the wires of the distribution cabinet through a lifting plate and a positioning component; the power system wiring auxiliary device adopts a sliding mechanism, and ensures the stable position of the branch line and the main line through a fastening part and a shaping part, improving the docking accuracy between wires. However, for the electrical laboratory scenario, the above - mentioned auxiliary devices need on - site installation and transformation, with high usage costs and low flexibility. In addition, most commercial software (such as EPLAN) supports the conversion from a schematic diagram to a 3D assembly diagram, and has functions such as automatic wiring and device number export, thus improving the wiring efficiency. However, the learning threshold of such software is relatively high, it is highly professional, and it takes a certain amount of time to master. In contrast, schematic design software such as AutoCAD Electrical is easier to get started, has a simple interface, is convenient to maintain, and supports secondary expansion. But it lacks the function of schematic diagram conversion and cannot be combined with actual devices to provide a more intuitive and convenient wiring scheme. Summary of the Invention

[0005] In view of this, the present invention provides a vision-based experimental wiring assistance method and system to solve the technical problems existing in the related art.

[0006] In a first aspect, the present invention provides a vision-based experimental wiring assistance method, including:

[0007] S1. In response to the schematic diagram picture and the experimental device picture uploaded by the client, obtain the schematic diagram picture and the experimental device picture;

[0008] S2. Use a deep learning-based object detection algorithm, Canny edge detection algorithm, line detection algorithm, and graph theory analysis method to identify the schematic diagram picture, and obtain the category, text symbol, parameters, and related note information corresponding to each component symbol in the schematic diagram picture, the quantity corresponding to each type of component symbol, and the connection relationship between each component symbol;

[0009] S3. Use image segmentation technology to divide the experimental device picture into multiple regions according to modules, and the multiple regions respectively correspond to different modules in the experimental device image; and use a pre-trained deep learning classifier to identify each region to obtain the identification information corresponding to each region; the identification information includes: module name, wiring hole position name, wiring hole position label, wiring hole position color, and wiring hole position coordinates;

[0010] S4. According to the category, text symbol, parameters, and related note information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationship between each component symbol, convert the schematic diagram picture into a wiring view, and generate a corresponding wiring path planning table; the wiring path planning table includes wire number, start and end points of the wire, recommended wire color, and wire diameter;

[0011] S5. According to the wiring path planning table and the wiring hole position coordinates corresponding to each module, map the start and end points of the wire and the connection path of the wire in the experimental device picture through image registration technology.

[0012] In an optional implementation manner, after S1, the method further includes:

[0013] Preprocess the schematic diagram picture and the experimental device picture respectively; the preprocessing includes: image denoising, contrast and brightness adjustment, geometric transformation, and perspective correction.

[0014] In an optional implementation manner, S2 includes:

[0015] S21. Use a deep learning-based target detection model to perform component symbol recognition on the schematic image, and obtain the category and bounding box coordinates corresponding to each component symbol in the schematic image, as well as the number of component symbols of each category;

[0016] S22, using an OCR algorithm to perform text symbol recognition on the schematic diagram image to obtain the text symbol, parameters, and related remarks corresponding to each component symbol;

[0017] S23, based on the bounding box coordinates corresponding to each component symbol, use the Canny edge detection algorithm and the line segment detection algorithm to perform line segment recognition on the schematic image to obtain the connecting wires between each component symbol;

[0018] S24. Build a topology diagram corresponding to the schematic diagram using component symbols as nodes and connecting wires as edges; the edges carry connection information; and the connection information includes the coordinates of the two endpoints corresponding to the connecting wires and the connection confidence between the two endpoints.

[0019] S25. Using graph theory methods to extract the connection relationship between connecting wires and component symbols in the topology diagram;

[0020] S26. Summarize the categories, text symbols, parameters and related remarks corresponding to each component symbol, and the quantity corresponding to each category of component symbols to generate a schematic component analysis table; summarize the connection relationships between each component symbol to generate a schematic component connection relationship table; the schematic component connection relationship table includes: the number of loops, the number of wires corresponding to each loop, and the starting position and ending position of the wires.

[0021] In an optional embodiment, the S23 includes:

[0022] S231, using the Canny edge detection algorithm and the line segment detection algorithm to identify the line segments of the schematic diagram image, and obtain the connecting line segments corresponding to the schematic diagram image; the connecting line segments include the internal line segments of the component symbols and the connecting wires between the component symbols;

[0023] S232, calculating the overlapping area between the target connecting line segment and the bounding box corresponding to each component symbol according to the bounding box coordinates corresponding to each component symbol;

[0024] S233. When the target connection line segment is completely within the bounding box of the target component symbol, the target connection line segment is an internal line segment of the target component symbol; when the target connection line segment spans the bounding boxes of multiple component symbols or has no overlapping area with all component symbols, the target connection line segment is a connecting wire.

[0025] In an alternative embodiment, after the experimental device picture is divided into multiple regions according to modules by using image segmentation technology, and the multiple regions respectively correspond to different modules in the experimental device image, S3 further includes:

[0026] In response to a fine-tuning operation performed by the client on the divided regions in the experimental device picture, the adjusted multiple regions are recognized by using a pre-trained deep learning classifier to obtain the recognition information corresponding to each adjusted region; the fine-tuning operation is an operation for adjusting the boundary line of the target region or the position of the wiring holes within the target region.

[0027] In an alternative embodiment, S4 includes:

[0028] S41. Generate a wiring view corresponding to the schematic diagram picture according to the category, text symbol, parameters, and related note information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationship between each component symbol; each component symbol in the wiring view carries the corresponding text symbol, parameters, and related note information;

[0029] S42. Match each component in the wiring view with the corresponding module according to a plurality of preset package libraries; and match the correct wiring holes for each component symbol according to the text symbol, parameters, and related note information carried by each component symbol;

[0030] S43. Summarize the wiring holes corresponding to each component symbol to generate a corresponding wiring path planning table.

[0031] In an alternative embodiment, after S5, the method further includes:

[0032] S6. Extract the starting position and ending position of the wire from the schematic diagram component connection relationship table to generate a data structure; extract the module name, wiring hole name, wiring hole identifier, wiring hole color, and wiring hole coordinates from the experimental device module analysis table to establish an index table;

[0033] S7. Search for the component category corresponding to the starting position and ending position of each wire in the data structure; and search the index table according to the component category to obtain the wiring hole identifier and wiring hole coordinates corresponding to the component category, and generate a matching table;

[0034] S8. Store the coordinates and positions to be wired corresponding to each wire in the experimental device picture by using a graph data structure, and compare the coordinates and wiring positions to be wired corresponding to each wire with the matching table to verify the connectivity and correctness of the wire.

[0035] In a second aspect, the present invention provides a vision-based experimental wiring assistance system, comprising:

[0036] A data acquisition module, configured to obtain the schematic diagram picture and the experimental device picture in response to the schematic diagram picture and the experimental device picture uploaded by the client;

[0037] A schematic diagram parsing module, configured to use a deep learning-based object detection algorithm, a Canny edge detection algorithm, a line detection algorithm, and a graph theory analysis method to identify the schematic diagram picture, and obtain the category, text symbol, parameters, and related note information corresponding to each component symbol in the schematic diagram picture, the quantity corresponding to each type of component symbol, and the connection relationship between each component symbol;

[0038] An experimental device diagram parsing module, configured to use image segmentation technology to divide the experimental device picture into multiple regions according to modules, and the multiple regions respectively correspond to different modules in the experimental device image; and use a pre-trained deep learning classifier to identify each region to obtain the identification information corresponding to each region; the identification information includes: module name, wiring hole position name, wiring hole position mark, wiring hole position color, and wiring hole position coordinates;

[0039] A path planning and automatic wiring module, configured to convert the schematic diagram picture into a wiring view according to the category, text symbol, parameters, and related note information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationship between each component symbol, and generate a corresponding wiring path planning table; the wiring path planning table includes wire number, starting and ending points of the wire, recommended wire color, and wire diameter;

[0040] An image synthesis and annotation module, configured to map the starting and ending points of the wire and the connection path of the wire onto the experimental device picture through image registration technology according to the wiring path planning table and the wiring hole position coordinates corresponding to each module.

[0041] The present invention has the following beneficial effects:

[0042] The present invention has a schematic diagram conversion function, which can intuitively map an abstract schematic diagram onto a real experimental device image and present it in a visual interaction manner on the client side, helping students understand the circuit structure more intuitively and reducing the learning threshold; the present invention can facilitate error checking. By comparing the standard wiring with the physical object, students can quickly locate wiring errors and shorten the experimental fault checking time; the present invention can help students quickly complete wiring by combining the automatic wiring result with the physical object picture, greatly reducing the error checking time, improving the experimental wiring efficiency and accuracy; the present invention has compatibility and diversity, not only with low cost but also high flexibility, and can be adapted to different models of experimental teaching devices, laying a foundation for subsequent development. Brief Description of the Drawings

[0043] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 is a schematic flowchart of a vision-based experimental wiring assistance method according to an embodiment of the present invention;

[0045] Figure 2 is an example diagram of a schematic diagram picture according to an embodiment of the present invention;

[0046] Figure 3 is an example diagram of an experimental device picture according to an embodiment of the present invention;

[0047] Figure 4 is a wiring view corresponding to the schematic diagram picture according to an embodiment of the present invention;

[0048] Figure 5 is a synthetic wiring path diagram according to an embodiment of the present invention;

[0049] Figure 6 is a schematic structural diagram of a vision-based experimental wiring assistance system according to an embodiment of the present invention. Detailed Description of the Embodiments

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0051] Figure 1 Illustrates a vision-based experimental wiring assistance method according to an embodiment of the present invention, as Figure 1 shown, the process includes the following steps:

[0052] S1. In response to the schematic diagram picture and the experimental device picture uploaded by the client, obtain the schematic diagram picture and the experimental device picture.

[0053] In step S1, the user can take pictures of the schematic diagram and the experimental device diagram through a mobile terminal such as a mobile phone or a tablet. The server recognizes the schematic diagram picture and the experimental device picture uploaded by the client, and obtains and analyzes and processes the schematic diagram picture and the experimental device picture.

[0054] For example, students can download and open the corresponding app, and click the "Take Picture" button on the app display interface to take pictures of the schematic diagram and the experimental device diagram. As Figure 2 and Figure 3 shown, after taking the picture, click the "Upload" button to upload the corresponding schematic diagram picture and experimental device picture to the server. The server responds to the schematic diagram picture and experimental device picture uploaded by the student, and obtains and starts to analyze and process the schematic diagram picture and the experimental device picture.

[0055] It should be noted that the experimental device picture is a panoramic picture of the hanging parts required during the wiring process of the front of the experimental device.

[0056] In an alternative embodiment, after S1, the method further includes:

[0057] Preprocessing the schematic diagram picture and the experimental device picture respectively; the preprocessing includes: image denoising, contrast and brightness adjustment, geometric transformation and perspective correction.

[0058] Specifically, first perform denoising processing on the schematic diagram picture and the experimental device picture respectively to reduce noise and facilitate subsequent image recognition and line extraction; then enhance image details through contrast and brightness adjustment, reduce uneven illumination, and improve the overall clarity of the image to ensure that the subsequent recognition algorithm can accurately detect the component symbols in the image. Due to possible angle deviation during the shooting process, use the geometric transformation algorithm to perform perspective correction on the image to restore the true proportion of the image and ensure the integrity of the image.

[0059] S2. Use a deep learning-based object detection algorithm, Canny edge detection algorithm, line detection algorithm, and graph theory analysis method to identify the schematic diagram picture, and obtain the category, text symbol, parameter, and relevant note information corresponding to each component symbol in the schematic diagram picture, the quantity corresponding to each type of component symbol, and the connection relationship between each component symbol.

[0060] In an alternative embodiment, step S2 includes:

[0061] S21. Use a deep learning-based object detection model to identify component symbols in the schematic diagram picture, and obtain the category and bounding box coordinates corresponding to each component symbol in the schematic diagram picture, and the quantity corresponding to each type of component symbol.

[0062] For example, asFigure 2 As shown, taking a voltmeter as an example, the voltmeter symbol corresponds to the symbol of "a circle with V1 or V2 written inside" in the figure. The category is voltmeter, the quantity is 2, the bounding box is the square border that wraps the voltmeter symbol, and the bounding box coordinates are the coordinates of the 4 corners corresponding to the square border. The symbols of other components are the same and will not be elaborated here.

[0063] S22. Use the OCR algorithm to recognize the text symbols in the schematic diagram picture, and obtain the text symbols, parameters, and relevant note information corresponding to each component symbol.

[0064] For example, Figure 2 As shown, still taking the voltmeter as an example, the text symbols corresponding to the voltmeter symbol are V1 and V2, the parameter is "Range: 0 - 500V", and the relevant note information is "Measure the output voltage of each phase respectively."

[0065] S23. According to the bounding box coordinates corresponding to each component symbol, use the Canny edge detection algorithm and the line segment detection algorithm to perform line segment recognition on the schematic diagram picture, and obtain the connecting wires between each component symbol.

[0066] In an alternative embodiment, step S23 includes:

[0067] S231. Use the Canny edge detection algorithm and the line segment detection algorithm to perform line segment recognition on the schematic diagram picture, and obtain the connecting line segments corresponding to the schematic diagram picture; the connecting line segments include the internal line segments of the component symbols and the connecting wires between the component symbols.

[0068] Specifically, on the basis of symbol recognition, the connecting line segments in the schematic diagram picture are separated by the Canny edge detection algorithm and the line segment detection algorithm. The obtained connecting line segments are divided into two categories, namely the internal line segments of the component symbols (such as the broken line of the resistor symbol) and the connecting wires between the component symbols (the connection lines between the component symbols).

[0069] S232. According to the bounding box coordinates corresponding to each component symbol, calculate the overlapping area between the target connecting line segment and the bounding box corresponding to each component symbol.

[0070] S233. When the target connecting line segment is completely within the bounding box of the target component symbol, the target connecting line segment is the internal line segment of the target component symbol; when the target connecting line segment crosses the bounding boxes of multiple component symbols or has no overlapping area with all component symbols, the target connecting line segment is the connecting wire.

[0071] S24. Use the component symbols as nodes and the connecting wires as edges to construct a topological structure diagram corresponding to the schematic diagram picture; among them, the edges carry connection information; the connection information includes: the coordinates of the two endpoints corresponding to the connecting wire and the connection confidence between the two endpoints.

[0072] Specifically, construct the topological structure diagram of the schematic diagram, use the component symbols as the nodes in the diagram, the wire connection as the edges in the diagram, connect the nodes corresponding to the two component symbols, and the information of the edges can be recorded in the endpoint coordinates of the wire and the detected connection confidence.

[0073] S25. Use graph theory methods to extract the connection relationships between the connecting wires and the component symbols in the topological structure diagram.

[0074] S26. Summarize the category, text symbol, parameters, relevant remarks information corresponding to each component symbol, and the quantity corresponding to each type of component symbol to generate a schematic component analysis table, as shown in Table 1 (the component name in Table 1 is the category corresponding to the component symbol); summarize the connection relationships between each component symbol to generate a schematic component connection relationship table; the schematic component connection relationship table includes: the number of loops, the number of wires corresponding to each loop, the starting position and ending position of the wires, as Figure 2 shown in the table.

[0075] Table 1

[0076]

[0077] Table 2

[0078]

[0079] It should be noted that the schematic component analysis table and the schematic component connection relationship table can be integrated into the standard.Netlist data format, which is convenient for subsequent importing the information into the Altium Designer software for automatic wiring and path design.

[0080] S3. Use image segmentation technology to divide the experimental device picture into multiple regions according to modules, and the multiple regions respectively correspond to different modules in the experimental device image; and use a pre-trained deep learning classifier to identify each region to obtain the identification information corresponding to each region; the identification information includes: module name, wiring hole position name, wiring hole position label, wiring hole position color, and wiring hole position coordinates.

[0081] Specifically, the image segmentation technology is used to divide the experimental device picture into multiple regions, which respectively correspond to different modules such as a voltmeter and an ammeter. A set of standard experimental device module information libraries (including the geometric features, text features of each module, and the position coordinates of the wiring ports) are established in the background. The deep learning classifier is used to identify the segmented regions, and each region is matched with the corresponding module in the standard module library to identify the module corresponding to each region, and the names, labels, colors, and coordinates of the wiring holes under each module are automatically marked.

[0082] In an alternative embodiment, the recognition information corresponding to each region is summarized to generate an experimental device module analysis table, as shown in Table 3:

[0083] Table 3

[0084]

[0085] It should be noted that the experimental device module analysis table can be exported in the XML file format for use by subsequent image synthesis, path planning, and automatic wiring modules.

[0086] In an alternative embodiment, after the experimental device picture is divided into multiple regions according to the modules by using the image segmentation technology, and the multiple regions respectively correspond to different modules in the experimental device image, step S3 further includes:

[0087] In response to the fine-tuning operation performed by the client on the regions divided in the experimental device picture, the adjusted multiple regions are recognized by using the pre-trained deep learning classifier to obtain the recognition information corresponding to each adjusted region; the fine-tuning operation is an operation for adjusting the boundary line of the target region or the position of the wiring holes within the target region.

[0088] Specifically, to ensure high-precision matching, the system provides an interactive calibration function that allows the user to fine-tune the automatically recognized module positions on the client. The user can adjust the boundaries of the modules and the positions of the wiring ports by using gestures such as dragging and zooming to ensure the accuracy of subsequent wiring markings.

[0089] S4. According to the category, text symbol, parameters, and related remarks information corresponding to each component symbol, the quantity of each type of electrical component symbol, and the connection relationship between each component symbol, convert the schematic diagram picture into a wiring view and generate a corresponding wiring path planning table; the wiring path planning table includes the wire number, the start and end points of the wire, the recommended color of the wire, and the wire diameter, as shown in Table 4.

[0090] Table 4

[0091]

[0092] In an alternative embodiment, step S4 includes:

[0093] S41. Generate a wiring view corresponding to the schematic diagram picture according to the category, text symbol, parameters, and related note information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationship between each component symbol; each component symbol in the wiring view carries the corresponding text symbol, parameters, and related note information, as Figure 4 shown.

[0094] S42. Match each component in the wiring view with the corresponding module according to a plurality of preset package libraries; and match the correct wiring hole positions for each component symbol according to the text symbol, parameters, and related note information carried by each component symbol.

[0095] It should be noted that before the method is executed, a package library of a simplified diagram of the relevant experimental device needs to be established in the Altium Designer software, the corresponding wiring positions are set, and relevant attributes including name, wiring hole position coordinates, etc. are added to each module to form a set of standard information libraries for subsequent classification recognition and matching of the experimental device pictures.

[0096] Exemplarily, first, according to the.Netlist table corresponding to the schematic component analysis table and the schematic component connection relationship table, use the Altium Designer software to automatically match each component with the pre-established package library, and convert the generated schematic diagram picture into a wiring view. In the layout of automatic wiring, wire connection rules including wire width, spacing, etc. are preset in advance. Use the automatic wiring function of Altium Designer to automatically wire each component according to the preset rules to generate a wiring scheme that meets the design requirements.

[0097] S43. Summarize the wiring hole positions corresponding to each component symbol to generate a corresponding wiring path planning table.

[0098] S5. According to the wiring path planning table and the wiring hole position coordinates corresponding to each module, map the starting point and ending point of the wire and the wire connection path in the experimental device picture by image registration technology.

[0099] Specifically, by combining the connection path planning table and the wiring hole position coordinates corresponding to each module, using image registration technology, the wire end points and routing information (i.e., the connection path of the wire) in the wiring view are mapped into the two-dimensional coordinate system of the experimental device picture, and the automatic wiring result is superimposed on the experimental device image. After the fusion is completed, different colors and line types will be used to label the connection path to distinguish different circuit or loop information, which is convenient for identification and inspection. The user can view the synthesized connection path diagram on the client side to obtain the detailed connection information of each wire, such as Figure 5 As shown, the dotted line indicates that the wire is selected, and the left information column represents the detailed connection information of the wire.

[0100] In some alternative embodiments, the user can be allowed to perform zooming, rotation, and clicking operations on the synthesized image on the client side to view the detailed connection information of each wire (such as the starting point, ending point, corresponding electrical components, and loop identification, etc.).

[0101] In an alternative embodiment, after S5, the method further includes:

[0102] S6. Extract the starting position and ending position of the wire from the schematic component connection relationship table to generate a data structure; extract the module name, wiring hole position name, wiring hole position identifier, wiring hole position color, and wiring hole position coordinates from the experimental device module analysis table to establish an index table.

[0103] S7. Find the component category corresponding to the starting position and ending position of each wire in the data structure; and according to the component category, search the index table to obtain the wiring hole position identifier and wiring hole position coordinates corresponding to the component category, and generate a matching table.

[0104] S8. Use a graph data structure to store the coordinates and positions to be wired corresponding to each wire in the experimental device picture, and compare the coordinates and wiring positions to be wired corresponding to each wire with the matching table to verify the connectivity and correctness of the wire.

[0105] The embodiments of the present invention can review the automatic wiring result. By comparing the generated wiring diagram with the logical connection relationship of the schematic diagram, the correctness of all physical connections is ensured. As long as the coordinates and positions to be wired corresponding to each wire in the experimental device picture have corresponding paths in the matching table, it means that the wire is correctly connected. This method not only realizes the effective fusion of data, but also provides an intuitive and interactive means of detection and review for users, thereby greatly improving the accuracy of experimental wiring and teaching efficiency.

[0106] In summary, the present invention has a schematic diagram conversion function, which can intuitively map the abstract schematic diagram onto the image of the real experimental device and present it in a visual interaction manner on the client side, helping students understand the circuit structure more intuitively and reducing the learning threshold; the present invention can facilitate error troubleshooting. By comparing the standard wiring with the physical object, students can quickly locate the wiring error and shorten the experimental fault troubleshooting time; the present invention can help students quickly complete the wiring by combining the automatic wiring result with the physical object picture, greatly reducing the error troubleshooting time and improving the experimental wiring efficiency and accuracy; the present invention has compatibility and diversity, not only with low cost but also high flexibility, and can be adapted to different models of experimental teaching devices, laying a foundation for subsequent development.

[0107] In an embodiment of the present invention, a vision-based experimental wiring assistance system is further provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, implementations in hardware, or combinations of software and hardware are also possible and contemplated.

[0108] An embodiment of the present invention provides a vision-based experimental wiring assistance system, as Figure 6 shown, including:

[0109] A data acquisition module 61, configured to obtain a schematic diagram picture and an experimental device picture in response to the schematic diagram picture and the experimental device picture uploaded by the client.

[0110] A schematic diagram parsing module 62, configured to use a deep learning-based object detection algorithm, a Canny edge detection algorithm, a line detection algorithm, and a graph theory analysis method to identify the schematic diagram picture, and obtain the category, text symbol, parameters, and relevant note information corresponding to each component symbol in the schematic diagram picture, the quantity corresponding to each type of component symbol, and the connection relationship between each component symbol.

[0111] An experimental device diagram parsing module 63, configured to use image segmentation technology to divide the experimental device picture into multiple regions according to modules, and the multiple regions respectively correspond to different modules in the experimental device image; and use a pre-trained deep learning classifier to identify each region to obtain the identification information corresponding to each region; the identification information includes: module name, wiring hole position name, wiring hole position mark, wiring hole position color, and wiring hole position coordinates.

[0112] The path planning and automatic wiring module 64 is used to convert the schematic diagram picture into a wiring view according to the category, text symbol, parameters, and related remarks information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationship between each component symbol, and generate a corresponding connection path planning table; the connection path planning table includes wire numbers, the starting and ending points of the wires, the recommended colors of the wires, and the wire diameters.

[0113] The image synthesis and annotation module 65 is used to map the starting and ending points of the wires and the connection paths of the wires in the experimental device picture through image registration technology according to the connection path planning table and the wiring hole position coordinates corresponding to each module.

[0114] In an optional implementation manner, the vision-based experimental wiring assistance system further includes:

[0115] The data preprocessing module 66 is used to preprocess the schematic diagram picture and the experimental device picture respectively; the preprocessing includes: image denoising, contrast and brightness adjustment, geometric transformation, and perspective correction.

[0116] In an optional implementation manner, the schematic diagram parsing module 62 includes:

[0117] The component symbol recognition unit 621 is used to recognize the component symbols in the schematic diagram picture by using an object detection model based on deep learning, and obtain the category and bounding box coordinates corresponding to each component symbol in the schematic diagram picture, as well as the quantity corresponding to each type of component symbol.

[0118] The text symbol recognition unit 622 is used to recognize the text symbols in the schematic diagram picture by using the OCR algorithm, and obtain the text symbols, parameters, and related remarks information corresponding to each component symbol.

[0119] The line segment recognition unit 623 is used to recognize the line segments in the schematic diagram picture by using the Canny edge detection algorithm and the line segment detection algorithm according to the bounding box coordinates corresponding to each component symbol, and obtain the connecting wires between each component symbol.

[0120] The topology structure construction unit 624 is used to construct a topology structure diagram corresponding to the schematic diagram picture with the component symbols as nodes and the connecting wires as edges; the edges carry connection information; the connection information includes: the coordinates of the two endpoints corresponding to the connecting wire and the connection confidence between the two endpoints.

[0121] The connection relationship extraction unit 625 is used to extract the connection relationship between the connecting wires and the component symbols in the topology structure diagram by using graph theory methods.

[0122] A data summarization unit 626 is configured to summarize data including the category, text symbol, parameters, relevant remarks information corresponding to each component symbol, and the quantity corresponding to each type of component symbol to generate a schematic component analysis table; summarize the connection relationships between each component symbol to generate a schematic component connection relationship table; the schematic component connection relationship table includes: the number of loops, the number of wires corresponding to each loop, and the starting and ending positions of the wires.

[0123] In an alternative embodiment, the line segment recognition unit 623 includes:

[0124] A connection line segment recognition sub-unit 6231 is configured to identify line segments in the schematic diagram picture by using the Canny edge detection algorithm and the line segment detection algorithm, and obtain the connection line segments corresponding to the schematic diagram picture; the connection line segments include the internal line segments of the component symbols and the connection wires between the component symbols.

[0125] An overlapping area calculation sub-unit 6232 is configured to calculate the overlapping area between the target connection line segment and the bounding box corresponding to each component symbol according to the bounding box coordinates corresponding to each component symbol.

[0126] A connection wire determination sub-unit 6233 is configured that when the target connection line segment is completely within the bounding box of the target component symbol, the target connection line segment is the internal line segment of the target component symbol; when the target connection line segment crosses the bounding boxes of multiple component symbols or has no overlapping area with all component symbols, the target connection line segment is a connection wire.

[0127] In an alternative embodiment, the experimental device diagram parsing module 63 further includes:

[0128] An interaction unit 631 is configured to respond to the fine-tuning operation performed by the client on the divided areas in the experimental device picture, and identify the adjusted areas by using a pre-trained deep learning classifier to obtain the recognition information corresponding to each adjusted area; the fine-tuning operation is an operation for adjusting the boundary line of the target area or the position of the wiring holes within the target area.

[0129] In an alternative embodiment, the path planning and automatic wiring module 64 includes:

[0130] A wiring view generation unit 641 is configured to generate a wiring view corresponding to the schematic diagram picture according to the category, text symbol, parameters, and relevant remarks information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationships between each component symbol; each component symbol in the wiring view carries the corresponding text symbol, parameters, and relevant remarks information.

[0131] A matching unit 642, configured to match a corresponding module for each component in the wiring view according to a plurality of preset packaging libraries; and match a correct wiring hole position for each component symbol according to the text symbol, parameters, and relevant note information carried by each component symbol.

[0132] A wiring path planning table generation unit 643, configured to summarize the wiring hole positions corresponding to each component symbol and generate a corresponding wiring path planning table.

[0133] In an alternative embodiment, the vision-based experimental wiring assistance system further includes:

[0134] A wiring path verification module 67, configured to extract the start position and end position of a wire from a schematic component connection relationship table to generate a data structure; extract the module name, wiring hole position name, wiring hole position identifier, wiring hole position color, and wiring hole position coordinates from an experimental device module analysis table to establish an index table; and configured to find the component category corresponding to the start position and end position of each wire in the data structure; and find the wiring hole position identifier and wiring hole position coordinates corresponding to the component category according to the component category to generate a matching table; and configured to store the coordinates and positions to be wired corresponding to each wire in the experimental device picture using a graph data structure, and compare the coordinates and wiring positions to be wired corresponding to each wire with the matching table to verify the connectivity and correctness of the wire.

[0135] The further function descriptions of the above-mentioned modules and units are the same as those in the corresponding embodiments above, and will not be repeated here.

[0136] The vision-based experimental wiring assistance system provided by the embodiments of the present invention can be applied to electrical experimental teaching in universities. Through an APP for assisting students in experimental wiring, combined with mobile phone camera recognition, image processing, and automatic wiring technologies, it can help students improve the efficiency of experimental wiring and facilitate students to check wiring errors. Specifically, the system can combine the planar schematic diagram with the modular components of the actual wiring platform, and intuitively superimpose the wiring result on the experimental device picture to display the electrical wiring result in an interactive manner, which can not only significantly improve the learning efficiency of students, but also improve the accuracy of experimental operations, providing a more intuitive and convenient auxiliary tool for electrical experimental teaching.

[0137] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A vision-based experimental wiring assistance method, characterized in that Including: S1. In response to the schematic diagram picture and the experimental device picture uploaded by the client, obtain the schematic diagram picture and the experimental device picture; S2. Use a deep learning-based object detection algorithm, Canny edge detection algorithm, line detection algorithm, and graph theory analysis method to identify the schematic diagram picture, and obtain the category, text symbol, parameters, and relevant note information corresponding to each component symbol in the schematic diagram picture, the quantity corresponding to each type of component symbol, and the connection relationship between each component symbol; S3. Use image segmentation technology to divide the experimental device picture into multiple regions according to modules, and the multiple regions respectively correspond to different modules in the experimental device image; And use a pre-trained deep learning classifier to identify each region, and obtain the identification information corresponding to each region; the identification information includes: module name, wiring hole position name, wiring hole position label, wiring hole position color, and wiring hole position coordinates; S4. According to the category, text symbol, parameters, and relevant note information corresponding to each component symbol, the quantity corresponding to each type of electrical component symbol, and the connection relationship between each component symbol, convert the schematic diagram picture into a wiring view, and generate a corresponding wiring path planning table; the wiring path planning table includes wire number, starting and ending points of the wire, recommended wire color, and wire diameter; S5. According to the wiring path planning table and the wiring hole position coordinates corresponding to each module, map the starting and ending points of the wire and the connection path of the wire in the experimental device picture through image registration technology.

2. The method according to claim 1, characterized in that After S1, the method further includes: Preprocess the schematic diagram picture and the experimental device picture respectively; the preprocessing includes: image denoising, contrast and brightness adjustment, geometric transformation, and perspective correction.

3. The method according to claim 1, wherein In S2, it includes: S21. Use a deep learning-based object detection model to identify component symbols in the schematic diagram picture, and obtain the category and bounding box coordinates corresponding to each component symbol in the schematic diagram picture, and the quantity corresponding to each type of component symbol; S22. Use the OCR algorithm to identify text symbols in the schematic diagram picture, and obtain the text symbols, parameters, and relevant note information corresponding to each component symbol; S23. According to the bounding box coordinates corresponding to each component symbol, use the Canny edge detection algorithm and the line segment detection algorithm to identify line segments in the schematic diagram picture, and obtain the connecting wires between each component symbol; S24. Use the component symbols as nodes and the connecting wires as edges to construct a topological structure diagram corresponding to the schematic diagram picture; the edge carries connection information; the connection information includes: the coordinates of the two endpoints corresponding to the connecting wire and the connection confidence between the two endpoints; S25. Use graph theory methods to extract the connection relationship between the connecting wires and the component symbols in the topological structure diagram; S26. Aggregate the category, text symbol, parameters, relevant remarks information corresponding to each component symbol, and the quantity of each type of component symbol to generate a schematic component analysis table; aggregate the connection relationships between each component symbol to generate a schematic component connection relationship table; the schematic component connection relationship table includes: the number of loops, the number of wires corresponding to each loop, the starting position and the ending position of the wire.

4. The method according to claim 3, characterized in that, The S23 includes: S231. Use the Canny edge detection algorithm and the line segment detection algorithm to identify line segments in the schematic diagram picture, and obtain the connection line segments corresponding to the schematic diagram picture; the connection line segments include the internal line segments of the component symbols and the connection wires between the component symbols. S232. Calculate the overlapping area between the target connection line segment and the bounding box corresponding to each component symbol according to the bounding box coordinates corresponding to each component symbol. S233. When the target connection line segment is completely within the bounding box of the target component symbol, the target connection line segment is the internal line segment of the target component symbol; when the target connection line segment crosses the bounding boxes of multiple component symbols or has no overlapping area with all component symbols, the target connection line segment is a connection wire.

5. The method according to claim 1, wherein After using the image segmentation technology to divide the experimental device picture into multiple regions according to modules, and the multiple regions respectively correspond to different modules in the experimental device image, the S3 further includes: In response to the fine-tuning operation performed by the client on the divided regions in the experimental device picture, use the pre-trained deep learning classifier to identify the adjusted multiple regions, and obtain the identification information corresponding to each adjusted region; the fine-tuning operation is an operation to adjust the boundary line of the target region or the position of the wiring holes within the target region.

6. The method according to claim 1, characterized in that, The S4 includes: S41. Generate a wiring view corresponding to the schematic diagram picture according to the category, text symbol, parameters, and relevant remarks information corresponding to each component symbol, the quantity of each type of electrical component symbol, and the connection relationships between each component symbol; each component symbol in the wiring view carries the corresponding text symbol, parameters, and relevant remarks information. S42. Match each component in the wiring view with the corresponding module according to a plurality of preset package libraries; and match the correct wiring holes for each component symbol according to the text symbol, parameters, and relevant remarks information carried by each component symbol. S43. Aggregate the wiring holes corresponding to each component symbol to generate a corresponding wiring path planning table.

7. The method according to claim 3, characterized in that After S5, the method further includes: S6. Extract the starting position and the ending position of the wire from the schematic component connection relationship table to generate a data structure; extract the module name, wiring hole position name, wiring hole position identifier, wiring hole position color, and wiring hole position coordinates from the experimental device module analysis table to establish an index table. S7. Search for the component category corresponding to the starting position and the ending position of each wire in the data structure; and search the index table according to the component category to obtain the wiring hole position identifier and the wiring hole position coordinates corresponding to the component category, and generate a matching table. S8. Store the coordinates and positions of the wires to be connected corresponding to each wire in the experimental device picture in a graph data structure, and compare the coordinates and connection positions of the wires to be connected corresponding to each wire with the matching table to verify the connectivity and correctness of the wires.

8. A vision-based experimental wiring assistance system, characterized in that Including: A data acquisition module, configured to obtain the schematic diagram picture and the experimental device picture in response to the schematic diagram picture and the experimental device picture uploaded by the client; A schematic diagram parsing module, configured to use an object detection algorithm based on deep learning, a Canny edge detection algorithm, a line detection algorithm, and a graph theory analysis method to identify the schematic diagram picture, and obtain the category, text symbol, parameters, and relevant note information corresponding to each component symbol in the schematic diagram picture, the quantity of each type of component symbol, and the connection relationship between each component symbol; An experimental device diagram parsing module, configured to divide the experimental device picture into multiple regions according to modules by using image segmentation technology, and the multiple regions respectively correspond to different modules in the experimental device picture; And identify each region by using a pre-trained deep learning classifier to obtain the identification information corresponding to each region; the identification information includes: module name, connection hole position name, connection hole position mark, connection hole position color, and connection hole position coordinates; A path planning and automatic wiring module, configured to convert the schematic diagram picture into a wiring view according to the category, text symbol, parameters, and relevant note information corresponding to each component symbol, the quantity of each type of electrical component symbol, and the connection relationship between each component symbol, and generate a corresponding connection path planning table; the connection path planning table includes wire number, start and end points of the wire, recommended wire color, and wire diameter; An image synthesis and annotation module, configured to map the start and end points of the wire and the connection path of the wire in the experimental device picture through image registration technology according to the connection path planning table and the connection hole position coordinates corresponding to each module.

Images