Auxiliary device and method

Abstract

To provide an auxiliary device for efficiently checking whether a terminal to which a cable is connected is appropriate. [Solution] An auxiliary device (10) used in the task of forming a test circuit by connecting a collective cable (32) extending from a tester (30) for testing a target device (20) to a terminal of the target device (20), comprises a mobile terminal (10A) including an imaging unit and a display unit for displaying visual data acquired by the imaging unit, and a main body (10B) including a guide generating unit, wherein the guide generating unit generates guidance information for connecting the cable to the terminal based on connection information between the terminal (22) for forming the test circuit and the cable, which is recorded in the manual for the target device and / or the tester, and displays the guidance information so as to be superimposed on first visual data including the terminal displayed on the display unit.

Description

[Technical Field]

[0001] The present disclosure relates to assistive devices and methods. [Background technology]

[0002] For maintenance of electrical equipment used in photovoltaic power generation facilities, etc., a test circuit is formed by connecting a tester cable to a terminal of the electrical equipment, and various evaluation tests are performed. In this case, it is necessary to connect the correct cable to each of the multiple terminals according to the test circuit. This work requires skilled techniques, and auxiliary techniques have been proposed to help even inexperienced workers perform this work, such as those described in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-171184 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional support technologies primarily focus on verifying whether the terminals to which cables are connected are appropriate (double-checking) on ​​behalf of or together with the worker, and do not contribute to improving work efficiency. Testers generally come with manuals that describe the correspondence between terminals and cables and the work procedures for the test circuit to be formed. However, terminal arrangements and locations may differ for each target device, and inexperienced workers must connect numerous cables on-site while comparing the manual with the terminals, which requires a lot of time. Given the recent shortage of science and engineering talent and the lack of people to transfer skills, there is a need for technology to improve work efficiency that does not rely on individual skills. The present disclosure aims to solve at least one of the problems with the above-mentioned conventional technologies. [Means for solving the problem]

[0005] The first auxiliary device of the present disclosure is an auxiliary device used in the task of forming a test circuit by connecting a cable extending from a tester for testing a target device to a terminal of the target device, and is equipped with an imaging unit, a display unit for displaying visual data acquired by the imaging unit, and a guide generation unit, wherein the guide generation unit generates guidance information for connecting the cable to the terminal based on connection information between the terminal and the cable for forming the test circuit that is recorded in the manual of the target device and / or the tester, and displays the guidance information so as to be superimposed on the first visual data including the terminal displayed on the display unit. [Effects of the Invention]

[0006] According to the present disclosure, at least one of the above problems of the prior art can be solved. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram of on-site operation of an auxiliary device. [Figure 2] FIG. 2 is a functional block diagram of an auxiliary device. [Figure 3] FIG. 10 is a diagram illustrating an example of connection information stored in a database. [Figure 4] 10 is a diagram illustrating an example of guidance information displayed superimposed on visual data displayed in real time on a display unit of a mobile terminal. FIG. [Figure 5] 10 is a diagram showing guidance information when the visual data does not include a connection destination terminal. FIG. [Figure 6] 10 is a diagram illustrating another example of guidance information displayed superimposed on visual data displayed in real time on the display unit of the mobile terminal. FIG. [Figure 7] 10 is a diagram illustrating another example of guidance information displayed superimposed on visual data displayed in real time on the display unit of the mobile terminal. FIG. [Figure 8] FIG. 10 is a flow chart showing a procedure for connecting a cable using an auxiliary device. [Figure 9]FIG. 10 is a diagram illustrating an example of an interface screen displayed on a display unit of a mobile terminal. DETAILED DESCRIPTION OF THE INVENTION

[0008] The first auxiliary device of the present disclosure is an auxiliary device used in the task of forming a test circuit by connecting a cable extending from a tester for testing a target device to a terminal of the target device, and is equipped with an imaging unit, a display unit for displaying visual data acquired by the imaging unit, and a guide generation unit, wherein the guide generation unit generates guidance information for connecting the cable to the terminal based on connection information between the terminal and the cable for forming the test circuit that is recorded in the manual of the target device and / or the tester, and displays the guidance information so as to be superimposed on the first visual data including the terminal displayed on the display unit.

[0009] One of the features of the first auxiliary device is that it includes a guide generating unit that generates guide information for connecting cables to terminals and displays the guide information superimposed on visual data including the terminals acquired by the imaging unit. Furthermore, the guide information is generated based on the manuals of the target device and / or the tester.

[0010] By using the first auxiliary device, a worker can smoothly connect cables by referring to the guidance information overlaid on the visual data displayed on the display unit while capturing an image of the terminals of the target device. This guidance information is generated based on the manuals for the target device and / or the tester, so the worker does not need to check the manuals on-site, and can perform the work smoothly and accurately even if he or she does not have sufficient work experience.

[0011] The second auxiliary device of the present disclosure is an auxiliary device that is the first auxiliary device and includes a cable determination unit, and the cable determination unit generates connection correctness information based on second visual data including the terminal and the cable connected to the terminal, and the connection information.

[0012] The cable judgment unit of the second auxiliary device generates correct / incorrect information on the cable connection based on visual data (second visual data) including the connected cable. The correct / incorrect cable connection has traditionally been checked by workers. The correct / incorrect information generated by the auxiliary device can be used to improve work accuracy and prevent accidents.

[0013] A third auxiliary device of the present disclosure is an auxiliary device in which, in the second auxiliary device, the connection information includes a work procedure for forming the test circuit, and the guide generation unit generates new guidance information based on the correct / incorrect information and displays it so as to be superimposed on the first visual data and / or the second visual data.

[0014] The new guidance information generated by the guide generation unit of the third auxiliary device is based on the correct / incorrect information generated by the cable judgment unit. The worker can refer to the guidance information based on the correct / incorrect information that is displayed overlaid on the visual data including the connected cable, and proceed with the work while constantly checking the accuracy of their work, allowing them to carry out the work more smoothly and accurately.

[0015] A fourth auxiliary device of the present disclosure is an auxiliary device in which, in the third auxiliary device, when the correct / incorrect information includes that the cable is connected to the wrong terminal, the new guidance information includes information indicating the terminal to which the cable is originally connected, or information instructing the cable to be disconnected.

[0016] Connecting cables in the wrong order or conducting a test with cables connected to the wrong terminals can cause damage to the target device or tester, or accidents such as fires. If the work procedure is incorrect, for example, if the cable connection order or the destination terminals are incorrect, the guidance generation unit of the fourth auxiliary device provides guidance on the correct connection order, the correct destinations, and disconnecting the cables. Based on this guidance information, the worker can take appropriate measures to prevent malfunctions and accidents. The content of this new guidance information can be determined based on the work procedure. For example, if the most recently connected cable is the correct cable to be connected but the destination is incorrect, the guidance information may include information guiding the worker to the correct destination. On the other hand, if the most recently connected cable is not the correct cable to be connected, i.e., if there is a cable that should have been connected before the cable, the guidance information may include information guiding the worker to disconnect the cable. Determining guidance information based on cable connection information contributes to more accurate work performance.

[0017] A fifth auxiliary device of the present disclosure is an auxiliary device in which, in any of the first to fourth auxiliary devices, the guide generation unit displays an interface screen on the display unit for accepting selection of the target device and / or the test circuit based on a pre-stored list to prompt the user to make a selection, and generates the guidance information based on the connection information stored for each of the target device and / or the test circuit selected by the user.

[0018] The guide generation unit of the fifth auxiliary device generates an interface screen for receiving selection of target equipment and / or test circuits from the user (operator) and displays it on the display unit. This selection display is based on a pre-stored list, and connection information is pre-stored in association with each item on the list. When the operator's selection is received, connection information corresponding to the selected target equipment and / or test circuit is read out, and guidance information is generated based on this. This configuration reduces the calculation cost of the guide generation unit and improves the processing efficiency of the auxiliary device. It also relaxes the hardware requirements for the auxiliary device, enabling smooth operation even with limited specifications. This is particularly preferable when the auxiliary device is a mobile terminal used on-site.

[0019] A sixth auxiliary device of the present disclosure is the second auxiliary device, wherein the cable determination unit includes a cable classification model that is machine-learned using training data composed of labeled images of the cable, and the cables extending from the tester are at least entirely or partially color-coded in different colors.

[0020] By color-coding multiple cables in whole or in part with different colors, the cable classification model can more accurately identify the cables. Furthermore, by assuming that the cables are color-coded with different colors, feature extraction becomes more efficient, thereby reducing the computational cost of the cable classification model. This improves the processing efficiency of the auxiliary device. The hardware requirements for the auxiliary device are relaxed, allowing smooth operation even with limited specifications. This is particularly preferable when the auxiliary device is a mobile terminal used on-site.

[0021] A seventh auxiliary device of the present disclosure is an auxiliary device that is the first auxiliary device and includes a tracking unit, wherein the tracking unit generates position information in the image of the terminal and / or the cable included in the visual data, which is an image, and the guide generation unit displays the guidance information on the display unit so as to be superimposed on the image based on the position information.

[0022] The tracking function allows guidance information to be overlaid on real-time visual data. Since workers can work while looking only at the display, the assistive device can be adapted to various form factors, such as wearable devices, depending on the purpose and location of use.

[0023] An eighth auxiliary device of the present disclosure is the third auxiliary device, wherein the guide generation unit generates a procedure list based on the connection information and displays it on the display unit, and if the correctness information includes that the cable is connected to the correct terminal, information indicating that the corresponding procedure in the procedure list has been completed is additionally displayed in the procedure list.

[0024] The procedure list displayed by the guide generation unit of the eighth auxiliary device helps the worker to work smoothly. Furthermore, by adding a check mark or the like to the completed procedure, the next task and the overall progress of the task can be confirmed, contributing to more accurate and smooth work.

[0025] A first method of the present disclosure is a method for using a first auxiliary device to perform an operation of connecting a cable extending from a tester for testing a target device to a terminal of the target device to form a test circuit, the method including: acquiring visual data including the terminal by the imaging unit; and connecting the cable to the terminal based on the guidance information displayed so as to be superimposed on the visual data displayed on the display unit.

[0026] According to the method of the present disclosure, the first auxiliary device is used, so that the worker does not need to check the manual on-site, and can carry out the work smoothly and accurately even if the worker does not have sufficient work experience.

[0027] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The specification exemplifies specific materials and methods for embodying the technical idea of ​​the disclosure. The technical idea of ​​the disclosure is not limited to the following specific examples. Various modifications can be made to the technical idea of ​​the disclosure within the scope of the matters described in the claims. In particular, it should be noted that the drawings are schematic and may differ from the actual product.

[0028] 1 is an explanatory diagram of on-site operation of an auxiliary device according to Example 1. The auxiliary device 10 includes a mobile terminal 10A and a main body 10B, which are connected via a network, and in this example, the main body 10B is installed in a location separate from the mobile terminal 10A and is connected via the Internet, allowing transmission and reception of data and the like.

[0029] The target device 20 is a device to be tested by the tester 30. In this example, the target device 20 is a ground fault directional relay included in a solar power generation facility. However, the target device 20 is not limited to the above. It may be any electrical equipment to which a tester is connected to form a test circuit and to which tests are to be performed for the purposes of performance verification, characteristic measurement, maintenance, calibration, etc. Examples of such electrical equipment include relays other than ground fault directional relays, circuit breakers, transformers, uninterruptible power supplies, inverters, synchronous generators, load switches, battery systems, power conditioners, phase modifiers, distributed power sources, capacitor banks, reactors, insulation monitoring devices, and wattmeters. A tester cable is connected to any of the target devices to form a test circuit and perform testing. The auxiliary device of the embodiment can be used to assist in the work of forming a test circuit for such target devices.

[0030] The target device 20 has a plurality of terminals 22. When the target device 20 is in operation, the required operational cables (not shown) are connected to the terminals 22, respectively. On the other hand, when forming a test circuit, a test bundled cable 32 is connected together with or instead of the operational cables. The bundled cable 32 is formed by bundling and aggregating a plurality of cables each with a different function. The bundled cable 32 has a plurality of clips 32A. Each clip 32A is placed at the end of a cable 32C having a different function. The clips 32A are connected to the corresponding terminals 22. A cable label 32B is attached near each clip 32A. The cable label 32B is used to identify each of the cables 32C included in the bundled cable 32.

[0031] Meanwhile, terminal labels 24 are also attached to the terminals 22. The relationship between the terminals 22 to be connected to the cables 32C when forming a specific test circuit is described or recorded in the manual for the target device 20 as the correspondence between the cable labels 32B and the terminal labels 24. By referring to this manual, the worker can correctly connect the two and form the test circuit.

[0032] The tester 30 is typically a portable device used to test the target device 20. In this example, the tester 30 is a phase characteristic tester. The phase characteristic tester applies a predetermined electrical signal to the ground fault directional relay to check whether various functions of the ground fault directional relay are operating normally. Note that the tester is not limited to the above and may be any device that is connected to the target device by cable and used for electrical testing of the target device. Examples of such testers include relay testers, insulation resistance meters, earth resistance meters, withstand voltage testers, circuit injectors, transformer ratio testers, partial discharge detectors, power quality analyzers, battery testers, and synchronization testers. Each tester is connected to the target device by cable, forms a test circuit, and is used to conduct a test.

[0033] The tester 30 includes switches 30A for performing various tests, knobs 30B (or sliders), a display 30C, and the like. The tester 30 also includes a terminal 30E for connecting a plurality of cables together. A bundled cable 32, which bundles and aggregates a plurality of cables 32C, is connected to the terminal 30E. The other end of the bundled cable 32 is made up of a plurality of cables 32C, and a clip 32A is arranged at the end (terminal) and connected to the terminal 22.

[0034] Each cable 32C is provided with a cable label 32B, allowing the worker to distinguish between the cables 32C by the cable label 32B. In this example, each cable has a conductor covered with an insulating layer and a protective layer, and the outermost protective layers are each color-coded. Color-coded cables allow for more accurate identification by the cable identification unit 50, as described below. However, the entire cable does not need to be color-coded; a similar effect can be achieved by color-coding only a portion of the cable. For example, color-coding the cables can be achieved by coloring the cable labels 32B in different colors. Alternatively, a method can be used in which the clips at the ends or their covering layers are each colored differently.

[0035] The manuals for the target device 20 and / or the tester 30 describe or record methods for conducting various tests. This information is referred to as connection information in this specification. The connection information may include a test circuit diagram, the correspondence between the terminals 22 and the cables 32C for forming the test circuit, and work procedures. The work procedures are a list of operations (work items) to be performed in the order in which they should be performed. Work items may include, for example, the operation of disconnecting an operating cable from a specific terminal 22 and the operation of connecting a specific cable 32C to a specific terminal 22. A skilled worker can connect the terminals 22 and the cables 32C in the appropriate order and in the appropriate correspondence depending on the type of test. On the other hand, inexperienced workers often refer to the manual when forming test circuits, which can be one of the reasons for the long work times.

[0036] The tester 30 is connected to form a test circuit for performing a phase characteristic test on the target device 20. The type of test can be varied depending on the target device and the purpose. Examples of such tests include electrical characteristic tests such as insulation resistance tests, withstand voltage tests, ground resistance tests, and transformer ratio tests; protective device-related tests such as operating current / voltage tests, time characteristic tests, trip interlock tests, and phase characteristic tests; signal / communication-related tests such as communication tests and protocol tests; and battery-related tests such as capacity tests, internal resistance tests, and discharge tests.

[0037] The mobile terminal 10A of the auxiliary device 10 is a smartphone terminal, a tablet terminal, or the like. The mobile terminal 10A has a processor, a memory, a camera, and a touch-sensitive display, and further has a communication function with the main body 10B. On the other hand, the main body 10B is a computer having a processor and a memory. In this example, the auxiliary device 10 has the mobile terminal 10A and the main body 10B. However, the auxiliary device is not limited to this form, and may be configured as a single portable device having the functions of both the mobile terminal 10A and the main body 10B.

[0038] Next, a detailed description will be given of the function of each unit of the auxiliary device 10. Fig. 2 is a functional block diagram of the auxiliary device 10. The mobile terminal 10A includes a control unit 12A, a display unit 12B, an imaging unit 12C, and a communication unit 12E.

[0039] The control unit 12A controls the entire mobile terminal 10A. The functions of the control unit 12A are realized by a processor or the like. The mobile terminal 10A may also include a storage unit (not shown). The storage unit stores programs and necessary data. The functions of the storage unit are realized by a memory or the like.

[0040] The display unit 12B is a touch-sensitive display that displays the visual data obtained by the imaging unit 12C, as well as guidance information and an interface screen generated by the guide generation unit 44 (described later). The display method and touch technology of the display are not particularly limited, and known methods can be combined as appropriate.

[0041] Examples of display formats include a liquid crystal display, an organic electroluminescence (EL) display, and a micro LED display. Examples of touch technologies include a capacitive type, a resistive type, an infrared type, and an optical type. By employing a touch-sensitive display, the display unit 12B also functions as an input device that accepts input from the operator.

[0042] The imaging unit 12C is a device for acquiring visual data of the terminals 22 and the like of the target device 20, and is a camera in this example. The imaging unit 12C may be something other than a camera as long as it can acquire visual data including the terminals 22 and the like of the target device 20, but it is preferable that it includes an image sensor. In addition to the image sensor, the imaging unit 12C may further include a depth sensor.

[0043] The visual data acquired by the imaging unit 12C may be a still image or a moving image (video). When the visual data is a video, the worker's understanding is more likely to be improved when the guidance information is overlaid and displayed.

[0044] The communication unit 12E is a function realized by hardware and software that handles network communication between the mobile terminal 10A and the main body 10B. The hardware includes a "Wi-Fi" module, a cellular communication module, a "Bluetooth" module, and the like as network interfaces. The communication unit 12E also includes driver programs corresponding to these pieces of hardware, and software such as a communication protocol stack. The communication unit 12E handles real-time data transmission and reception with the main body.

[0045] The main body 10B is a computer equipped with a processor and memory, and further includes software for implementing each function. First, the control unit 58 is implemented by a processor and has the function of comprehensively controlling the operation of the entire auxiliary device 10. The memory unit 56 is a component for storing various information, data, and programs necessary for the operation of the auxiliary device 10, and holds, for example, related data used to generate connection information and guidance information. Furthermore, the communication unit 54 has the function of transmitting and receiving data to and from the target device 20, the tester 30, the mobile terminal 10A, and / or external devices, enabling the acquisition and transmission of information necessary for the operation of the auxiliary device 10.

[0046] The visual data acquisition unit 40 is a function realized by the processor executing a program stored in memory, and receives visual data acquired by the imaging unit 12C and stores it in the storage unit 56. This communication is performed by the communication unit 54, and the function of the communication unit 54 is similar to the function of the communication unit 12E of the mobile terminal 10A.

[0047] The terminal detection unit 42 is a function realized by the processor executing a program stored in memory, and detects the terminals 22 of the target device 20 included in the acquired visual data. There are no particular limitations on the method for detecting the terminals 22 from the visual data, but one example is a method for detecting the terminals 22 from the visual data using a known object detection algorithm.

[0048] In one embodiment, the terminal detection unit 42 includes a terminal classification model for detecting and identifying terminals included in the visual data. The basic structure of the terminal classification model can be a combination of "YOLO" and "Faster R-CNN," among others. When using the above model as the basic structure, machine learning is performed in advance using an image dataset of labeled terminals 22.

[0049] Furthermore, the terminal detection unit 42 can also use an edge detection algorithm such as "Canny" in addition to the object detection algorithm to detect the terminals 22. If the visual data is video, the data may be divided into frames, and the terminals 22 may be detected in each image using the above method.

[0050] The terminal detection unit 42 may further detect the terminal labels 24 and read information written on the terminal labels 24 to identify the terminals. Reading the terminal labels 24 can be achieved by cutting out an area including the terminal labels 24 from the area of ​​the terminals 22 cut out by an object detection algorithm, and reading characters from that area using an OCR library such as "Tessaeract." Note that the method for cutting out the area including the terminal labels 24 from the area of ​​the terminals 22 is not particularly limited, and an object detection algorithm similar to that used to detect the terminals 22 may be used.

[0051] The terminal detection unit 42 may refer to a list stored in advance for each type of target device 20 to check whether the character string read from the terminal label 24 matches a specific character string. In other words, the OCR detection unit 42 may be configured to supplement the detection accuracy with the pre-stored list. Generally, the terminal label 24 provided in the target device 20 is described or recorded in the manual of the target device 20. The list is prepared in advance based on the manual of the target device 20 and stored in the storage unit 56. Comparing the character string read from the terminal label 24 with the list enables more accurate reading of the terminal label 24. If the read character string does not exist in the list, the reading is deemed to have failed, and the terminal label 24 may be read again.

[0052] The procedure for training the terminal classification model of the terminal detection unit 42 begins with collecting visual data of terminals of various shapes and structures as training data. The collected visual data includes the type, color, label, and relative position of the terminals. A comprehensive data set is also constructed, including data of terminals in connected and unconnected states, and data photographed under different background environments. This ensures a wide variety of training data for improving the discrimination accuracy of the terminal detection unit 42.

[0053] The terminal detection unit 42 is trained using the above dataset to learn the characteristics of terminals from the input visual data. In this training, characteristics such as terminal shape, color, size, and arrangement pattern are extracted, and a classification model is generated. Specifically, supervised learning is applied, and training is performed by associating the correct label for each terminal with the visual data. After training, the terminal detection unit 42 becomes able to detect and identify terminals with high accuracy from unknown visual data.

[0054] The tracking unit 46 is a function realized by the processor executing a program stored in memory, and tracks the position of the terminal 22 identified in the acquired visual data. The tracking unit 46 works in concert with the terminal detection unit 42. As a specific example, when the image acquired by the imaging unit 12C is divided into frames, the area of ​​the terminal 22 is extracted from one of the frames using an object detection algorithm in the terminal detection unit 42. The tracking unit 46 tracks the movement of the detected terminal 22. Once tracking by the tracking unit 46 has begun, it is not necessary for the terminal detection unit 42 to detect the terminal 22 again unless tracking fails or the terminal 22 is no longer visible in the visual data.

[0055] Preferably, the tracking unit 46 is configured to track the movement of the cable 32C detected by the cable detection unit 48, which will be described later, using a similar procedure. The tracking unit 46 generates position information of the terminal 22 and the cable 32C in the visual data (if the visual data is video, the position information is generated for each frame or for each frame at a predetermined interval), and passes it to the guide generation unit 44. This makes it easier for the guide generation unit 44 to display guidance information so that it is superimposed on the visual data.

[0056] For tracking, known tracking algorithms such as "Optical Flow," "Kalman Filter," and "Deep SORT" can be used. Note that tracking may fail if the mobile terminal 10A is moved quickly. In this case, the terminal detection unit 42 may be configured to perform object detection at regular intervals regardless of whether tracking is successful. By providing the terminal detection unit 42 and the tracking unit 46, real-time overlay display of guidance information becomes possible.

[0057] The guide generation unit 44 is a function realized by the processor executing a program stored in memory, and generates guidance information in accordance with the work procedure based on the connection information recorded in the manual for the target device 20 and / or the tester 30, and displays the guidance information so that it is superimposed on the visual data including the terminals 22 displayed on the display unit 12B.

[0058] The manual for the target device 20 and / or the tester 30 describes or records connection information for the terminals 22 and cables 32C for forming a test circuit. It is preferable that the connection information be organized in advance and compiled into a database for each type of target device 20, tester 30, and test circuit. However, even if the connection information is organized for each type of target device 20 and test circuit, substantially the same effect can be achieved. This is because, although testers 30 often have different specifications depending on the manufacturer, once the test circuit is specified, the work procedures are often similar even when using different testers 30.

[0059] FIG. 3 is a diagram showing an example of connection information stored in a database. The connection information 60 includes the correspondence between the terminals 22 and the cables 32C, as well as work procedures. The work procedures include operations for disconnecting the operating cable connected to the terminal 22, such as "disconnecting the P1 and P2 terminals of the relay" (step 60A) and "disconnecting the a and c terminals of the relay" (step 60B), and operations for connecting the cable 32C to the terminal 22, such as "connecting the P1 cable to the P1 terminal of the relay" (step 60C) and "connecting the P2 cable to the P2 terminal of the relay" (step 60D). The connection information 60 may be generated automatically based on the manuals for the target device 20 and / or the tester 30, or may be generated manually (i.e., by an operator). The connection information 60 is generated in advance, organized into a database, and stored in the storage unit 56.

[0060] The "guidance information" generated by the guide generating unit 44 in accordance with the work procedure is generated based on the connection information. FIG. 4 is a diagram showing an example of guidance information displayed superimposed on visual data 66 displayed in real time on the display unit 12B of the mobile terminal 10A. The guidance information 64A first includes information guiding the connection of a specific cable 32C to a specific terminal 22. This guidance information 64A includes a cursor image 62A pointing to the specific terminal 22, text information 62B indicating the type of cable 32C to be connected to this terminal 22, and the like. The guidance information 64A may be displayed on the display unit 12B until the connection status, which will be described later, is confirmed.

[0061] The cursor image 62A may be displayed only when the destination terminal 22 is included in the visual data 66. FIG. 5 is a diagram showing guidance information 64B when the destination terminal 22 is not included in the visual data 66. The guidance information 64B includes text information 62B and a cursor image 62C. In this case, the destination terminal 22 is not included in the visual data 66, so the cursor image 62C points outside the screen. In other words, it indicates that the mobile terminal 10A should be moved leftward.

[0062] The guidance information may also include information that provides guidance when cable 32C is connected to the wrong terminal 22. Fig. 6 is a diagram showing another example of guidance information that is displayed superimposed on visual data 66 that is displayed in real time on display unit 12B of mobile terminal 10A. Guidance information 64C includes a frame image 62E that points to the wrongly connected cable 32C, a cursor image 62F that points to the terminal 22 to which cable 32C is originally connected, text information 62G that instructs the user to remove (disconnect) or move the connected cable 32C, and the like.

[0063] The guidance information may further include a work procedure (procedure list) and information indicating that a specific procedure in the procedure list has been completed. FIG. 7 is a diagram showing another example of guidance information superimposed on visual data 66 displayed in real time on the display unit 12B of the mobile terminal 10A. The guidance information 64E includes a procedure list 62H generated by the guide generator 44 and information 62J, consisting of checks and strikethroughs, indicating procedures that have already been completed. The procedure list 62H may be generated based on connection information. Furthermore, the information 62J indicating procedures that have already been completed may be generated based on correct / incorrect information generated by the cable evaluator 50, which will be described later.

[0064] It is preferable that the images and text information included in the guidance information are generated in advance either automatically or manually, and are stored in memory unit 56 in a state where they are linked to each condition, categorizing them according to the work procedure (procedure list), the state of the visual data (whether the target terminal, cable is included in the image, etc.), and whether the work is correct or incorrect.

[0065] Returning to FIG. 2, the cable detection unit 48 is a function realized by a processor executing a program stored in memory, and detects the cable 32C of the tester 30 contained in the acquired visual data. There is no particular limitation on the method for detecting the cable 32C from the visual data, but an algorithm similar to that of the terminal detection unit 42 may be adopted. In this case, the cable 32C may be distinguished using the cable label 32B instead of the terminal label 24, or by the color of the cable 32C. In particular, distinguishing by color is preferred because of its high accuracy.

[0066] The cable detection unit 48 may use the same method for identifying the cable 32C by character identification and color identification as that used by the terminal detection unit 42. That is, the cable detection unit 48 is provided with a cable classification model for detecting and identifying cables included in visual data, and by inputting the visual data, detects and identifies the cable 32C included in the visual data.

[0067] The procedure for training the cable classification model of the cable detection unit 48 is as follows: first, visual data of cables 32C (which may include clips 32A and cable labels 32B, collectively referred to as "cables, etc.") of various shapes and structures is collected as training data. The collected visual data includes the type, color, label, and relative position of the cables, etc. Furthermore, a comprehensive data set is constructed, including data of cables 32C in connected and unconnected states, and also data captured under different background environments. This ensures a wide variety of training data for improving the discrimination accuracy of the cable detection unit 48.

[0068] The cable detection unit 48 is trained using the above dataset to learn the characteristics of cables and the like from input visual data. In this training, characteristics such as the shape, color, dimensions, and placement pattern of cables and the like are extracted, and a classification model is generated. Specifically, supervised learning is applied, and training is performed by associating correct labels for cables and the like with the visual data. After learning, the cable detection unit 48 becomes able to detect and identify cables 32C with high accuracy from unknown visual data.

[0069] The cable determination unit 50 is a function realized by the processor executing a program stored in memory, and detects the connection of the cable 32C to the terminal 22, and determines whether the correspondence between the connected terminal 22 and the cable 32C is correct or incorrect based on the detection results by the terminal detection unit 42 and the cable detection unit 48.

[0070] The cable determination unit 50 first detects that the cable 32C has been connected to the terminal 22 (connected state) based on the visual data acquired by the visual data acquisition unit 40. In this specification, for convenience, the visual data after the terminal 22 to be guided and the cable 32C have been connected may be referred to as second visual data, and the visual data when the terminal 22 to be guided and the cable 32C are not yet connected may be referred to as first visual data.

[0071] The method by which the cable determination unit 50 detects the connection state is not particularly limited, but may employ a method (difference method) in which the first visual data when the cable 32C is disconnected is compared with the visual data at the time of determination (which may be "second visual data") to detect changes. In the difference method, the cable determination unit 50 first performs preprocessing on the acquired visual data. The preprocessing method is not particularly limited, but examples include resizing the visual data, color conversion, noise removal, edge detection, extraction of a region of interest (ROI), binarization, and contour detection. Template matching or an object detection model can be used for ROI extraction. Furthermore, "YOLO" or "OpenCV" can be used for contour detection.

[0072] Next, if the detected positional relationship between the cable 32C and the terminal 22 is within a predetermined range, the cable determination unit 50 determines that the cable 32C and the terminal 22 are connected, and then performs a true / false determination. The method for determining true / false is not particularly limited, but examples include identifying the characters on the terminal label 24 and the cable label 32B (character identification) and color identification. Note that if the terminal and the cable 32C in the visual data have already been detected, identified, and tracked by the terminal detection unit 42 and the cable detection unit 48, respectively, there is no need to perform new detection and identification.

[0073] By comparing the correspondence between the terminal 22 and the cable 32C in the connected state with the connection information, the cable determination unit 50 generates correct / incorrect information about the cable 32C connected to the terminal 22. The cable determination unit 50 generates determination information on whether the connection is correct or incorrect, and sends it to the guide generation unit 44, etc.

[0074] Next, a description will be given of the procedure for connecting a cable using the auxiliary device 10. Fig. 8 is a flow chart showing the procedure for connecting a cable using the auxiliary device.

[0075] First, in step S10, the guide generating unit 44 displays an interface screen on the display unit 12B of the mobile terminal 10A for accepting selection of the target device 20, the tester 30, and the test circuit based on a list stored in advance. The type of target device 20 and the test circuit (the type of test to be performed) can be determined from visual data, but accepting at least the selection of the target device 20 or the test circuit from the user (operator) and using that information can speed up processing or enable processing even on a computer (tablet) with low processing power.

[0076] FIG. 9 is a diagram illustrating an example of an interface screen displayed on the display unit 12B of the mobile terminal 10A. The interface screen 70 includes a first drop-down list 71 for selecting the target device 20, a second drop-down list 72 for selecting the tester 30, a third drop-down list 73 for selecting the test circuit, a cancel button 74, and a confirm button 75. The contents of the lower drop-down list change in conjunction with a selection in the upper drop-down list. That is, the list displayed in the lower drop-down list changes in conjunction with a selection in the upper drop-down list, which is displayed based on a certain list. Specifically, the selectable contents of the tester 30 in the second drop-down list 72 and the selectable contents of the test circuit in the third drop-down list 73 change in conjunction with a selection in the first drop-down list 71. Furthermore, the selectable contents of the third drop-down list 73 change in conjunction with a selection of the tester 30 in the second drop-down list 72. Images and lists required for these displays are stored in advance in the storage unit 56.

[0077] The guide generation unit 44 displays the interface screen 70 on the display unit 12B to prompt the user (operator) to make a selection. The operator selects the target device 20, the tester 30, and the test circuit based on the interface screen 70, touches the decision button 75, and transmits the selection. The guide generation unit 44 reads out the corresponding connection information stored in the storage unit 56 based on the selection. As described above, the connection information is information generated based on the manuals for the target device 20 and / or the tester 30, etc. In other words, once the target device 20, the tester 30, the test circuit, etc. are identified, the connection information can also be easily identified. The storage unit 56 stores the connection information linked to this information. The selected connection information is used in subsequent processing. The decision button 75 on the interface screen 70 may be used as a signal to start acquiring visual data.

[0078] Next, in step S11, visual data including the terminals 22 of the target device 20 is acquired. Specifically, the worker points the imaging unit 12C of the mobile terminal 10A at the terminals 22 of the target device to acquire the visual data. The visual data acquisition unit 40 of the main body 10B acquires this visual data via the communication unit 12E of the mobile terminal 10A and temporarily stores it in the memory unit 56.

[0079] Next, in step S12, the terminal detection unit 42 detects the terminals 22 included in the visual data. The method for detecting the terminals 22 has already been described.

[0080] Next, in step S13, the control unit 58 references the connection information and determines whether all work procedures for forming the target test circuit have been completed. If this determination results in all work procedures being completed (step S13: YES), the processing flow ends. On the other hand, if there are any incomplete work procedures (step S13: NO), the guide generation unit 44 generates guidance information corresponding to the incomplete work procedures that should be processed first, based on the connection information stored in the storage unit 56 and the detected terminals 22 (step S14). This guidance information is generated as image data, text data, or a combination of these.

[0081] Next, in step S15, the tracking unit 46 identifies the terminal 22 that is the target of the work in the current work procedure (hereinafter referred to as the target terminal) and starts tracking. The "work procedure" here includes, for example, the work of disconnecting an already connected operating cable from the target terminal and the work of connecting a specific cable of a tester to the target terminal. The tracking unit 46 tracks the target terminal in real time using visual data transmitted from the mobile terminal 10A. Furthermore, if the target terminal temporarily disappears from the visual data (moves off the screen), the tracking unit 46 cooperates with the terminal detection unit 42 to perform re-detection and continue tracking. This makes it possible to accurately grasp the position and status of the target terminal. Then, based on the position information generated by the tracking unit 46, the guide generation unit 44 displays guidance information superimposed on the visual data displayed in real time on the display unit 12B.

[0082] Next, in step S16, the cable detection unit 48 detects the connection of the cable 32C to the target terminal. The cable detection unit 48 identifies the cable 32C included in the visual data based on the color of the cable label 32B and the cable 32C. The identified cable 32C is tracked by the tracking unit 46.

[0083] Furthermore, in step S17, the cable determination unit 50 determines whether the cable 32C is connected to the target terminal. For example, by using a differential method, it is determined whether the cable 32C and the terminal 22 are connected. If it is determined that the cable 32C and the terminal 22 are not connected (step S17: NO), the processes from step S16 onwards are repeated again. On the other hand, if it is determined that the cable 32C and the terminal 22 are connected (step S17: YES), then the authenticity of the connected cable 32C is determined (step S18).

[0084] The determination of whether the connected cable 32C is correct or incorrect is performed by comparing the correspondence between the terminal 22 and the cable 32C detected and tracked by the terminal detection unit 42, the cable detection unit 48, and the tracking unit 46 with the connection information. If it is determined as a result of this comparison that the correct cable 32C is connected to the target terminal (step S18: YES), the operation on the target terminal is completed, and information indicating that the current work procedure is completed is stored in the storage unit 56. Then, the process returns to step S13. On the other hand, if it is determined that the correct cable is not connected (step S18: NO), the guide generation unit 44 generates new guidance information indicating that the operation performed based on the current work procedure is incorrect and indicating the correct operation method, and displays the generated guidance information on the display unit 12B (step S19). Then, the process returns to step S15.

[0085] Through the above procedure, necessary guidance information is appropriately displayed on the display unit of mobile terminal 10A until all work procedures included in the connection information are completed. This allows the worker to perform cable connection reliably and efficiently while checking the visual guide.

[0086] Note that steps S16 to S18 are described for the case where the work content (operation) is connecting cable 32C to terminal 22. If the work content is work to disconnect an already connected operating cable from the target terminal, steps S16 to S18 can be changed depending on the work content. Furthermore, if the guidance information includes a procedure list, when completion information for the current work procedure is generated in step S19, the guide generation unit 44 may be configured to update the guidance information and display information indicating that the specified procedure has been completed. [Explanation of symbols]

[0087] 10 auxiliary device, 10A portable terminal, 10B main body, 20 target device, 22 terminal, 24 terminal label, 30 tester, 32B cable label, 32C cable, 40 visual data acquisition unit, 42 terminal detection unit, 44 guide generation unit, 46 tracking unit, 48 cable detection unit, 50 cable judgment unit

Claims

1. An auxiliary device used in an operation of forming a test circuit by connecting a cable extending from a tester for testing a target device to a terminal of the target device, An imaging unit, a display unit for displaying visual data acquired by the imaging unit, and a guide generation unit, the guide generation unit generates guidance information for connecting the cable to the terminal based on connection information between the terminal and the cable for forming the test circuit, which is recorded in a manual for the target device and / or the tester, and displays the guidance information so as to be superimposed on first visual data including the terminal displayed on the display unit; The guide generation unit displays an interface screen on the display unit to accept selection of the target device and / or the test circuit based on a pre-stored list, prompting the user to make a selection, and generates the guidance information based on the connection information stored for each of the target device and / or the test circuit selected by the user.

2. A cable determination unit is provided, The auxiliary device according to claim 1 , wherein the cable determination unit generates correct / incorrect connection information based on the connection information and second visual data including the terminal and the cable connected to the terminal.

3. the connection information includes a procedure for forming the test circuit; The assist device according to claim 2 , wherein the guide generation unit generates new guidance information based on the correct / incorrect information and displays the new guidance information so as to be superimposed on the first visual data and / or the second visual data.

4. The auxiliary device of claim 3, wherein when the correct / incorrect information includes that the cable is connected to the wrong terminal, the new guidance information includes information indicating the terminal to which the cable is originally connected or information instructing the cable to be disconnected.

5. the cable determination unit includes a cable classification model that is machine-learned using training data configured from labeled images of the cable; 3. The auxiliary device according to claim 2, wherein at least the entire or part of the cables extending from the tester are color-coded with different colors.

6. Equipped with a tracking unit, the tracking unit generates position information of the terminal and / or the cable included in the visual data, which is an image, in the image; The assist device according to claim 1 , wherein the guide generating unit displays the guidance information on the display unit so as to be superimposed on the video based on the position information.

7. the guide generation unit generates a procedure list based on the connection information and displays the procedure list on the display unit; The auxiliary device of claim 3, wherein, when the correct / incorrect information includes that the cable is connected to the correct terminal, information indicating that the corresponding procedure in the procedure list has been completed is additionally displayed in the procedure list.

8. 10. A method for assisting an operation of forming a test circuit by connecting a cable extending from a tester for testing a target device to a terminal of the target device, using the auxiliary device according to claim 1, comprising: generating guidance information for connecting the cable to the terminal based on connection information between the terminal and the cable for forming the test circuit, which is recorded in a manual for the target device and / or the tester; displaying the guidance information so as to overlap first visual data including the terminal displayed on a display unit of the auxiliary device; a method in which an interface screen for accepting selection of the target device and / or the test circuit based on a pre-stored list is displayed on the display unit to prompt the user to make a selection, and the guidance information is generated based on the connection information stored for each of the target device and / or the test circuit selected by the user.

Images