Assembly support device and assembly support method
The assembly support device and method enhance aircraft MRO efficiency by using learning models to identify parts from multiple images, reducing man-hours and improving accuracy through augmented reality assistance.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- MITSUBICHI HEAVY IND AERO ENGINES LTD
- Filing Date
- 2022-05-16
- Publication Date
- 2026-05-18
Smart Images
Figure 0007860813000001 
Figure 0007860813000002 
Figure 0007860813000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to an assembly support device and an assembly support method.
Background Art
[0002] Patent Document 1 discloses a parts management device for managing aircraft parts. This parts management device includes a mobile device. The mobile device displays the position information of aircraft parts on the mobile device, and when the mobile device is within a zone including a certain position, it determines whether a part exists at that position using a sensor system within the mobile device, and is configured to indicate that a part exists at that position when a part exists at that position.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] By the way, when performing an assembly operation in aircraft MRO (Maintenance, Repair and Overhaul), an operator has to obtain information such as a parts list, manual, drawing, work record, work instruction, etc. from an information database, and perform the assembly operation while identifying parts by comparing that information with the shape of the parts. Therefore, proficiency of the operator in knowledge about parts etc. is required, and it takes a lot of man-hours. Also, Patent Document ① does not disclose any findings for solving such problems.
[0005] In view of the above circumstances, at least one embodiment of the present disclosure aims to provide an assembly support device and an assembly support method capable of reducing the man-hours of an assembly operation in aircraft MRO.
Means for Solving the Problems
[0006] To achieve the above objective, the assembly support device according to at least one embodiment of this disclosure is An assembly support device for assisting in the assembly of aircraft engines, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, then an identifier estimation unit is configured to receive the multiple anisotropic images as input, estimate the identifier of the target component using a learning model that shows the relationship between the component image and the component identifier, and output estimation result information indicating at least one provisional candidate for the identifier of the target component for each of the anisotropic images. A true candidate information output unit is configured to output true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on a plurality of estimation result information output from the identifier estimation unit by inputting the plurality of images with different orientations into the identifier estimation unit. A display control unit configured to display the aforementioned candidate information on a worker's terminal, It is equipped with.
[0007] To achieve the above objective, the assembly support method according to at least one embodiment of this disclosure is An assembly support method for assisting the assembly work of an aircraft engine, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, the candidate estimation step involves estimating the identifier of the target component using a learning model that shows the relationship between the component image and the component identifier, based on the multiple anisotropic images, and outputting estimation result information indicating at least one provisional candidate for the identifier of the target component for each of the anisotropic images. A true candidate information output step outputs true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on the set of estimation result information output for each of the disparately oriented images, The steps include: performing display control to display the aforementioned candidate information on the operator's terminal; It is equipped with. [Effects of the Invention]
[0008] According to at least one embodiment of this disclosure, an assembly support device and assembly support method are provided that can reduce the man-hours required for assembly work in aircraft MRO (Maintenance, Repair, and Overhaul). [Brief explanation of the drawing]
[0009] [Figure 1] This is a block diagram showing the schematic configuration of an assembly support system 100 according to one embodiment. [Figure 2] This figure shows an example of the hardware configuration of the assembly support device 6. [Figure 3] This is a block diagram showing the functional configuration of the assembly support device 6. [Figure 4] This figure shows a part of the method for assisting the assembly of aircraft engine parts using the assembly support system 100. [Figure 5] This figure shows an example of estimation result information output by the identifier estimation unit 10 for each of the three two-dimensional line drawings when those three two-dimensional line drawings are input to the identifier estimation unit 10. [Figure 6] This figure shows a part of the method for assisting the assembly of aircraft engine parts using the assembly support system 100. [Figure 7] This figure shows screen E1, which is an example of the screen of AR device 2 in landscape mode. [Figure 8] This figure shows screen E2, which is an example of screen 2 of the AR device in superimposed display mode. [Figure 9] This figure shows screen E3, which is an example of the screen of AR device 2 in simple display mode. [Figure 10] This figure shows screen E4, an example of the screen of AR device 2 in parallel display mode. [Figure 11] This diagram illustrates how the user can switch between display modes. [Modes for carrying out the invention]
[0010] Hereinafter, some embodiments of the present disclosure will be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of the components described as embodiments or shown in the drawings are not intended to limit the scope of the invention thereto, but are merely illustrative examples. For example, expressions representing relative or absolute arrangements such as "in a certain direction", "along a certain direction", "parallel", "orthogonal", "center", "concentric", or "coaxial" not only strictly represent such arrangements, but also represent a state in which there are tolerances or relative displacements with angles and distances that can achieve the same function. For example, expressions representing a state in which things such as "identical", "equal", and "homogeneous" are equal not only strictly represent an equal state, but also represent a state in which there are tolerances or differences that can achieve the same function. For example, expressions representing shapes such as a rectangular shape or a cylindrical shape not only represent shapes such as a rectangular shape or a cylindrical shape in a geometrically strict sense, but also represent shapes including concave and convex portions, chamfered portions, etc. within a range where the same effect can be obtained. On the other hand, the expressions "comprising", "having", "including", or "possessing" a certain component are not exclusive expressions that exclude the existence of other components.
[0011] (Assembly Support System) FIG. 1 is a block diagram showing a schematic configuration of an assembly support system 100 according to an embodiment. The assembly support system 100 is a system for supporting the assembly work in the MRO of an aircraft engine (not shown), and includes an AR device 2 (operator terminal), an instructor terminal 4, and an assembly support device 6.
[0012] The AR device 2 is a terminal for use by an operator (hereinafter simply referred to as "operator") who performs assembly work in the MRO of aircraft engines. The AR device 2 may be, for example, a smartphone, a tablet terminal, or AR glasses (smart glasses). In this specification, "AR" means Augmented Reality. The AR device 2 includes a computer including, for example, a processor, a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), a camera, a display, and a communication device.
[0013] The instructor terminal 4 is a terminal for use by an instructor who gives instructions for assembly work to an operator in the MRO of aircraft engines. The instructor terminal 4 may be a portable terminal such as a tablet terminal, a smartphone, or a laptop computer, or a stationary terminal such as a desktop computer. The instructor terminal 4 is constituted by a computer including, for example, a processor, a memory such as a ROM and a RAM, a display, and a communication device.
[0014] The assembly support device 6 is a device for assisting in the assembly of an aircraft engine and may be configured, for example, as a server device. Figure 2 shows an example of the hardware configuration of the assembly support device 6. In the example shown in Figure 2, the assembly support device 6 is configured using a computer that includes, for example, a processor 72, RAM 74, ROM 76, HDD (Hard Disk Drive) 78, input I / F 80, and output I / F 82, which are connected to each other via a bus 84. The assembly support device 6 is configured to communicate wirelessly or otherwise with the AR device 2 and the instruction terminal 4, respectively, via the input I / F 80 and output I / F 82. However, in some embodiments, the assembly support device 6 may be built into the AR device 2 or the instruction terminal 4. Note that the hardware configuration of the assembly support device 6 is not limited to the above and may be configured as a combination of a control circuit and a storage device. The assembly support device 6 is configured by a computer executing a program that realizes each of the functions of the assembly support device 6. The functions of each part of the assembly support device 6 described below are realized, for example, by loading a program held in ROM 76 into RAM 74 and executing it with processor 72, as well as by reading and writing data to RAM 74 and ROM 76.
[0015] (Details of the assembly support device) Figure 3 is a block diagram showing the functional configuration of the assembly support device 6. As shown in Figure 3, the assembly support device 6 includes a two-dimensional line drawing conversion unit 8, an identifier estimation unit 10, a true candidate information output unit 12, an identifier acquisition unit 14, a work manual selection unit 15, a display control unit 16, and a storage unit 18.
[0016] The 2D line drawing conversion unit 8 is configured to convert the image of the component into a 2D line drawing. In this specification, "component" includes modules.
[0017] The identifier estimation unit 10 (predictor) is configured to receive a two-dimensional line drawing of a target part, which is a part to be identified in an aircraft engine, as input. It uses a learning model 20 that shows the relationship between the two-dimensional line drawing of the part and an identifier such as the part's model number to estimate the identifier of the target part. The unit is configured to output estimation result information that includes multiple provisional candidates for the identifier of the target part shown by the input two-dimensional line drawing, and an indicator of the confidence level of each of the provisional candidates (e.g., rank, probability, similarity, etc.). The identifier estimation unit 10 receives a two-dimensional line drawing of a target part as input. It uses the learning model 20 to estimate the identifier of the target part and outputs estimation result information that includes at least one provisional candidate for the identifier of the target part shown by the input two-dimensional line drawing, and an indicator of the confidence level of each of the provisional candidates. The learning model 20 is obtained by machine learning training data that associates 2D line drawings of aircraft engine parts with corresponding parts. Well-known methods such as neural networks, SVM (Support Vector Machine), random forests, and regression analysis may be applied to this machine learning. The learning model 20 is stored in the memory unit 18 and is read from the memory unit 18 to output estimation result information based on the 2D line drawings.
[0018] The true candidate information output unit 12 is configured to output true candidate information that includes multiple true candidates for the identifier of the part to be identified, and an indicator of the reliability of each of the multiple true candidates (e.g., rank and probability of reliability), based on multiple estimation result information output from the identifier estimation unit 10. The true candidate information output unit 12 outputs true candidate information that includes at least one true candidate for the identifier of the part to be identified, and an indicator of the reliability of each of the true candidates, based on multiple estimation result information output from the identifier estimation unit 10.
[0019] If the number of true candidate identifiers included in the true candidate information output by the true candidate information output by the true candidate information output unit 12 is one, the identifier acquisition unit 14 determines that one true candidate as the correct identifier for the component to be identified and acquires the identifier of that correct answer. In this case, the display control unit 16 causes the true candidate information, including the one true candidate, to be displayed on the AR device 2.
[0020] If the number of true candidate identifiers included in the true candidate information output by the true candidate information output unit 12 is multiple, the display control unit 16 displays the true candidate information including the multiple true candidates on the AR device 2, and the operator checks the display content of the AR device 2 and selects the correct identifier of the part to be identified from the multiple true candidates. In this case, the identifier acquisition unit 14 acquires the correct identifier selected by the operator.
[0021] The work manual selection unit 15 is configured to select and read a work manual for a part indicated by an identifier obtained by the identifier acquisition unit 14 from the work manual database 22 stored in the storage unit 18.
[0022] The display control unit 16 is configured to control the display content shown on the AR device 2 and the display content shown on the user terminal 4.
[0023] The memory unit 18 stores the learning model 20 and the work manual database 22. The work manual database 22 includes a collection of work manuals for aircraft engine assembly work, and the work manual database 22 is composed of a collection of work manuals associated with each of the multiple parts of the aircraft engine.
[0024] Figure 4 is a flowchart showing a method for supporting the assembly of aircraft engine parts using the assembly support system 100 described above. As shown in Figure 4, in S101, the worker takes photographs of the part to be identified from multiple different directions using a camera (not shown). The multiple images obtained from this photography are transmitted wirelessly or by other means from the AR device 2 to the assembly support device 6. This camera may be built into the AR device 2, or it may be a separate unit that is capable of transmitting the captured images to the AR device 2 or the assembly support device 6.
[0025] In S102, the 2D line drawing conversion unit 8 performs image processing (preprocessing) to convert each of the multiple images of the target part obtained by shooting in S101 into a 2D line drawing, thereby generating multiple 2D line drawings (for example, if a total of m images were taken in S101, one from each of m directions, then m 2D line drawings). A 2D line drawing is an image that has no background (the pixel values of the background are 0), and only the vertices and edges of the part have pixel values. Also, m is an integer of 2 or more.
[0026] In S103, the identifier estimation unit 10 receives the multiple 2D line drawings generated in S102 (multiple 2D line drawings corresponding to multiple images obtained by photographing the part to be identified from multiple directions) as input to the identifier estimation unit 10. In S103, upon receiving the multiple 2D line drawings generated in S102, the identifier estimation unit 10 outputs estimation result information from the learning model 20 for each input 2D line drawing, which includes at least one provisional candidate for the identifier of the part to be identified corresponding to the input 2D line drawing and an index relating to the confidence level of each provisional candidate. Therefore, in S103, upon receiving the multiple 2D line drawings generated in S102 as input to the learning model 20, the identifier estimation unit 10 outputs the same number of estimation result information from the learning model 20 as the number of 2D line drawings, and each of these multiple estimation result information includes at least one provisional candidate for the identifier of the part and an index relating to the confidence level of each provisional candidate.
[0027] Figure 5 shows an example of estimation result information output from the identifier estimation unit 10 for each 2D line drawing when three 2D line drawings are input to the identifier estimation unit 10. In the example shown in Figure 5, three images obtained by photographing the wing, which is the part to be identified, from three directions are converted into 2D line drawings, and the estimation result information output from the identifier estimation unit 10 for each 2D line drawing is shown when the three converted 2D line drawings are input to the identifier estimation unit 10. In the example shown in Figure 5, for each 2D line drawing input to the identifier estimation unit 10 (for each image), estimation result information is shown that includes multiple provisional candidates for the part identifier (numbers such as 0265, 0266, etc.), the rank of each of the multiple provisional candidates (the rank of the confidence level of each of the multiple provisional candidates), and the similarity of each of the multiple provisional candidates.
[0028] In S104, the true candidate information output unit 12 outputs true candidate information that includes at least one true candidate for the identifier of the part to be identified, and an indicator relating to the reliability of each true candidate (e.g., rank and probability relating to reliability), based on the multiple estimation result information output from the identifier estimation unit 10 in S103. Here, the true candidate information output unit 12 may, for example, aggregate the number of occurrences of each provisional candidate for the identifier for the multiple provisional candidates for the identifier included in the multiple estimation result information output from the identifier estimation unit in S103, and output true candidate information that includes the top 1 to nth provisional candidates (where n is an integer of 1 or more) in terms of the number of applications when the provisional candidates for the identifier are arranged in descending order of the number of occurrences, as true candidates from 1 to nth in order. Furthermore, in this case, if the multiple estimation result information output from the identifier estimation unit 10 in S103 includes multiple provisional candidates with the same number of occurrences, the ranking among the multiple provisional candidates with the same number of occurrences may be performed based on the sum of the indicators related to the reliability of the provisional candidates (for example, the sum of the probabilities of each provisional candidate). For example, among the multiple provisional candidates with the same number of occurrences, the provisional candidate with the relatively higher sum of the reliability probabilities may be designated as the true candidate with the higher reliability ranking.
[0029] In the example shown in Figure 5, the provisional candidate identifiers 0265 and 0266 for the part to be identified are included in all three estimation results based on three 2D line drawings, and 0265 and 0266 appear the most frequently, at 3 times (3 votes). However, since the sum of the similarity scores for identifier 0265 in the three 2D line drawings is 1.986, which is greater than the sum of the similarity scores for identifier 0266 in the three 2D line drawings is 1.946, identifier 0265 is ranked as the first true candidate and identifier 0266 as the second true candidate.
[0030] If the number of true candidates for identifiers included in the true candidate information output by the true candidate information output unit 12 in S104 is one, then in S105, the display control unit 16 causes the true candidate information including the one true candidate to be displayed on the AR device 2.
[0031] If the number of true candidate identifiers included in the true candidate information output by the true candidate information output unit 12 in S104 is multiple, in S105, the display control unit 16 displays the true candidate information including the multiple true candidates on the AR device 2, and the operator checks the display content of the AR device 2 and selects the correct identifier of the part to be identified from the multiple true candidates. The identifier estimation unit 10 may be configured to update the learning model 20 by performing machine learning using the correct identifier selected in S105 and the multiple 2D line drawings input to the identifier estimation unit 10 as new training data.
[0032] As shown in Figure 6, in S106, the identifier acquisition unit 14 acquires the correct identifier selected in S105 (or the identifier of the one true candidate if there is only one true candidate in the true candidate information).
[0033] In S107, the work manual selection unit 15 selects and reads the work manual for the assembly work of the part indicated by the identifier obtained in S106 from the work manual database 22 stored in the storage unit 18.
[0034] In S108, the display control unit 16 displays the relevant section of the work manual selected in S107 on the AR device 2. Regarding the display of the work manual in S108, the display control unit 16 is configured to execute landscape mode, superimposed display mode, simple display mode, and parallel display mode, as explained using Figures 7 to 10, and is configured to be switchable between these display modes.
[0035] Figure 7 shows screen E1, which is an example of the screen of AR device 2 in landscape mode. As shown in Figure 7, the landscape mode is a display mode in which the operation manual is not displayed, and the real world scenery is visible through the AR device 2. In the example shown in Figure 7, the combustor of the aircraft engine, the combustor panel, and the hand holding the combustor panel are visible as the real world scenery through the AR device 2. The "real world scenery" visible through the AR device 2 may be, for example, a real world scenery captured by the built-in camera of the AR device 2 if the AR device 2 is a smartphone or tablet, or it may be a real world scenery seen through the glasses (transparent screen) of the AR glasses if the AR device 2 is AR glasses.
[0036] Figure 8 shows screen E2, an example of the screen of AR device 2 in superimposed display mode. As shown in Figure 8, superimposed display mode is a display mode in which the relevant section of the work manual is superimposed (overlaid) on the real world scenery visible through AR device 2. In the example shown in Figure 8, the combustor of an aircraft engine, the combustor panel, and the hand holding the combustor panel are the real world scenery, and the screen shows the relevant section of the work manual for assembling the combustor panel superimposed on the real world scenery. The relevant section of the work manual displayed in superimposed display mode may simply be an arrow or something similar indicating the location and method of work, as shown in Figure 8, or it may be an indication of the relevant page in the work manual. In the example shown in Figure 8, the display control unit 16 displays a next work button 23 on AR device 2 to display the explanation of the next work in the work manual.
[0037] Figure 9 shows screen E3, an example of the AR device 2 screen in simple display mode. Note that simple display mode is a display mode that assumes AR device 2 is a smartphone or tablet device. As shown in Figure 9, simple display mode is a mode in which the relevant section of the work manual is displayed on the screen of AR device 2, and the real world scenery is not displayed on AR device 2. In the example shown in Figure 9, the relevant page of the work manual for the assembly method of the combustor panel of an aircraft engine, and the adjacent pages are displayed across the entire screen of AR device 2.
[0038] Figure 10 shows screen E4, an example of the screen of AR device 2 in parallel display mode. Note that parallel display mode is a display mode that assumes AR device 2 is a smartphone or tablet device. As shown in Figure 10, parallel display mode is a mode in which the relevant section of the work manual is displayed side by side with the real world scenery seen through AR device 2. In parallel display mode, the real world scenery seen through AR device 2 may be, for example, a real world scenery captured by the built-in camera of AR device 2 or a camera separate from AR device 2. In the example shown in Figure 10, a screen is shown in which the real world scenery captured by the camera and the relevant section of the work manual for the assembly method of combustor panel A of an aircraft engine are displayed side by side. In parallel display mode, it may also be possible to pinch the screen of AR device 2 (smartphone or tablet device) to enlarge or reduce parts of the work manual.
[0039] In S109, the display control unit 16 executes a screen sharing mode, which displays the same screen as the AR device 2 on the instructor terminal 4. The screen sharing mode is a display mode that assumes the AR device 2 is a smartphone or tablet terminal. In S109, while the screen sharing mode is being executed, the display control unit 16 can execute each of the above-mentioned landscape mode, superimposed display mode, simple display mode, and parallel display mode. Therefore, in screen sharing mode, the screen of the AR device 2 shown in Figures 7 to 10 is shared with the instructor terminal 4 and displayed on the instructor terminal 4. As shown in Figures 7 to 10, the display control unit 16 displays a mode switching button 24 on both the AR device 2 and the instructor terminal 4 to switch between the landscape mode, superimposed display mode, simple display mode, and parallel display mode. Therefore, as shown in Figure 11, in screen sharing mode, the display control unit 16 can switch the execution mode between landscape mode, superimposed display mode, simple display mode, and parallel display mode by having the instructor operate the mode switching button 24 displayed on the instructor terminal 4. The mode switching button 24 may be, for example, a drop-down list (pull-down menu), or a mode switching button 24 corresponding to each mode may be provided. In addition, in screen sharing mode, the instructor may be able to give voice instructions to the worker using the voice input / output devices provided on both the AR device 2 and the instructor terminal 4. Furthermore, in screen sharing mode, two-way information transmission may be possible by inputting text or pictures on the screen and sharing the screen between the AR device 2 and the instructor terminal 4.
[0040] The following describes the effects of the assembly support device 6 described above. According to the assembly support device 6 described above, when performing assembly work in aircraft MRO, multiple 2D line drawings obtained by capturing and converting images of the parts to be identified from multiple methods are input to the identifier estimation unit 10. The identifier estimation unit 10 then uses the learning model to output estimation result information indicating at least one provisional candidate for the parts to be identified for each image from different directions. Based on the multiple estimation result information corresponding to the multiple images from different directions, true candidate information, including the true candidate of the identifier output from the true candidate information output unit 12, can be displayed on the AR device 2. Therefore, when workers performing aircraft engine assembly work confirm at least one true candidate of the identifier included in the true candidate information displayed on the AR device 2, the number of identifier candidates is narrowed down or identified as one when determining the identifier of the parts to be identified. As a result, even workers who are not highly skilled and possess extensive knowledge of the parts can easily identify the parts, and the amount of work required for part identification can be reduced. Consequently, assembly work in aircraft MRO can be performed more easily, and the amount of work required for assembly can be reduced. Furthermore, compared to estimating identifiers based solely on images of the target component taken from a single direction, this method can achieve a higher accuracy in estimating identifiers.
[0041] Furthermore, by estimating part identifiers based on 2D line drawings, it is possible to suppress the decrease in the accuracy of identifier estimation due to the effects of shadows, etc. Also, compared to estimating identifiers based on color images, identifiers can be estimated with less data capacity.
[0042] Furthermore, the true candidate information output unit 12 aggregates the number of occurrences of each provisional candidate for each provisional candidate included in the multiple estimation result information output from the estimation result information, and outputs true candidate information that includes the provisional candidates ranked 1st to nth in terms of occurrence frequency when the provisional candidates are arranged in descending order of occurrence frequency. Therefore, the true candidate output unit can output true candidates with high reliability by considering the occurrence frequency of the provisional candidates. In addition, the operator can select an identifier from the true candidates narrowed down to 1st to nth place.
[0043] Furthermore, even if multiple estimation results include multiple tentative candidates with equal occurrences, the ranking among these candidates is based on the sum of the confidence indicators for each candidate. Therefore, even if multiple estimation results include multiple tentative candidates with equal occurrences, the true candidates for identifiers ranked from 1st to nth place can be displayed on the operator's terminal.
[0044] Furthermore, assembly work in aircraft MRO (Maintenance, Repair, and Overhaul) can be performed while referring to the work manual for the part indicated by the identifier acquired by the identifier acquisition unit 14 on the AR device 2. This makes it easy for even inexperienced workers with extensive knowledge of the parts to perform the assembly work, thereby reducing the man-hours required for assembly. In addition, by sharing the same screen between the AR device 2 and the instructor terminal 4, assembly work can be performed while receiving instructions remotely from an instructor with extensive knowledge of the parts. Therefore, even inexperienced workers can easily perform the assembly work, thereby reducing the man-hours required for assembly.
[0045] Furthermore, by operating the mode switching button 24 displayed on the instructor terminal 4, the instructor can switch between four modes—landscape mode, superimposed display mode, simple display mode, and parallel display mode—and send appropriate instructions to the worker. This effectively reduces the man-hours required for assembly work.
[0046] This disclosure is not limited to the embodiments described above, but also includes modified forms of the embodiments described above, as well as forms that combine these forms as appropriate.
[0047] For example, in the example shown using Figure 11, the display control unit 16 is shown as being configured to execute landscape mode, superimposed display mode, simple display mode, and parallel display mode. However, the display control unit 16 does not need to be configured to execute all four of these modes. For example, if the AR device 2 is a smartphone or tablet terminal, it may be configured to execute any one to four of these four modes. Also, if the AR device 2 is AR glasses or the like, the display control unit 16 may be configured to execute, for example, landscape mode and superimposed display mode. In this case, the display control unit 16 may be configured so that the landscape mode and superimposed display mode can be switched by the user operating the mode switching button 24.
[0048] Furthermore, in the embodiment described above, if there are multiple true candidate identifiers included in the true candidate information output by the true candidate information output unit 12 in S104, in S105, the display control unit 16 displays the true candidate information including the multiple true candidates on the AR device 2, and the operator checks the display content of the AR device 2 and selects the correct identifier for the part to be identified from the multiple true candidates. However, if there are multiple true candidate identifiers included in the true candidate information output by the true candidate information output unit 12 in S104, the display control unit 16 may also display the true candidate information including the multiple true candidates on the instructor terminal 4 in S105, and the instructor checks the display content of the instructor terminal 4 and selects the correct identifier for the part to be identified from the multiple true candidates.
[0049] The contents described in each of the above embodiments can be understood, for example, as follows:
[0050] (1) An assembly support device according to at least one embodiment of the present disclosure (for example, the assembly support device 6 described above) An assembly support device for assisting in the assembly of aircraft engines, If multiple images obtained by photographing the target part of the aircraft engine from multiple different directions are defined as multiple anisotropic images (for example, multiple images taken in S101 above, or multiple 2D line drawings generated in S102), then an identifier estimation unit (for example, the identifier estimation unit 10 above) is configured to receive the multiple anisotropic images as input, estimate the identifier of the target part using a learning model (for example, the learning model 20 above) that shows the relationship between the image of the part and the identifier of the part, and output estimation result information indicating at least one provisional candidate for the identifier of the target part for each of the anisotropic images, A true candidate information output unit (for example, the true candidate information output unit 12 described above) is configured to output true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on a plurality of estimation result information output from the identifier estimation unit by inputting the plurality of images with different orientations into the identifier estimation unit, A display control unit (e.g., the display control unit 16 described above) is configured to display the true candidate information on a worker's terminal (e.g., the AR device 2 described above), It is equipped with.
[0051] According to the assembly support device described in (1) above, when performing assembly work in aircraft MRO, multiple images from different directions obtained by taking images of the part to be identified from multiple methods are input to the identifier estimation unit. The identifier estimation unit can then use the learning model described above to output estimation result information indicating at least one provisional candidate for the part to be identified for each image from different directions. Based on the multiple estimation result information corresponding to the multiple images from different directions, true candidate information, including the true candidate of the identifier output from the true candidate information output unit, can be displayed on the worker's terminal. Therefore, when workers performing aircraft engine assembly work confirm at least one true candidate of the identifier included in the true candidate information displayed on the worker's terminal, the number of identifier candidates is narrowed down or identified as one when determining the identifier of the part to be identified. As a result, even workers who are not highly skilled and possess extensive knowledge of the parts can easily identify the parts, and the man-hours required for part identification can be reduced. Consequently, assembly work in aircraft MRO can be performed more easily, and the man-hours required for assembly work can be reduced. Furthermore, compared to estimating the identifier based only on an image of the part to be identified taken from one direction, a higher accuracy in identifier estimation can be achieved.
[0052] (2) In some embodiments, in the assembly support device described in (1) above, Each of the aforementioned multiple images taken from different directions is an image obtained by converting each of the multiple images of the identified part taken from multiple different directions into a two-dimensional line drawing.
[0053] According to the assembly support device described in (2) above, by estimating part identifiers based on two-dimensional line drawings, it is possible to suppress the decrease in the accuracy of identifier estimation due to the effects of shadows, etc. Furthermore, compared to estimating identifiers based on color images, identifiers can be estimated with less data capacity.
[0054] (3) In some embodiments, in the assembly support device described in (1) or (2) above, Each of the estimation result information output from the identifier estimation unit includes at least one provisional candidate for the identifier of the part to be identified, and an indicator relating to the confidence level of each of the at least one provisional candidate (e.g., rank, probability, and similarity as described above).
[0055] According to the assembly support device described in (3) above, the true candidate output unit can output a true candidate identifier that is highly reliable, taking into account the provisional candidate and an indicator related to its reliability.
[0056] (4) In some embodiments, in the assembly support device described in any of (1) to (3) above, The true candidate information output unit is configured to output true candidate information that includes the true candidates ranked from 1st to nth, respectively, for each of the multiple provisional candidates included in the multiple estimation result information output from the estimation result information, and when the provisional candidates are arranged in descending order of frequency of occurrence, the provisional candidates ranked from 1st to nth (where n is an integer of 1 or more) are each ranked from 1st to nth.
[0057] According to the assembly support device described in (4) above, the true candidate output unit can output a highly reliable true candidate identifier by considering the frequency of occurrence of the provisional candidates. In addition, the operator can select an identifier from the true candidates narrowed down from 1st to nth place.
[0058] (5) In some embodiments, in the assembly support device described in (4) above, Each of the estimation result information output from the identifier estimation unit includes at least one provisional candidate for the identifier of the part to be identified, and an index relating to the reliability of each of the at least one provisional candidate. The true candidate information output unit is configured to rank multiple provisional candidates that have the same number of occurrences based on the sum of the indicators, when the multiple estimation result information output from the estimation result information includes multiple provisional candidates that have the same number of occurrences.
[0059] According to the assembly support device described in (5) above, even if multiple estimated result information includes multiple provisional candidates with the same frequency of occurrence, the ranking of these provisional candidates with the same frequency of occurrence can be performed based on the sum of the confidence index values for each provisional candidate. Therefore, even if multiple estimated result information includes multiple provisional candidates with the same frequency of occurrence, the true candidates for identifiers ranked from 1st to nth place can be displayed on the operator's terminal.
[0060] (6) In some embodiments, in the assembly support device described in any of (1) to (5) above, The display control unit is configured to, when the true candidate information output by the true candidate information output unit includes multiple true candidates, to display at least a portion of the work manual for the assembly work related to the part indicated by the identifier selected on the worker terminal or the instructor terminal.
[0061] According to the assembly support device described in (6) above, the worker can check the work manual for the part indicated by the selected identifier on the worker's terminal. Therefore, the worker can perform the assembly work while checking the relevant section of the work manual, thereby reducing the man-hours required for assembly.
[0062] (7) In some embodiments, in the assembly support device described in any of (1) to (6) above, The display control unit is configured to execute a screen sharing mode that displays the same screen on the operator's terminal as on the supervisor's terminal.
[0063] According to the assembly support device described in (7) above, by sharing the same screen between the instructor's terminal and the worker's terminal, assembly work can be performed while receiving instructions remotely from an instructor who has extensive knowledge of the parts. Therefore, even inexperienced workers can easily perform assembly work, and the amount of work required can be reduced.
[0064] (8) An aircraft engine assembly support method according to at least one embodiment of the present disclosure is An assembly support method for assisting the assembly work of an aircraft engine, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, the candidate estimation step involves estimating the identifier of the target component using a learning model that shows the relationship between the component image and the component identifier, based on the multiple anisotropic images, and outputting estimation result information indicating at least one provisional candidate for the identifier of the target component for each of the anisotropic images. A true candidate information output step outputs true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on the set of estimation result information output for each of the disparately oriented images, The steps include: performing display control to display the aforementioned candidate information on the operator's terminal; It is equipped with.
[0065] According to the assembly support device described in (8) above, when performing assembly work in aircraft MRO, the learning model can be used to output estimation result information indicating at least one provisional candidate for the part to be identified for each of the multiple images taken from different directions by capturing images of the part to be identified from multiple directions. Then, true candidate information, including true candidate identifiers output based on the multiple estimation result information corresponding to the multiple images taken from multiple directions, can be displayed on the worker's terminal. As a result, when workers performing aircraft engine assembly work confirm at least one true candidate identifier included in the true candidate information displayed on the worker's terminal, the number of identifier candidates is narrowed down or identified as one when determining the identifier of the part to be identified. This makes it easy to identify parts even for workers who are not highly skilled and possess extensive knowledge of the parts, and reduces the man-hours required for part identification. Therefore, assembly work in aircraft MRO can be performed more easily, and the man-hours required for assembly work can be reduced. Furthermore, compared to estimating identifiers based only on images taken from one direction of the part to be identified, a higher accuracy in estimating identifiers can be achieved. [Explanation of Symbols]
[0066] 2 devices 4. Terminal for the Instructor 6. Assembly support device 8. Two-dimensional line drawing conversion unit 10 Identifier Estimation Unit 12 True Candidate Information Output Unit 14. Identifier acquisition unit 15. Operation Manual Selection Section 16 Display Control Unit 18 Memory section 20 Learning Models 22. Work Manual Database 23 Next task button 24 Mode switching buttons 72 processors 74 RAM 76 ROM 78 HDD 80 Input Interfaces 82 Output Interfaces 84 Bus 100 Assembly Support System
Claims
1. An assembly support device for assisting in the assembly of aircraft engines, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, then an identifier estimation unit is configured to receive the multiple anisotropic images as input, estimate the identifier of the target component using a learning model that shows the relationship between the component image and the component identifier, and output estimation result information indicating at least one provisional candidate for the identifier of the target component for each of the anisotropic images. A true candidate information output unit is configured to output true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on a plurality of estimation result information output from the identifier estimation unit by inputting the plurality of images with different orientations into the identifier estimation unit. A display control unit configured to display the aforementioned candidate information on a worker's terminal, Equipped with, The assembly support device is configured such that the true candidate information output unit aggregates the number of occurrences of each of the multiple provisional candidates included in the multiple estimation result information output from the identifier estimation unit, and outputs true candidate information that includes the provisional candidates ranked 1st to nth (where n is an integer of 1 or more) in terms of the number of occurrences of each provisional candidate when the provisional candidates are arranged in descending order of the number of occurrences.
2. The assembly support device according to claim 1, wherein each of the plurality of images taken from different directions is an image obtained by converting each of the plurality of images taken of the part to be identified from different directions into a two-dimensional line drawing.
3. The assembly support device according to claim 1, wherein each of the estimation result information output from the identifier estimation unit includes at least one provisional candidate for the identifier of the part to be identified, and an index relating to the reliability of each of the at least one provisional candidate.
4. Each of the estimation result information output from the identifier estimation unit includes at least one provisional candidate for the identifier of the part to be identified, and an index relating to the reliability of each of the at least one provisional candidate. The assembly support device according to claim 1, wherein the true candidate information output unit is configured to rank the multiple provisional candidates with equal occurrences based on the sum of the indicators when the multiple estimation result information output from the identifier estimation unit includes multiple provisional candidates with equal occurrences.
5. An assembly support device for assisting in the assembly work of an aircraft engine, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, then an identifier estimation unit is configured to receive the multiple anisotropic images as input, estimate the identifier of the target component using a learning model that shows the relationship between the component image and the component identifier, and output estimation result information indicating at least one provisional candidate for the identifier of the target component for each of the anisotropic images. A true candidate information output unit is configured to output true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on a plurality of estimation result information output from the identifier estimation unit by inputting the plurality of images with different orientations into the identifier estimation unit. A display control unit configured to display the aforementioned candidate information on a worker's terminal, Equipped with, The display control unit is configured to, when the true candidate information output by the true candidate information output unit includes multiple true candidates, to display at least a portion of the work manual for the assembly work relating to the part indicated by the identifier on the worker terminal or the instructor terminal, based on the identifier selected on the worker terminal or the instructor terminal, as an assembly support device.
6. The assembly support device according to claim 1, wherein the display control unit is configured to execute a screen sharing mode that displays the same screen on the operator's terminal as on the supervisor's terminal.
7. An assembly support method for assisting the assembly work of an aircraft engine, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, then a candidate estimation step is performed in which the identifier of the target component is estimated using a learning model that shows the relationship between the image of the component and the identifier of the component, based on the multiple anisotropic images, and estimation result information indicating at least one provisional candidate for the identifier of the target component is output for each of the anisotropic images. A true candidate information output step outputs true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on the set of estimation result information output for each of the disparately oriented images, The steps include: performing display control to display the aforementioned candidate information on the operator's terminal; Equipped with, An aircraft engine assembly support method, wherein in the true candidate information output step, for each of the multiple provisional candidates included in the multiple estimation result information output, the number of occurrences of each provisional candidate is tallied, and true candidate information is output that includes the provisional candidates ranked 1st to nth (where n is an integer of 1 or more) in terms of occurrence frequency when the provisional candidates are arranged in descending order of occurrence frequency, as true candidates ranked 1st to nth.
8. An assembly support method for supporting the assembly work of an aircraft engine, If multiple images obtained by photographing the target component of the aircraft engine from multiple different directions are defined as multiple anisotropic images, then a candidate estimation step is performed in which the identifier of the target component is estimated using a learning model that shows the relationship between the image of the component and the identifier of the component, based on the multiple anisotropic images, and estimation result information indicating at least one provisional candidate for the identifier of the target component is output for each of the anisotropic images. A true candidate information output step outputs true candidate information indicating at least one true candidate for the identifier of the part to be identified, based on the set of estimation result information output for each of the disparately oriented images, The steps include: performing display control to display the aforementioned candidate information on the operator's terminal; Equipped with, An aircraft engine assembly support method, wherein, in the step of performing the display control, if the true candidate information output by the true candidate information output step includes multiple true candidates, the worker terminal or the instructor terminal is used to display at least a portion of the work manual for the assembly work relating to the part indicated by the identifier on the worker terminal.