A method for generating and displaying assembly guidance information based on target detection

Abstract

The present invention relates to a method for generating and displaying assembly induction information based on target detection, comprising the following steps: defining assembly induction information, determining the relationship between each component and the assembly process of an assembly, and defining the association relationship between the assembly process and the induction information; acquiring an image of the current assembly in real time, identifying all assembled components on the current assembly through a component detection model, and reading the position information of each assembled component; confirming the assembly process according to the assembled components, and querying the corresponding induction information based on the association relationship between the assembly process and the induction information; confirming a blank area in a display screen of a display device according to the position information of each assembled component, and displaying the corresponding induction information in the blank area.

Description

Technical Field

[0001] The present invention relates to a method for generating and displaying assembly guidance information based on target detection, and belongs to the field of computer-aided intelligent manufacturing, and in particular to the field of assembly maintenance guidance and augmented reality application technology. Background Art

[0002] Assembly and maintenance are the key factors affecting assembly performance and quality. The assembly and maintenance of complex products such as aircraft and spacecraft are mainly manual and discrete assembly, which has the characteristics of many assembly operations, complex assembly process, many assembly parts and links, and long assembly cycle. In addition, mass production of personalized customization is the development trend of home appliances and other products. In mass production of personalized customization, different models of products are produced on the same production line at the same time, and assemblers need to frequently change the assembly process according to the product model. Therefore, in the assembly and maintenance of complex products and the assembly of mass customized products, the assembly difficulty of operators increases, the assembly knowledge required increases, and errors occur from time to time during assembly and maintenance. Paper and electronic manuals are widely used in product assembly and maintenance guidance, but when using paper and electronic manuals, assembly operators need to operate the equipment while manually checking the manual, which is difficult and inefficient; and attention and hands need to be frequently switched between the manual and the equipment, which is easily affected by the surrounding environment, causing assembly errors and other problems.

[0003] Augmented reality technology can directly superimpose virtual information generated by computers on physical scenes for display. However, in AR-based assembly and maintenance, how to improve human-computer interactivity, how to generate visual induction information that operators urgently need based on the assembly process of operators and products, and how to push and display the information to operators naturally and efficiently are issues that need to be addressed urgently. On the other hand, when performing AR display, if the induction information is superimposed (occluded) on useful physical scenes (such as assemblies, tools, etc.), it will hinder assembly and fail to play the role of information induction. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a method for generating and displaying assembly induction information based on target detection, which obtains the assembly process by detecting the parts on the assembly, generates corresponding induction information, selects the blank area in the display screen, and superimposes the induction information in the blank area for display.

[0005] The technical solution of the present invention is as follows:

[0006] A method for generating and displaying assembly induction information based on target detection comprises the following steps:

[0007] Define assembly induction information, determine the relationship between each component and the assembly process of the assembly, and define the association between the assembly process and the induction information;

[0008] Collect the image of the current assembly in real time, identify all the assembled parts on the current assembly through the parts detection model, and read the position information of each assembled part;

[0009] Confirm the assembly process according to the assembled parts, and query the corresponding induction information based on the association between the assembly process and the induction information;

[0010] According to the position information of each assembled component, a blank area in the display screen of the display device is confirmed, and the corresponding guidance information is displayed in the blank area.

[0011] As a preferred implementation, the step of determining the relationship between each component and the assembly process of the assembly is specifically as follows:

[0012] Identify all assembly steps for the assembly;

[0013] Defines the name set of all components that need to be assembled on the assembly in each assembly step.

[0014] As a preferred implementation, the step of defining the association relationship between the assembly process and the induction information is specifically as follows:

[0015] Define the induced information that needs to be generated after each assembly step is completed;

[0016] The induction information includes one or a combination of more than two of text information, picture information, and video information.

[0017] As a preferred implementation, the component detection model is generated through pre-training, and the pre-training steps are specifically as follows:

[0018] Corresponding to each assembly step of the assembly, a plurality of assembly images at different viewing angles are captured;

[0019] Label each component in each assembly image and put the labeled images into the training set;

[0020] The neural network-based target detection model is trained using a training set to obtain the component detection model.

[0021] As a preferred implementation, in the step of reading the position information of each assembled component, the position information is specifically the coordinates of the anchor frame vertices of each assembled component output by the component detection model in the display screen.

[0022] As a preferred implementation, the step of confirming the blank area in the display screen of the display device according to the position information of each assembled component is specifically:

[0023] Gridding the display screen, and adding grid attributes to each grid, wherein the grid attributes include grid number, grid size, grid center point coordinates and occupancy index, setting the index threshold, and comparing the occupancy index with the index threshold to determine whether the grid is occupied by the anchor box;

[0024] Calculate the occupancy index of each grid, calculate the coordinates of each grid vertex according to the grid size and the coordinates of the grid center point of each grid; calculate whether each grid vertex is in any anchor frame according to the coordinates of each anchor frame vertex in the display screen and the coordinates of each grid vertex, and determine the occupancy index of the grid according to the vertex in the anchor frame;

[0025] Calculate the unoccupied area in the display screen, traverse all the unoccupied grids in the display screen, calculate the connected area of ​​the unoccupied grid with the largest area in the display screen, and use the connected area as the blank area.

[0026] As a preferred implementation, the induction information is superimposed on a blank area for display in an AR augmented reality manner.

[0027] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating and displaying assembly induction information based on target detection as described in any embodiment of the present invention is implemented.

[0028] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for generating and displaying assembly induction information based on target detection as described in any embodiment of the present invention is implemented.

[0029] The present invention has the following beneficial effects:

[0030] 1. The present invention provides a method for generating and displaying assembly induction information based on target detection. The method obtains the assembly process by detecting the parts on the assembly, generates corresponding induction information, selects a blank area in the display screen, and superimposes the induction information in the blank area for display, thereby solving the defect in the prior art that the induction information will block the effective scene in the screen.

[0031] 2. The present invention provides a method for generating and displaying assembly induction information based on target detection, which determines assembly steps according to identified parts and defines induction information corresponding to each assembly step, and can efficiently push and display the induction information to the operator.

[0032] 3. The present invention provides a method for generating and displaying assembly induction information based on target detection. It calculates all unoccupied grids according to the position information of the anchor frame, and selects the connected area of ​​the unoccupied grid with the largest area as the blank area. It can dynamically adjust the blank area and has followability, thereby improving the display effect of the induction information. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of a method according to Embodiment 1 of the present invention;

[0034] Figure 2 An example diagram showing the gridding of a picture and the anchor frames of components in the picture in an embodiment of the present invention;

[0035] Figure 3 This is an example diagram of a grid being occupied by an anchor box in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0038] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0039] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0040] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0041] Embodiment 1:

[0042] See also Figure 1 , a method for generating and displaying assembly induction information based on target detection, comprising the following steps:

[0043] Pre-training component detection models based on target detection algorithms;

[0044] Define assembly induction information, correspond to each assembly, determine the relationship between each component in the assembly and the assembly process of the assembly, manually write or import the induction information corresponding to the assembly, and define the association relationship between the assembly process and the induction information;

[0045] The image of the current assembly is acquired in real time by an image acquisition device, and the component detection model is used to identify the image of the current assembly to perform component detection, identify all assembled components on the current assembly, and read the position information of each assembled component based on the component detection model;

[0046] According to all assembled parts, query the relationship between each part in the assembly and the assembly process of the assembly to confirm the assembly process; query the corresponding induction information based on the association relationship between the assembly process and the induction information;

[0047] According to the position information of each assembled component, a blank area in the display screen (screen) of the display device is confirmed, and corresponding guidance information is displayed in the blank area;

[0048] Determine whether the current assembly process has ended. If so, exit. If not, return to re-identify the currently assembled components on the assembly.

[0049] In one embodiment, the step of determining the relationship between each component and the assembly process of the assembly is specifically:

[0050] Determine all assembly steps of the corresponding type of assembly; for example, step i, step ii, step iii;

[0051] Define the name set of all parts that need to be assembled on the assembly in each assembly step, such as assembly step i (large gear, bearing end cover, long shaft, bearing end cover).

[0052] In one embodiment, the step of defining the association relationship between the assembly process and the induction information is specifically as follows:

[0053] Define the induced information that needs to be generated after each assembly step is completed;

[0054] For example, after assembly step i is completed, component C needs to be assembled between components A and B.

[0055] The induction information may be presented in any form of text information, picture information, or video information;

[0056] It can also be presented in a combination of text information + picture information, a combination of picture information + video information, text information + video information, or text information + picture information + video information.

[0057] In one embodiment, the component detection model is generated through pre-training, and the pre-training steps are specifically as follows:

[0058] Corresponding to each assembly step of the assembly, a number of assembly images at different viewing angles are taken. In this embodiment, 50 assembly images at different viewing angles are taken at each assembly stage.

[0059] Use the LabelImg target detection and annotation tool to label each component in each assembly image, and put the labeled assembly images into the training set;

[0060] The neural network-based target detection model is trained using a training set. In this embodiment, the target detection model adopts a YOLOV5 model to obtain the component detection model.

[0061] In one embodiment, in the step of reading the position information of each assembled component, the position information is specifically the coordinates of the anchor box vertices of each assembled component output by the component detection model in the display screen.

[0062] In one embodiment, the step of confirming the blank area in the display screen of the display device according to the position information of each assembled component is specifically:

[0063] See Figure 2 , the display screen is gridded. In this embodiment, the display screen is divided into a number of square grids; and grid attributes are added to each grid, the grid attributes including the grid number, the grid size (in this embodiment, because the grid is a square, the grid size is specifically the pixel length l of the grid), the grid center point coordinates P (x i ,y i ) and an occupancy index O. In this embodiment, the occupancy index includes 0 and 1. When O=1, it means that the grid is occupied by the anchor frame of the component, and O=0 means that the grid is not occupied by the anchor frame of the component;

[0064] Calculate the occupancy index of each grid, and calculate the coordinates of each vertex of the grid according to the grid size and grid center point coordinates of each grid, which are P1(x i -l,y i -l)、P2(x i +l,y i -l)、P3(x i -l,y i +l)、P4(x i+l,y i +l); calculate whether each vertex of the mesh is in any anchor frame according to the coordinates of each anchor frame vertex in the display screen and the coordinates of each vertex of the mesh. Since the screen mesh is small (the mesh is divided densely to improve accuracy) and the target mesh is large, in order to simplify the calculation, in this embodiment, as long as one of the vertices of the mesh is in the anchor frame, the screen mesh is considered to be occupied, that is, the occupancy index of the mesh is O=1, otherwise O=0.

[0065] like Figure 3 As shown, Figure 3 In the example, rectangle ABCD is a grid, rectangle EFGH is an anchor frame of a component, and the method for calculating the vertices of the grid in the anchor frame in this embodiment is as follows: for any screen grid vertex C(x, y), calculate g=(yy e )(x h -x e )-(xx e )(y h -y e ), when g>0, C(x, y) is located on the left side of ray EH, and when g<0, C(x, y) is located on the right side of ray EH. If C(x, y) is between ray EH and ray GF and between ray FE and ray HG, it means that C(x, y) is located in anchor frame EFGH. As long as any vertex of the screen mesh is in anchor frame EFGH, the occupation index of the screen mesh is set to O=1. If all vertices of the screen mesh are not in the anchor frame of any component, the occupation index of the screen mesh is O=0, that is, it is not occupied by the anchor frame of the component.

[0066] Calculate the unoccupied area in the display screen, traverse all unoccupied grids in the display screen, start from the first grid in the upper left corner of the display screen, and scan to the right and bottom of the display screen according to a ratio of 4:3 or 16:9 (that is, the display ratio of the display screen, and this embodiment uses a ratio of 4:3), calculate the unoccupied grid connected area with the grid as the upper left corner, and use the unoccupied screen grid connected area with the largest area in the entire display screen (the more total number of grids in the connected area, the larger the area) as a blank area, and display assembly guidance information in the blank area.

[0067] In one embodiment, the induction information is superimposed on a blank area for display in an AR augmented reality manner.

[0068] Embodiment 2:

[0069] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating and displaying assembly induction information based on target detection as described in any embodiment of the present invention is implemented.

[0070] Embodiment three:

[0071] This embodiment provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for generating and displaying assembly induction information based on target detection as described in any embodiment of the present invention is implemented.

[0072] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating and displaying assembly induction information based on target detection, characterized in that: The following steps are involved: Define assembly induction information, determine the relationship between each component and the assembly process of the assembly, and define the association between the assembly process and the induction information; Collect the image of the current assembly in real time, identify all the assembled parts on the current assembly through the parts detection model, and read the position information of each assembled part; Confirm the assembly process according to the assembled parts, and query the corresponding induction information based on the association between the assembly process and the induction information; confirming a blank area in a display screen of a display device according to position information of each assembled component, and displaying corresponding guidance information in the blank area; The component detection model is generated through pre-training, and the pre-training steps are as follows: Corresponding to each assembly step of the assembly, a plurality of assembly images at different viewing angles are captured; Label each component in each assembly image and put the labeled images into the training set; The target detection model based on the neural network is trained by using the training set to obtain the component detection model; In the step of reading the position information of each assembled component, the position information is specifically the coordinates of the anchor frame vertices of each assembled component output by the component detection model in the display screen; The step of confirming the blank area in the display screen of the display device according to the position information of each assembled component is specifically as follows: Gridding the display screen, and adding grid attributes to each grid, wherein the grid attributes include grid number, grid size, grid center point coordinates and occupancy index, setting the index threshold, and comparing the occupancy index with the index threshold to determine whether the grid is occupied by the anchor box; Calculate the occupancy index of each grid, calculate the coordinates of each grid vertex according to the grid size and the coordinates of the grid center point of each grid; calculate whether each grid vertex is in any anchor frame according to the coordinates of each anchor frame vertex in the display screen and the coordinates of each grid vertex, and determine the occupancy index of the grid according to the vertex in the anchor frame; Calculate the unoccupied area in the display screen, traverse all the unoccupied grids in the display screen, calculate the connected area of ​​the unoccupied grid with the largest area in the display screen, and use the connected area as the blank area.

2. The method for generating and displaying assembly induction information based on target detection according to claim 1, characterized in that: The step of determining the relationship between each component and the assembly process of the assembly is specifically: Identify all assembly steps for the assembly; Defines the name set of all components that need to be assembled on the assembly in each assembly step.

3. The method for generating and displaying assembly induction information based on target detection according to claim 2, characterized in that: The step of defining the association relationship between the assembly process and the induction information is specifically as follows: Define the induced information that needs to be generated after each assembly step is completed; The induction information includes one or a combination of more than two of text information, picture information, and video information.

4. The method for generating and displaying assembly induction information based on target detection according to claim 1, characterized in that: The induction information is superimposed on the blank area for display in an AR augmented reality manner.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for generating and displaying assembly induction information based on target detection as described in any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for generating and displaying assembly induction information based on target detection as described in any one of claims 1 to 4 is implemented.

Images