An ar-based hole counterbore detection method, system, device, and medium

Abstract

The present application relates to the technical field of augmented reality, in particular to a hole and counterbore detection method, system, device and medium based on AR, which realizes automatic identification of the skin and automatic acquisition of the skin information based on image matching; realizes hole position positioning through projection technology, and intuitively displays the required work flow and area, related matters needing attention on the skin, so as to achieve the effect of assisting manual hole and counterbore, and reduce quality problems caused by human negligence; compares the to-be-detected skin with the out-of-tolerance of the standard skin within the threshold range, labels the projected out-of-tolerance part, improves the detection accuracy, realizes paperless AO process control, and has high practicability and feasibility.

Description

TECHNICAL FIELD

[0001] The application relates to the field of augmented reality technology, in particular to an AR-based hole and counterbore detection method, system, device and medium. BACKGROUND

[0002] Augmented Reality, AR for short, is a new technology developed on the basis of Virtual Reality, VR for short, and aims to provide a harmonious interactive mixed environment for users through computer vision, fusion, display and other technologies

[0003] The virtual object generated by the computer is superimposed on the real scene to constitute the expansion and enhancement of the real environment, and various perception and interaction devices are integrated to provide a harmonious interactive mixed environment for users. In the scene where a large amount of information needs to be processed by the human, the perception accuracy is higher than the efficiency, and the functions mainly include warning guidance, error reporting and request.

[0004] In the process of aircraft component assembly, the work of hole making and counterbore is huge, and at present, most of the hole making and counterbore work is completed manually. In the process of manual operation, due to the negligence of personnel, deformation of the sample plate, lack of intuitive reference and other reasons, quality problems such as hole making and counterbore out-of-tolerance are caused. The quality of hole making and counterbore has become an important factor restricting high-quality and efficient component assembly. Therefore, paperless positioning of hole making and accuracy of counterbore are of great significance in aircraft manufacturing. SUMMARY

[0005] The application proposes an AR-based hole and counterbore detection method, system, device and medium to solve the quality problems such as hole making and counterbore out-of-tolerance in the prior art. The skin is automatically recognized and the skin information is automatically acquired through image matching. The hole position is positioned through projection technology, and the required work flow and area, related matters needing attention are intuitively displayed on the skin, so as to assist the manual hole making and counterbore, reduce the quality problems caused by human negligence, compare the out-of-tolerance of the skin to be detected with the standard skin within the threshold range, mark the out-of-tolerance position, improve the detection accuracy, realize the paperless AO process control, and have high practicability and feasibility.

[0006] The application specifically realizes the following contents:

[0007] An AR-based hole and counterbore detection system comprises the following steps:

[0008] Step 1: extracting the information of the skin to be detected and storing the information of the skin to be detected in the database;

[0009] Step 2: obtaining all hole position information of the skin to be detected according to the reference hole position of the skin to be detected;

[0010] Step 3: Obtain the model of the skin to be detected according to the skin image of the skin to be detected on the current fixed station acquired from the camera;

[0011] Step 4: According to the set hole position tolerance, select a projector with corresponding resolution, and project the hole position information and operation information to the surface of the skin to be detected;

[0012] Step 5: Obtain the tool information of the current tool box from the camera, select the tool required in the current step according to the operation information, and perform hole making or dimpling;

[0013] Step 6: After the hole making and dimpling are completed, compare the current skin state information acquired from the camera with the standard skin information to determine whether there is a hole position tolerance and a missed hole making, if the skin to be detected and the standard skin are within the threshold range without tolerance, then the next operation is performed, if the skin to be detected and the standard skin are within the threshold range with tolerance, then an alarm is given and the projected tolerance position is marked.

[0014] In order to better realize the present application, further, the step 2 specifically comprises the following steps:

[0015] Step 21: Obtain the rectangular information of the operation table surface according to the positioning points of the operation table surface;

[0016] Step 22: Extract the contour information of the skin to be detected on the current operation table surface according to the grid information acquired from the projector;

[0017] Step 23: Obtain all the hole position information of the skin to be detected by comparing the rectangular information obtained in step 21 and the contour information obtained in step 22 according to the reference hole position of the skin to be detected acquired from the camera.

[0018] In order to better realize the present application, further, the specific operation of the step 22 is: first, extract the feature information of the skin to be detected according to the grid information acquired from the projector, then, according to the distortion of the grid, the corresponding relationship between the projection image and the real image of the skin to be detected is obtained, the projection image is transformed until the projection image and the real image are one-to-one corresponding, and the contour information of the skin to be detected on the current operation table surface is extracted.

[0019] In order to better realize the present application, further, the step 3 specifically comprises the following steps:

[0020] Step 31: Obtain the skin image of the skin to be detected on the current fixed station from the camera;

[0021] Step 32: Binaryzation of the skin image obtained in step 31, extraction of the skin contour feature points, and obtaining a standard geometric figure;

[0022] Step 33: One-to-one matching of the geometric figure and the standard image in the database is performed, and a corresponding matched model is obtained, that is, the model of the skin to be detected.

[0023] In order to better realize the present application, further, the step 4 of selecting the corresponding resolution projector specifically includes the following operations: firstly, the lateral resolution of the projector is obtained according to the ratio of the length of the skin to be detected to the product of the set accuracy and the pixel accuracy of the skin to be detected, secondly, the longitudinal resolution of the projector is obtained according to the ratio of the width of the skin to be detected to the product of the set accuracy and the pixel accuracy of the skin to be detected, then the resolution of the projector is calculated according to the lateral resolution of the projector and the longitudinal resolution of the projector, and finally, the corresponding resolution projector is selected according to the set hole position tolerance.

[0024] In order to better realize the present application, further, the information of the skin to be detected includes the position of each step of hole and counterbore, the required tool model, and the matters needing attention of each step of hole and counterbore.

[0025] Based on the above-mentioned AR-based hole and counterbore detection method, in order to better realize the present application, further, an AR-based hole and counterbore detection system is provided, which includes a skin recognition and positioning unit, a hole position guiding unit, a tool selection unit, and a hole position quality detection and feedback unit.

[0026] The skin recognition and positioning unit is used to extract the information of the skin to be detected, and obtain all hole position information of the skin to be detected and the model of the skin to be detected.

[0027] The hole position guiding unit is used to select a corresponding resolution projector according to the set hole position tolerance, and project the hole position information and the operation information to the surface of the skin to be detected.

[0028] The tool selection unit is used to obtain the tool information of the current tool box, select the required tool of the current step according to the operation information, and perform hole or counterbore.

[0029] The hole position quality detection and feedback unit is used to compare the current skin state information obtained from the camera with the standard skin information, judge whether there is hole position tolerance and error and omission, if there is no tolerance within the threshold range between the skin to be detected and the standard skin, the next operation is performed, if there is tolerance within the threshold range between the skin to be detected and the standard skin, an alarm is given, and the projected tolerance position is marked.

[0030] In order to better realize the present application, further, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the above-mentioned AR-based hole and counterbore detection method.

[0031] In order to better realize the present application, further, a computer readable storage medium is proposed, the computer readable storage medium has a computer program stored thereon, and the processor executes the computer program to realize the AR-based hole and counterbore detection method.

[0032] The present application has the following beneficial effects:

[0033] (1) The present application realizes automatic recognition of the skin and automatic acquisition of skin information through image matching; realizes hole positioning through projection technology, and intuitively displays the required work flow and area, related matters needing attention on the skin, so as to achieve the effect of assisting manual hole and counterbore, and reduce the quality problems caused by human negligence.

[0034] (2) The present application realizes detection and feedback of the results after hole and counterbore through structured light technology and image recognition detection technology, and improves the hole and counterbore detection precision.

[0035] (3) The AR-based hole and counterbore detection system proposed by the present application has very high practicability and feasibility, can achieve paperless AO, realize real process control, and can correct errors in time. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a structural schematic diagram of the AR-based hole and counterbore detection system proposed by the present application;

[0037] Figure 2 is a positioning schematic diagram of the AR-based hole and counterbore detection system proposed by the present application;

[0038] Figure 3 is a system connection block diagram of the AR-based hole and counterbore detection system proposed by the present application;

[0039] Figure 4 is an implementation flow schematic diagram of the AR-based hole and counterbore detection system proposed by the present application;

[0040] Among them, 1 is an integrated control unit, 2 is a skin recognition and positioning unit, 3 is a hole position guiding unit, 4 is a tool selection unit, and 5 is a hole quality detection and feedback unit. DETAILED DESCRIPTION

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the described embodiments are only a part of the embodiments of the present application, and should not be considered as limiting the scope of protection. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0042] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "set", "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0043] Embodiment 1:

[0044] This embodiment proposes a detection system for AR hole and counterbore, as shown in the following steps: Figure 4

[0045] Step 1: Extract the information of the skin to be detected, and store the information of the skin to be detected in the database;

[0046] Step 2: Obtain all hole information of the skin to be detected according to the reference hole position of the skin to be detected;

[0047] Step 3: Obtain the model of the skin to be detected according to the skin image of the skin to be detected on the current fixed station obtained from the camera;

[0048] Step 4: According to the set hole position tolerance, select the corresponding resolution projector, project the hole position information and operation information to the surface of the skin to be detected;

[0049] Step 5: Obtain the tool information of the current tool box from the camera, select the tool required for the current step according to the operation information, and perform hole or counterbore;

[0050] Step 6: After the hole and counterbore are completed, compare the current skin state information obtained from the camera with the standard skin information to determine whether there is hole position tolerance and error or omission, if the skin to be detected and the standard skin are within the threshold range without tolerance, the next operation is performed, if the skin to be detected and the standard skin are within the threshold range with tolerance, an alarm is given, and the projection of the tolerance part is marked.

[0051] ​Working principle: The embodiment realizes automatic recognition of the skin and automatic acquisition of skin information based on image matching; realizes positioning of the hole site through projection technology, and intuitively displays the required work flow and area, related matters needing attention on the skin, so as to assist manual hole drilling and dimpling, reduce quality problems caused by human negligence, compare the out-of-tolerance of the skin to be detected with the standard skin within the threshold range, mark the out-of-tolerance position, improve the detection accuracy, realize paperless AO process control, and have high practicability and feasibility. The information of the skin to be detected includes the position of each step of hole drilling and dimpling, the required tool model, and the matters needing attention of each step of hole drilling and dimpling.

[0052] Embodiment 2

[0053] The embodiment 2 is based on the embodiment 1, and the step 2 is described in detail.

[0054] The step 2 specifically includes the following steps:

[0055] Step 21: Obtain the rectangular information of the operation table according to the positioning point of the operation table;

[0056] Step 22: Extract the contour information of the skin to be detected on the current operation table according to the grid information obtained from the projector;

[0057] The specific operation of the step 22 is: first, extract the feature information of the skin to be detected according to the grid information obtained from the projector, then according to the distortion of the grid, find the corresponding relationship between the projection image of the skin to be detected and the real image, transform the projection image until the projection image and the real image are one-to-one, and extract the contour information of the skin to be detected on the current operation table.

[0058] Step 23: Obtain all the hole site information of the skin to be detected by comparing the rectangular information obtained in step 21 and the contour information obtained in step 22 according to the reference hole site of the skin to be detected obtained from the camera.

[0059] The other parts of the embodiment are the same as the embodiment 1, and will not be described again.

[0060] Embodiment 3

[0061] The embodiment 3 is based on any one of the embodiments 1-2, and the step 3 is described in detail.

[0062] The step 3 specifically includes the following steps:

[0063] Step 31: Obtain the skin image of the skin to be detected on the current fixed station from the camera;

[0064] Step 32: binarizing the skin picture obtained in step 31, extracting skin contour feature points, and obtaining a standard geometric figure;

[0065] Step 33: matching the geometric figure with the standard images in the database one by one, and obtaining the corresponding matched model, which is the model of the skin to be detected.

[0066] The other parts of this embodiment are the same as any one of the above-mentioned embodiments 1-2, and thus will not be described again.

[0067] Embodiment 4:

[0068] This embodiment is based on any one of the above-mentioned embodiments 1-3, and the specific operation of selecting a corresponding resolution projector in step 4 is described in detail.

[0069] First, the lateral resolution of the projector is obtained according to the ratio of the length of the skin to be detected to the product of the set accuracy and the pixel accuracy of the skin to be detected. Second, the longitudinal resolution of the projector is obtained according to the ratio of the width of the skin to be detected to the product of the set accuracy and the pixel accuracy of the skin to be detected. Then, the resolution of the projector is calculated according to the lateral resolution of the projector and the longitudinal resolution of the projector. Finally, a corresponding resolution projector is selected according to the set hole position tolerance.

[0070] The other parts of this embodiment are the same as any one of the above-mentioned embodiments 1-3, and thus will not be described again.

[0071] Embodiment 5:

[0072] This embodiment is based on any one of the above-mentioned embodiments 1-4, and as shown in Figure 1 a detection system for AR drilling counterbore based on the above-mentioned embodiments 1-4 is proposed, which includes a skin recognition and positioning unit 2, a hole position guiding unit 3, a tool selection unit 4, and a hole position quality detection and feedback unit 5.

[0073] The skin recognition and positioning unit 2 is used to extract the information of the skin to be detected, and obtain all the hole position information of the skin to be detected and the model of the skin to be detected.

[0074] The hole position guiding unit 3 is used to select a corresponding resolution projector according to the set hole position tolerance, and project the hole position information and the operation information to the surface of the skin to be detected.

[0075] The tool selection unit 4 is used to obtain the tool information of the current tool box, select the tool required for the current step according to the operation information, and perform drilling or counterbore.

[0076] The hole site quality detection feedback unit 5 is used for comparing the current skin state information acquired from the camera with the standard skin information, judging whether there is hole site out-of-tolerance and error and omission, if the to-be-detected skin and the standard skin are within the threshold range without out-of-tolerance, then the next step is performed, if the to-be-detected skin and the standard skin are within the threshold range with out-of-tolerance, then an alarm is given and the out-of-tolerance part projected is marked.

[0077] The other parts of the embodiment are the same as any one of the above-mentioned embodiments 1-4, and thus will not be described again.

[0078] Embodiment 6:

[0079] On the basis of any one of the above-mentioned embodiments 1-5, as shown in Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 a hole and counterbore detection system based on AR is proposed.

[0080] Working principle: including an integrated control unit, a skin identification and positioning unit, a hole site guiding unit, a tool selection unit and a hole site quality detection feedback unit.

[0081] The integrated control unit includes a control computer and an integrated circuit, which is used for connecting various units to form a complete closed-loop control system. According to the process flow, each step is displayed on the interface in the form of a flowchart, and the operation schematic and matters needing attention of each step are displayed on the interface. After starting the operation process, whether the operation is completed is guided and fed back according to the flowchart, and after confirming the completion, the next process is executed until the process is terminated.

[0082] The skin identification and positioning unit includes a skin model automatic identification module and a skin automatic positioning module based on image recognition, which is used for identifying the model of the skin placed on the current work station, automatically calling out the related projection information and hole and counterbore process of the corresponding model, and projecting the related information to the skin surface through precise positioning. The system is composed of a monocular camera, a binocular camera and a projector, and a customized operation table. The monocular camera collects the image on the current operation table, extracts the characteristic value and matches it with the data information stored in the current database, identifies the corresponding model, calls out the corresponding projection information and process, and prepares for the next work. The customized operation table mainly refers to four positioning points in the four corners, which are used for the camera to acquire the position of the current operation table surface and the reference rectangle. When the skin and other parts are placed in the operation table, the binocular camera collects the current part state, measures the datum hole or characteristic position of the skin part itself, and completes the precise positioning of all hole sites according to the datum point position and the skin contour information. The distortion coefficient is calculated through the grid deformation, and the projection information is fine-tuned.

[0083] The hole site guiding unit comprises a projection display module, which is used for guiding the operator to perform the hole making and dimpling operation at the specified position. The system is composed of a high-resolution projector connected with a control computer, which calls the skin information obtained by the skin recognition positioning unit, projects the hole site information to the skin surface according to the steps, adopts different projection modes according to different types and regions of the skin information, and can mark the operation matters needing attention on the side.

[0084] The tool selection unit comprises a tool selection module, which is used for guiding the selection of the tool required in the current step, and judging whether the tool is selected correctly. In the case of taking the wrong tool, the buzzer alarm is sounded. The system is composed of an industrial camera, a guide light, a pressure sensor and an alarm. The industrial camera is used for identifying the tool model in the current tool box, and marking the tools one by one. When the process reaches the step of selecting the corresponding tool, the green light is projected on the corresponding tool according to the information of the camera system marking, and the tool in the tool box is identified again after the tool is taken manually according to the feedback of the pressure sensor. If the correct tool is taken, the guide light flashes twice and then disappears. If the wrong tool is taken, the guide light turns red and the buzzer alarm is sounded. When the use of the tool is completed, the tool is put back to the original position, the pressure sensor feeds back to the control system, and the next tool selection process is performed.

[0085] The hole site quality detection feedback unit comprises a structured light detection module and an image recognition detection module, which are used for quality detection and feedback after hole making and dimpling. The system is composed of a projector, an industrial camera and an alarm. After hole making, the industrial camera is used to collect the current skin state information and compare it with the standard skin information to determine whether there is hole site out-of-tolerance and missed hole making. If there is no out-of-tolerance within the threshold range of the standard part, the next operation can be performed. If the out-of-tolerance condition is detected, the alarm is prompted, the out-of-tolerance position is projected and marked for reminding. After dimpling, the structured light is used for depth detection, the analysis result is projected onto the skin to display, and the hole site out-of-tolerance part is marked red for reminding.

[0086] The specific steps are as follows:

[0087] Firstly, an integrated control software is constructed. The software mainly integrates the control of the computer, projector, camera and alarm. In the computer, the skin information required for the operation is extracted according to the digital model and stored in the database, and the integrated control software is used to control the process of each step.

[0088] Secondly, the skin type automatic identification positioning system is constructed. The system is composed of single and double cameras, a projector and an operating platform with fixed position and positioning points. The single camera is used to collect the skin image on the platform. After the collection, the image matching technology is used to determine the type of the skin on the platform, and the control software starts to execute the process steps required by the skin type. In the image matching, the skin image collected at the fixed position is binarized, the contour feature points of the skin are extracted, a standard geometric figure is formed, and the standard geometric figure is matched with the standard images in the database one by one to obtain the corresponding type. In the skin positioning process, the positioning points of the operating platform are used to determine the position and rectangular information of the operating platform. The double camera is used to measure the reference hole position or feature position of the skin part. The reference hole position and the contour information are used to locate all the hole position information of the current skin part. The contour information of the skin part on the current operating platform is extracted, and the rectangular information of the operating platform is used to form the feature comparison. Then, the grid information projected by the projector is used to obtain the feature information of the current skin part again. The corresponding relationship between the two-dimensional image of the skin part and the real image is obtained by the distortion of the grid. The projected image is transformed to correspond to each other.

[0089] Thirdly, the hole and dimple guiding system is constructed. The system is mainly composed of a high-resolution projector. According to the definition of hole position tolerance in the production site, the corresponding resolution projector is selected to project and guide the hole and dimple. Since the positioning accuracy of the hole is required to be high, the high-resolution projector is selected to project as much as possible. The resolution (A*B) of the projector can be selected by the following method, which is represented as follows:

[0090] A=X / T*P

[0091] B=Y / T*P

[0092] Wherein A is the resolution of the projector in the horizontal direction, B is the resolution of the projector in the vertical direction, X is the length of the object to be projected, Y is the width of the object to be projected, T is the accuracy requirement of the object projection, and P is the pixel accuracy.

[0093] Fourthly, the tool picking system is constructed. The system is composed of an industrial camera, a guide light, a pressure sensor and an alarm. After the skin is identified, the relevant information is projected onto the skin surface. At this time, the tool information in the current tool box is obtained by the industrial camera through the process guide. The guide light points to the required tool. After the tool is picked up, the pressure sensor under the tool box feeds back information to the system. The current tool information state in the tool box is obtained by the industrial camera. Whether the tool is picked correctly is judged. If it is wrong, an alarm is given. The tools in the tool box in the system are pasted on the tool surface according to the model number. Different tools are used for different purposes. They are evenly arranged in the tool box. After the binary image is extracted from the picture collected by the industrial camera, the feature points of different tools are recognized. There is a corresponding guide light on each corresponding tool. After the tool is returned, the signal is fed back to the control system by the pressure sensor. The next step of hole position information projection and tool picking guide is executed until the hole making and dimpling process is completed.

[0094] Fifthly, the hole quality detection feedback unit is constructed. The system is composed of a structured light detection module and an image recognition detection module. It mainly includes a projector and a camera. The corresponding relationship between the projector and the camera about the world coordinate system point is obtained by using the circular dot feature pseudo camera method. The internal and external parameters of the system are calculated according to Zhang Zhengyou's calibration method. After the hole is made, the current skin state information is collected by the camera. Whether there is hole position out-of-tolerance and missed manufacturing and the like is compared with the standard skin information. If there is no out-of-tolerance within the threshold range, the next operation can be performed. If the out-of-tolerance condition is detected, an alarm is given and the out-of-tolerance part is marked for reminding. After dimpling is completed, the depth is detected by using structured light. The structured light device hard triggers the projection and collection module, which is used for fast projection and collection of the preset projection pattern. The projection device and the collection device are connected by hard trigger. The analysis results are projected onto the skin to display. The hole position out-of-tolerance part is marked red for reminding.

[0095] The system proposed in the embodiment realizes paperless process control by developing flow control software to replace the traditional process flow. The skin is automatically identified and the skin information is automatically obtained based on image matching. The hole position is positioned based on projection technology. The required working process and area and related matters needing attention are intuitively displayed on the skin to assist manual hole making and dimpling and reduce quality problems caused by human negligence. The tool picking system is designed based on the combination of image recognition technology and pressure sensor, which can be applied to tool use in the same scene. The result detection and feedback after hole making and dimpling are realized based on structured light technology and image recognition detection technology. The detection accuracy can be improved by improving the hardware condition. The results show that the hole making and dimpling automatic identification detection and prompting system based on augmented reality proposed in the embodiment has very high practicability and feasibility. Paperless AO can be realized to truly control the process and errors can be corrected in time.

[0096] The other parts of this embodiment are the same as any of the above-mentioned embodiments 1-5, and thus will not be described again.

[0097] Embodiment 7:

[0098] This embodiment is based on any of the above-mentioned embodiments 1-6, and is illustrated in detail with a specific embodiment as shown in Figure 1 , Figure 2 , Figure 3 , Figure 4 .

[0099] Working principle: as shown in Figure 1 , the system includes an integrated control unit, a skin identification and positioning unit, a hole position guiding unit, a tool selection unit, and a hole quality detection and feedback unit.

[0100] Among them, the integrated control unit 1 includes a control computer and an integrated circuit, which is used to connect various units to form a complete closed-loop control system.

[0101] Among them, the skin identification and positioning unit 2 includes a skin model automatic identification module and an automatic positioning module based on image recognition, which is used to identify the skin model placed on the current work station, and automatically extract the projection information and hole and dimple process steps corresponding to the model, and then match the skin physical object and the projection information through the binocular camera and the projector, to achieve the purpose of accurately positioning the hole position.

[0102] Among them, the hole position guiding unit 3 includes a projection calibration module and a projection display module, which guides the operator to perform hole and dimple operations at the specified position according to the process steps.

[0103] Among them, the tool selection unit 4 includes a tool identification module and a tool selection module, which is used to guide the selection of the tool required for the current step, and to judge whether the tool is selected correctly. If the tool is taken incorrectly, a buzzer alarm will be sounded, and only after the tool used in the previous step is returned, the tool used in the next step will be guided to be selected.

[0104] Among them, the hole quality detection and feedback unit 5 includes a structured light detection module, an image recognition detection module, and a projection display module, which is used for quality detection and feedback after hole and dimple completion.

[0105] It includes the following steps:

[0106] (1) Build an integrated control software. The software mainly integrates the computer, projector, camera and alarm. In the computer, the skin information required for the operation is extracted from the digital model and stored in the database, and the process personnel flow each step of operation, involving the hole and dimple position of each step, the required tool type and the matters needing attention, to achieve the process control of each step, such asFigure 3 as shown.

[0107] (2) Construct a skin type automatic identification positioning system. Construct a skin type automatic identification positioning system. The system is composed of a single or double camera, a projector and a relatively fixed operating platform. The operating platform has a positioning point. The single camera is used to collect the skin image on the platform. After the collection is completed, the image matching technology is used to determine the type of the skin on the current platform, and the process steps required for the skin are started in the control software. Among them, the image matching is carried out by binarizing the skin pictures collected at fixed positions, extracting the skin contour feature points, forming a standard geometric figure, and matching one by one with the standard images in the database, and finally obtaining the corresponding matching type. In the skin positioning process, the positioning point of the operating platform is used to determine the operating platform position and the rectangular information. The double camera is used to measure the reference hole position or the feature position of the skin part. The reference hole position and the contour information are used to position all the hole position information under the current skin part. Among them, the contour information of the skin part on the current operating platform is extracted, and the rectangular information of the operating platform is formed to form a feature comparison. Then the grid information projected by the projection is used to obtain the feature information of the current skin part again. The corresponding relationship between the two-dimensional image of the skin part and the real image is obtained by the distortion of the grid, and the projected image is transformed to make it one-to-one correspondence.

[0108] (3) Construct a hole and dimple guiding system. The system is mainly composed of a high-resolution projector. According to the definition of hole position tolerance in the production site, a projector with corresponding resolution is selected to project and guide the hole and dimple. Because the positioning accuracy of the hole is required to be high, a high-resolution projector is selected for projection as much as possible. The resolution (A*B) of the projector can be selected by the following method, which is shown as follows:

[0109] A = X / T*P

[0110] B = Y / T*P

[0111] Among them, A is the resolution of the projector in the horizontal direction, B is the resolution of the projector in the vertical direction, X is the length of the object to be projected, Y is the width of the object to be projected, T is the accuracy requirement of the object projection, and P is the pixel accuracy.

[0112] Example: If a 1500mm*1000mm workpiece on the work station is to be projected, the accuracy requirement is 0.5mm, and the pixel accuracy is 1 (the pixel accuracy will change according to the projection distance), the projection resolution is as follows:

[0113] A = X / T*P = 1500 / 0.5*1 = 3000pixcel

[0114] B = Y / T*P = 1000 / 0.5*1 = 2000pixcel

[0115] That is, the resolution of the projector needs at least 3000*2000 pixcel

[0116] (4) Construct a tool picking system. The system is composed of an industrial camera, a guide light, a pressure sensor and an alarm. After the skin is identified, the relevant information is projected onto the skin surface. At this time, through the process guide, the industrial camera is used to obtain the tool information in the current tool box. The guide light is used to point to the tool needed to be used. After the tool is taken, the pressure sensor under the tool box feeds back information to the system. The industrial camera is used to obtain the current tool information state in the tool box to determine whether the tool is picked correctly. If it is wrong, an alarm is given. The tools in the tool box in the system are pasted on the tool surface according to the model number. Different types have different functions and are evenly arranged in the tool box. After the binary image collected by the industrial camera is extracted, the feature points are recognized to identify different tools. Each corresponding tool has a corresponding guide light. After the tool is returned, the pressure sensor obtains the signal feedback to the control system to perform the next step of hole position information projection and tool picking guide until the hole making and dimpling process is completed.

[0117] (5) Construct a hole quality detection feedback unit. The system is composed of a structured light detection module and an image recognition detection module, mainly including a projector and a camera. The circular dot feature pseudo camera method is used to obtain the corresponding relationship of the projector and the camera about the world coordinate system point. The internal and external parameters of the system are calculated according to Zhang Zhengyou's calibration method. After the hole is made, the camera is used to collect the current skin state information and compare it with the standard skin information to determine whether there is hole position out-of-tolerance and missed manufacturing, etc. If there is no out-of-tolerance within the threshold range of the standard part, the next operation can be performed. If the out-of-tolerance condition is detected, an alarm is given and the out-of-tolerance position is projected and marked for reminding. After dimpling, the structured light is used for depth detection. The structured light device hard trigger projection and collection module are used for fast projection and collection of the preset projection pattern. The projection device and the collection device are connected between the hard trigger. Through comparison and analysis with the digital model information, the analysis result is projected onto the skin to display. The out-of-tolerance is marked in red and an alarm is given.

[0118] The other parts of the embodiment are the same as any one of the above-mentioned embodiments 1-6, and thus will not be described again.

[0119] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change made according to the technical essence of the present application to the above embodiments falls within the protection scope of the present application.

Claims

1. An AR-based detection method for a counterbore, characterized in that, The method comprises the following steps: Step 1: extracting information of the skin to be detected and storing the information of the skin to be detected into a database; Step 2: obtaining all hole position information of the skin to be detected according to a reference hole position of the skin to be detected; The step 2 specifically comprises the following steps: Step 21: obtaining rectangular information of the operation table surface according to a positioning point of the operation table surface; Step 22: extracting contour information of the skin to be detected on the current operation table surface according to grid information obtained from a projector; Step 23: obtaining all hole position information of the skin to be detected by comparing the rectangular information obtained in the step 21 and the contour information obtained in the step 22 according to the reference hole position of the skin to be detected obtained from a camera; In the skin positioning process, the position of the operation table surface and the rectangular information are determined through the positioning point of the operation table surface, the reference hole position or the characteristic position of the skin part is measured through the binocular camera, all hole position information under the current skin part is positioned through the reference hole position and the contour information, wherein the contour information of the skin part on the current operation table surface is extracted, and the rectangular information of the operation table surface is formed as a feature for comparison, then the feature information of the current skin part is obtained again through the grid information projected by the projection, the corresponding relationship between the two-dimensional image of the skin part and the real image is obtained through the distortion of the grid, and the projected image is transformed to be one-to-one corresponding; Step 3: obtaining the model of the skin to be detected according to a skin image of the skin to be detected on the current fixed station obtained from a camera; The step 3 specifically comprises the following steps: Step 31: obtaining the skin image of the skin to be detected on the current fixed station from the camera; Step 32: binarizing the skin image obtained in the step 31, extracting skin contour feature points, and obtaining a standard geometric figure; Step 33: matching the geometric figure with the standard images in the database one by one, and obtaining the corresponding matched model as the model of the skin to be detected; The skin image on the platform is collected by using the monocular camera, the model of the skin on the current platform is determined by using the image matching technology after the collection is completed, and the process steps required for the model of the skin are started to be executed in the control software; wherein the image matching is performed by binarizing the skin image collected at the fixed position, extracting the skin contour feature points, forming a standard geometric figure, matching the standard geometric figure with the standard images in the database one by one, and finally obtaining the corresponding matched model; Step 4: selecting a projector with corresponding resolution according to the set hole position tolerance, and projecting the hole position information and the operation information to the surface of the skin to be detected; The step 4 of selecting the projector with corresponding resolution specifically comprises the following operations: firstly, obtaining the horizontal resolution of the projector according to the ratio of the length of the skin to be detected to the product of the set accuracy of the skin to be detected and the pixel accuracy, secondly, obtaining the vertical resolution of the projector according to the ratio of the width of the skin to be detected to the product of the set accuracy of the skin to be detected and the pixel accuracy, then calculating the resolution of the projector according to the horizontal resolution of the projector and the vertical resolution of the projector, and finally selecting the projector with corresponding resolution according to the set hole position tolerance; A = X / T * P; B = Y / T * P; Wherein, A is the resolution of the projector in the horizontal direction, B is the resolution of the projector in the vertical direction, X is the length of the object to be projected, Y is the width of the object to be projected, T is the accuracy requirement of the object projection, and P is the pixel accuracy. Step 5: Obtain the tool information of the current tool box from the camera, select the required tool according to the operation information, and perform hole making or dimpling; Step 6: After hole making and dimpling are completed, compare the current skin state information obtained from the camera with the standard skin information to determine whether there is hole position out-of-tolerance and missed hole making, if the to-be-detected skin and the standard skin are within the threshold range without out-of-tolerance, the next operation is performed, if the to-be-detected skin and the standard skin are within the threshold range with out-of-tolerance, an alarm is given and the out-of-tolerance position is marked.

2. The AR-based hole counterbore detection method of claim 1, wherein, The information of the to-be-detected skin includes the position of each step of hole making and dimpling, the required tool model, and the matters needing attention of each step of hole making and dimpling.

3. An AR-based hole counterbore detection system for performing an AR-based hole counterbore detection method as claimed in claim 1, characterized by, It comprises a skin identification and positioning unit (2), a hole position guiding unit (3), a tool selection unit (4), and a hole position quality detection and feedback unit (5). The skin identification and positioning unit (2) is used to extract the information of the to-be-detected skin to obtain all hole position information of the to-be-detected skin and the model of the to-be-detected skin. The hole position guiding unit (3) is used to select a projector with corresponding resolution according to the set hole position out-of-tolerance, and project the hole position information and operation information onto the surface of the to-be-detected skin. The tool selection unit (4) is used to obtain the tool information of the current tool box, select the required tool according to the operation information, and perform hole making or dimpling. The hole position quality detection and feedback unit (5) is used to compare the current skin state information obtained from the camera with the standard skin information to determine whether there is hole position out-of-tolerance and missed hole making, if the to-be-detected skin and the standard skin are within the threshold range without out-of-tolerance, the next operation is performed, if the to-be-detected skin and the standard skin are within the threshold range with out-of-tolerance, an alarm is given and the out-of-tolerance position is marked.

4. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the method of any one of claims 1-2.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the processor executes the computer program to realize the method of any one of claims 1-2.

Images