Inspection method, inspection equipment and remote assistance method based on augmented reality

Abstract

This application relates to an augmented reality-based inspection method, inspection equipment, and remote assistance method, comprising the following steps: a positioning step, obtaining a target image within a viewfinder and matching it with a preset image in a database, and determining a problem object based on the matching result, wherein the viewfinder is set on the eyepiece, and the target image is an image of the target object that falls within the viewfinder after the viewfinder is aligned with the target object; and a teaching step, obtaining a solution corresponding to the problem object and projecting teaching information corresponding to the solution on the eyepiece. This application has the advantage of enabling non-professionals to repair thermal pipelines.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and in particular to an inspection method and system based on augmented reality. Background Art

[0002] Traditional heat pipe repairs often reveal a wide variety of leak locations and causes. For example, rusted screws on flanges can lead to loose connections, aged and damaged seals can cause steam or hot water leaks, or cracks in the pipes can cause leaks. Leak inspection is a lengthy process, and different repair methods often vary depending on the leak location. This often requires on-site inspection and diagnosis by skilled professional repair personnel, making it difficult for untrained users to perform maintenance themselves. Summary of the Invention

[0003] In order to enable non-professionals to repair thermal pipelines, the present application provides an inspection method, inspection equipment and remote assistance method based on augmented reality.

[0004] In the first aspect, the present application provides an inspection method based on augmented reality, which adopts the following technical solutions:

[0005] An inspection method based on augmented reality comprises the following steps:

[0006] a positioning step, obtaining a target image within a viewfinder and matching it with a preset image in a database, and determining the problem object based on the matching result, wherein the viewfinder is set on the eyepiece, and the target image is an image of the target object that falls into the viewfinder after the viewfinder is aligned with the target object;

[0007] The teaching step is to obtain the solution corresponding to the problem object and project the teaching information corresponding to the solution on the eyepiece.

[0008] By adopting the above technical solution, the orientation of the eyepiece can be adjusted when a damage incident occurs and needs to be resolved. When in use, the eyepiece needs to be adjusted to a standard posture. For example, the viewfinder in the middle of the eyepiece needs to be located in the user's line of sight facing forward. When the user stares at the target object and turns his head, the position of each viewfinder will change, so that different viewfinders can frame the target image during the head rotation. When the target object is located in the appropriate viewfinder, the target object will be located at a specified position in the image captured by the camera. The system extracts the target object in the target image and identifies it. The identification method can use a multi-classification neural network model or other methods to determine whether the target image corresponds to the object information in the database and whether the target object is a problem object. This method can be used to traverse and identify various objects on the scene, thereby screening out problem objects. Then, the pre-stored solution for the problem object is extracted from the cloud or local data, or the solution is obtained through remote manual customer service guidance. The solution can be projected on the eyepiece in the form of a dynamic image or other forms. During the projection process, it can overlap with the central projection of the target object on the eyepiece to produce an interactive effect, such as forming a light spot falling on the target object, or the light spot moving on the image of the target object to indicate the operation steps, or directly playing the maintenance image on the eyepiece to guide the user to resolve the damage incident.

[0009] In summary, this method enables non-professionals to repair thermal pipelines, saving a lot of manpower and shortening the problem handling cycle.

[0010] Optionally, the positioning step includes:

[0011] Photograph the target object in the viewfinder and perform feature extraction;

[0012] Based on the comparison result between the preset feature information and the extracted features, the type of the target object is identified;

[0013] Based on the comparison result of the pre-recorded damage features of the recognition result and the extracted features, damage information of the target object is obtained;

[0014] Determine whether the target object is a problem object based on the damage information.

[0015] By adopting the above technical solution, the target object in the viewfinder has characteristics such as color and shape. The color is not only visible to the naked eye, but also includes infrared imaging color, etc. The shape specifically refers to the proportion of each line on the target object, the continuity of the lines and other characteristics. The system extracts this type of information from the object in the image. The system's database pre-stores preset feature information for various parts, and the type of target object is determined by comparing this feature information. After identifying the target object, the pre-recorded damage features of the target object are compared with the extracted features to determine whether the object is damaged. If damaged, the target object is a problem object. If not damaged, the target object is a normal object.

[0016] Optionally, the positioning step further includes:

[0017] Connect to the remote assistance system based on the problem object judgment result;

[0018] Taking pictures of the target object in the viewfinder and transmitting them to the remote assistance system;

[0019] Obtaining a calibration instruction issued by the remote assistance system and projecting a calibration point onto the eyepiece, wherein the calibration point corresponds to the object calibrated in the calibration instruction;

[0020] Acquire a calibration image within a viewfinder and transmit it to the remote assistance system, wherein the viewfinder is selected based on the distance between the eyepiece and the calibration object, and the calibration image is an image of the calibration object that falls within the viewfinder after the viewfinder is aligned with the calibration object. The calibration object is disassembled based on instructions from the remote assistance system;

[0021] Determine whether the marked object is a problem object based on the remote assistance system.

[0022] By adopting the above technical solution, if after traversing all parts, the problem object cannot be identified through the system's built-in database information, the user can choose to connect to the remote assistance system. The remote assistance system is remotely controlled by a professional. The professional calibrates based on the image sent to the remote assistance system. The calibration point is located on the object selected by the professional in the image. The remote assistance system sends the calibration instruction to the local inspection system. The local inspection system reflects the calibration point on the eyepiece, and the calibration point just corresponds to the calibrated object. The user can select the calibration object for secondary inspection based on the calibration point. The inspection method may include multi-angle viewing or disassembly viewing. The patent personnel on the remote assistance system use the remote image to determine whether the calibrated object is a problem object.

[0023] Optionally, the teaching step includes:

[0024] Automatically or manually connecting to an assistance system based on the problem object determination result, wherein the assistance system includes a local assistance system or a remote assistance system;

[0025] An indicator point movement trajectory is generated on the eyepiece based on the assistance system and the display module, wherein the problem object has a central projection image on the eyepiece relative to the wearer's eyes, the indicator point movement trajectory is located on the central projection of the problem object, and the indicator point movement trajectory corresponds to the operational steps of the solution to the problem object.

[0026] By adopting the above technical solution, after obtaining the problem object, the auxiliary system generates an indicator point movement trajectory on the eyepiece to indicate steps such as the disassembly sequence or disassembly direction.

[0027] Optionally, the step of aligning the viewfinder with the target object includes:

[0028] Selecting a framing frame based on the distance between the eyepiece and the target object, wherein each scale mark on the vertical scale line is centered to form a framing frame around it;

[0029] Adjust the position of the eyepiece so that the target object moves relatively to the center of the viewfinder.

[0030] By adopting the above technical solution, a framing frame is formed around each scale, corresponding to different distances between the target object and the eyepiece. By adjusting the framing frame, the image of the object in the selected framing frame can be located in the center of the captured image.

[0031] In a second aspect, the present application provides a remote assistance method based on augmented reality, which adopts the following technical solutions:

[0032] A remote assistance method based on augmented reality comprises the following steps:

[0033] Obtain remote images uploaded by inspection equipment;

[0034] Entering a calibration instruction, wherein the calibration instruction is input based on the position of a calibration object on the remote image, the calibration object having a central projection image on the eyepiece relative to the wearer's eye, and the calibration point corresponding to the calibration instruction is located on the calibration object on the central projection image;

[0035] outputting the calibration instruction to the inspection device;

[0036] Re-acquire the remote image uploaded by the inspection device and determine whether the object corresponding to the viewfinder in the remote image is the problem object;

[0037] Outputting the judgment result to the inspection device;

[0038] Re-acquire the remote image uploaded by the inspection device;

[0039] Entering an assistance instruction, wherein an object corresponding to the viewfinder on the remote image is a problem object, the problem object has a central projection image relative to the wearer's eyes on the inspection device, and the assistance instruction includes a movement trajectory of an indicator point located on the central projection of the problem object on the inspection device, the movement trajectory of the indicator point corresponding to an operation step of a solution to the problem object;

[0040] The assistance instruction is output to the inspection device.

[0041] By adopting the above technical solution, the inspection equipment captures parts on-site and uploads the captured images to the remote assistance system. The remote assistance system then inputs calibration instructions, which are input by a professional based on the location of the calibration object in the remote image. After the remote assistance system transmits the calibration instructions to the inspection equipment, a calibration point corresponding to the calibration object is displayed on the inspection equipment's eyepiece. When the maintenance personnel performs an operation and the inspection equipment transmits the image to the remote assistance system, the professional determines whether the calibration object in the image is the problem object. If so, the professional inputs assistance instructions to generate a moving trajectory of an indicator point on the inspection equipment's eyepiece as a solution. The solution can be projected onto the eyepiece in the form of a dynamic image or other form. The projection process can overlap with the center projection of the target object in the eyepiece, creating an interactive effect. For example, a light spot falls on the target object, a light spot moves across the target object's image to illustrate the operation steps, or a maintenance video is directly displayed on the eyepiece to guide the user in resolving the damage.

[0042] In a third aspect, the present application provides an augmented reality-based inspection device for the above-mentioned inspection method, which adopts the following technical solution:

[0043] An inspection device comprising:

[0044] A frame having a longitudinal reference surface and a transverse reference surface perpendicular to each other, wherein the transverse reference surface is used as a plane for looking straight ahead, and the longitudinal reference surface is used as a plane for looking straight ahead during pitching;

[0045] an eyepiece mounted on a frame and having longitudinal scale lines on a longitudinal reference surface, wherein the longitudinal scale lines are provided with calibration scales on a transverse reference surface;

[0046] A display module is provided on the frame for forming an image on the eyepiece;

[0047] A camera is arranged on the frame, and the camera can measure the distance of objects in the central framing area. The optical axis of the camera lens is parallel to the longitudinal reference plane. The projection of the camera lens optical axis on the longitudinal reference plane forms an acute angle with the projection of the eyepiece mirror optical axis on the longitudinal reference plane. The intersection position of the projection of the line of sight passing through the scale on the longitudinal scale line on the longitudinal reference plane and the projection of the camera lens optical axis on the longitudinal reference plane is the longitudinal landing point position, and adjacent landing point positions are equidistant.

[0048] Currently, devices capable of capturing eye direction and emitting lasers to allow third parties to track the gaze of a person are available on the market. However, these devices are complex and expensive, making them difficult to commercialize. Others determine the actual position of the eye relative to surrounding objects by detecting head tilt and displacement, but this method requires high signal latency and measurement accuracy. The purpose of all these eye-tracking techniques is to simulate the field of view of the eye, ensuring that the image projected by the AR optical engine on the eyepiece closely matches the image projected from the center of the eyepiece. Other technologies aim to circumvent eye-tracking to achieve high-precision AR. These computer vision-based AR technologies use computer vision methods to establish a mapping between the real world and the screen, allowing drawn graphics or 3D models to appear on the screen as if they were attached to real objects. In other words, the goal is to find a plane in the real world, map this 3D plane onto a 2D plane, and then draw the graphics on this plane. This method usually requires AI chips to perform high-speed calculations through algorithms such as neural network models, which requires relatively high computing power and is relatively costly. Obviously, this is unrealistic for inspection equipment with relatively simple functional requirements.

[0049] By adopting the above technical solution, the frame is used to support and fix other structural components, and the user has requirements for the position of the frame when using the frame, and the frame needs to be adjusted to a standard posture so that when the user looks straight ahead, the horizontal plane is on the horizontal reference plane, and when the user looks up or down while staring at the calibration scale, the pitch plane of the head is on the longitudinal reference plane.

[0050] Since people are not sensitive to the rotation angle of their heads, but are relatively sensitive to distance perception, this solution comprehensively utilizes variable sight lines, adjustable scale selection, and the fixed shooting direction of the camera relative to the eyepiece. The viewfinder corresponding to the scale is selected based on the person's estimation of the distance. The object in the viewfinder will correspond to the shooting center of the camera. Therefore, there is no need to waste computing power to traverse and identify all parts of the photograph to obtain the object, which also improves the accuracy of the discrimination. The camera measures the distance between the object and the eyepiece by measuring the distance of the object in the central framing area, thereby establishing a mapping relationship between the camera image and the central projection image on the eyepiece. When the remote assistance system marks a point on the camera image, it can accurately map it to the corresponding position of the central projection on the eyepiece.

[0051] In summary, this device, assisted by simple component pairs with set matching relationships, can be effectively used for the above-mentioned inspection method, with low requirements for computing power and low production costs.

[0052] Optionally, a gyroscope sensor is provided on the frame to detect the deflection angle of the eyepiece.

[0053] By adopting the above technical solution, the gyroscope sensor measures the deflection angle of the eyepiece and, based on this deflection, determines the position of the viewfinder on the horizontal plane. The user can simply move the viewfinder corresponding to the scale onto the horizontal plane, allowing the system to directly determine the viewfinder to use, thereby reducing the need for camera distance measurement. Of course, this can also be combined with the camera's distance measurement to more accurately determine the user's selected viewfinder.

[0054] Optionally, a microphone is further included, and the microphone is arranged on the frame.

[0055] By adopting the above technical solution, remote assistance personnel can communicate with on-site personnel through a microphone to assist in problem solving.

[0056] Fourthly, the present application provides an augmented reality-based remote assistance system, which adopts the following technical solutions:

[0057] A remote assistance system based on augmented reality, comprising:

[0058] The first image module is used to obtain remote images uploaded by the inspection device;

[0059] a calibration instruction input module, configured to input a calibration instruction, wherein the calibration instruction is input based on the position of a calibration object on a remote image, the calibration object having a central projection image on the eyepiece relative to the wearer's eye, and the calibration point corresponding to the calibration instruction is located on the calibration object on the central projection image;

[0060] A calibration instruction output module, used for outputting the calibration instruction to the inspection device;

[0061] The second image module is used to obtain the remote image uploaded by the inspection device again and determine whether the object corresponding to the viewfinder in the remote image is a problem object;

[0062] A judgment output module, configured to output the judgment result to an inspection device;

[0063] The third image module is used to obtain the remote image uploaded by the inspection device again;

[0064] an assistance instruction input module, configured to input assistance instructions, wherein the object corresponding to the viewfinder on the remote image is a problem object, the problem object has a central projection image relative to the wearer's eyes on the inspection device, and the assistance instruction includes a movement trajectory of an indicator point on the central projection of the problem object on the inspection device, the movement trajectory of the indicator point corresponding to an operation step of a solution to the problem object;

[0065] The assistance instruction output module is used to output the assistance instruction to the inspection device.

[0066] In a fifth aspect, the present application provides a host computer that adopts the following technical solution:

[0067] A host comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the above-mentioned inspection method.

[0068] In a sixth aspect, the present application provides a readable storage medium, which adopts the following technical solution:

[0069] A readable storage medium stores a computer program that can be loaded by a processor and execute the above inspection method.

[0070] In a seventh aspect, the present application provides a host computer, which adopts the following technical solution:

[0071] A host comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the remote assistance method as described above.

[0072] In an eighth aspect, the present application provides a readable storage medium, which adopts the following technical solution:

[0073] A readable storage medium stores a computer program that can be loaded by a processor and execute the above remote assistance method. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] FIG1 is a schematic diagram illustrating an inspection device based on augmented reality according to a certain embodiment of the present application, in which only the frame, eyepiece and camera of the inspection device are functionally illustrated.

[0075] Figure 2 A flowchart illustrating an augmented reality-based inspection method according to an embodiment of the present application.

[0076] Figure 3 A flowchart illustrating steps for a user to adjust a viewfinder to align with a target object according to an embodiment of the present application.

[0077] Figure 4 A flowchart illustrating the positioning steps of an embodiment of the present application.

[0078] Figure 5 A flowchart illustrating the teaching steps of an embodiment of the present application.

[0079] Figure 6 Used to illustrate a remote assistance method based on augmented reality according to a certain embodiment of the present application. Description of the drawings:

[0081] 1. Frame; 2. Eyepiece; 3. Horizontal reference surface; 4. Vertical reference surface; 5. Scale line; 6. Viewfinder; 7. Camera. DETAILED DESCRIPTION

[0082] The present application will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0083] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the inventive concepts. Some of the figures in the drawings of the present disclosure, which are part of this specification, represent structures and devices in block diagram form to avoid making the disclosed principles complicated and obscure. For the sake of clarity, not all features of an actual implementation are necessarily described. In addition, the language used in this disclosure has been selected primarily for readability and instructional purposes and may not have been selected to delineate or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such inventive subject matter. References in this disclosure to "one embodiment" or "an embodiment" mean that the specific features, structures or characteristics described in conjunction with that embodiment are included in at least one embodiment, and multiple references to "one embodiment" or "an embodiment" should not be understood to necessarily all refer to the same embodiment.

[0084] Computer vision-based AR uses computer vision methods to establish a mapping relationship between the real world and the screen, so that drawn graphics or 3D models can be displayed on the screen as if they were attached to real objects. In essence, it is to find an attached plane in the real scene, and then map the plane in this three-dimensional scene to the two-dimensional screen, and then draw the graphics on this plane.

[0085] In practice, engineering doesn't require dynamic imagery for interaction, nor does it require the creation of stereoscopic images in real life, as with HoloLens. Nor does it require a virtual assistant. In civilian scenarios, virtual images typically use computer vision to establish a mapping between the real world and the screen, allowing the desired graphics or 3D models to appear as if attached to a real object on a flat surface, such as a screen, wall, or floor. However, in industrial scenarios, finding a flat surface makes it difficult to accurately render virtual images in reality. Furthermore, current AR glasses generally have a very low viewing angle, resulting in relatively poor augmented reality results.

[0086] The present application discloses an augmented reality-based inspection device. Referring to FIG1 , the inspection device includes at least a frame, an eyepiece, a display module, and a camera, and may also include other auxiliary components such as a gyroscope sensor and a microphone.

[0087] A longitudinal reference plane and a transverse reference plane perpendicular to each other are formed on the frame. The transverse reference plane is used as a horizontal plane, and the longitudinal reference plane is used as a plane for the line of sight when pitching or lowering. Specifically, in FIG1 , the transverse reference plane is shown as a dotted line arranged horizontally, and the longitudinal reference plane is shown as a dotted line arranged perpendicular to the transverse reference plane. The eyepiece is set on the frame and has longitudinal scale lines located on the longitudinal reference plane. The longitudinal scale lines are provided with calibration scales on the transverse reference plane. Specifically, the longitudinal scale lines are specifically shown as scale lines arranged longitudinally on the eyepiece of FIG1 . There is a corresponding viewfinder around each longitudinal scale line. The viewfinder is used by the user to make the object that needs to be focused on correspond to the longitudinal scale line when adjusting the posture of the frame. Specifically, the viewfinder is specifically shown as a dotted frame surrounding each scale line in FIG1 . The frame is used to support and fix other structural components, and the user has requirements for the position of the frame when using it. The frame needs to be adjusted to a standard posture so that when the user looks straight ahead, the horizontal plane is on the horizontal reference plane, and when the user looks up or down at the calibration scale, the pitch plane of the head is on the vertical reference plane.

[0088] The display module is set on the frame for imaging on the eyepiece. It requires data fed back by the camera so that the imaging of the display module on the eyepiece can match the image of the external object projected on the eyepiece relative to the center of the eyeball. For example, if there is an apple on the table and it passes through a viewfinder above the eyepiece, then if the display module needs to display a cursor on the apple, the display module must make the cursor fall accurately on the apple when displaying the cursor. In different embodiments, the display module can be an optical machine, a transparent display screen set on the eyepiece, or other devices. Any device that can accurately generate a corresponding image at a corresponding position in the eyepiece based on an input signal can be used as a display module.

[0089] To this end, the camera is arranged on the frame, and the optical axis of the camera lens is parallel to the longitudinal reference plane. The projection of the camera lens optical axis on the longitudinal reference plane forms an acute angle with the projection of the eyepiece mirror optical axis on the longitudinal reference plane. The intersection of the projection of the line of sight passing through the scale on the longitudinal scale line on the longitudinal reference plane and the projection of the camera lens optical axis on the longitudinal reference plane is the longitudinal landing point position, and adjacent landing points are equidistant.

[0090] Since people are not sensitive to the rotation angle of their heads, but are relatively sensitive to distance perception, this solution comprehensively utilizes variable sight lines, adjustable scale selection, and the fixed shooting direction of the camera relative to the eyepiece. The viewfinder corresponding to the scale is selected based on the person's estimation of the distance. The object in the viewfinder will correspond to the shooting center of the camera. Therefore, there is no need to waste computing power to traverse and identify all parts of the photograph to obtain the object, which also improves the accuracy of the discrimination. The camera measures the distance between the object and the eyepiece by measuring the distance of the object in the central framing area, thereby establishing a mapping relationship between the camera image and the central projection image on the eyepiece. When the remote assistance system marks a point on the camera image, it can accurately map it to the corresponding position of the central projection on the eyepiece.

[0091] Reference Figure 1b For example, when a person looks at an infinite distance, he or she stares at the calibration scale, and then the person's line of sight and the extended line of the camera's shooting intersect at infinity. Figure 1c, when a person looks at something through a scale above the calibration scale, the person's line of sight intersects with the camera optical axis at a close distance; when a person looks at something through a scale above the calibration scale, the person's line of sight intersects with the camera optical axis at an even closer distance. By setting the longitudinal landing point positions at equal intervals, it is helpful for users to memorize the preset distances. For example, with an interval of 0.2m, there are five consecutive longitudinal landing points located at 0.4m, 0.6m, 0.8m, 1.0m, and 1.2m from the eyepiece. These longitudinal landing points correspond to five consecutive adjacent longitudinal scale lines and viewfinders, respectively. Optionally, the landing point distance can be marked next to the viewfinder on the eyepiece to assist users in selecting a suitable viewfinder.

[0092] In some embodiments, the inspection device further includes a gyroscope sensor mounted on the frame for detecting the deflection angle of the eyepiece. The gyroscope sensor measures the deflection angle of the eyepiece and, based on this deflection, determines the position of the viewfinder on the horizontal plane. The user can move the viewfinder corresponding to the corresponding scale onto the horizontal plane, allowing the system to directly determine the viewfinder in use, thereby reducing the need for camera distance measurement. Of course, this can also be combined with the camera's distance measurement to more accurately determine the user's selected viewfinder.

[0093] In some embodiments, the inspection device further includes a microphone, which is disposed on the frame. The microphone cooperates with the enhanced image on the eyepiece to improve the ability of people to interact with the scene.

[0094] Augmented reality can be applied in a variety of industrial scenarios. While this example uses traditional thermal pipe repair as an example, the use of this detection method is not limited to this specific scenario. In traditional thermal pipe repair, the location and cause of leaks in heat pipes are often variable. For example, rusted screws on flanges can cause loose connections, or aging and damage to sealing rings can lead to steam or hot water leaks, or cracks in the pipes can cause leaks. Leak detection is a lengthy process, and different repair methods often vary depending on the leak location. This typically requires on-site inspection and diagnosis by skilled professional maintenance personnel, making it difficult for users without extensive training to perform maintenance themselves.

[0095] Accordingly, refer to Figure 2 The present application also discloses an augmented reality-based inspection method that can be applied to the above-mentioned inspection equipment, including a positioning step and a teaching step:

[0096] S100. Positioning step: obtaining a target image within a viewfinder and matching it with a preset image in a database, and determining a problem object based on the matching result, wherein the viewfinder is set on the eyepiece, and the target image is an image of the target object that falls into the viewfinder after the viewfinder is aligned with the target object.

[0097] Here, a problem object refers to an object with defects or damage that can cause a leak in a thermal pipeline, such as a broken seal or a rusted bolt. In other scenarios, a problem object is the object being searched for. It's important to note that the naming of a problem object doesn't necessarily imply that the object has a problem; it's simply a name for the object being searched for, and doesn't limit the type of object being searched. A target object refers to the object to be inspected during the search for the problem object. The selection of this object is subjective and can be arbitrary. A preset image in the database refers to standard image information pre-stored in the database, such as an intact seal or bolt. This information can include information such as size, color, and shape. Matching refers to the process of matching the standard image information in the database with the information in the acquired target image. The matching information can include, but is not limited to, size, color, and shape. During the matching process, the degree of match between each piece of information is scored. When the combined score exceeds a preset threshold, the object is considered the same.

[0098] When a damage event occurs and needs to be resolved, the orientation of the eyepiece can be adjusted. When in use, the eyepiece needs to be adjusted to a standard posture. For example, the viewfinder in the middle of the eyepiece needs to be located in the user's line of sight facing forward. When the user stares at the target object and turns his head, the position of each viewfinder will change, so that different viewfinders can frame the target image during the head rotation. When the target object is located in the appropriate viewfinder, the target object will be located at the specified position in the image captured by the camera. The system extracts the target object in the target image and identifies it. The identification method can use a multi-classification neural network model or other methods to determine whether the target image corresponds to the object information in the database and whether the target object is a problem object.

[0099] Specifically, refer to Figure 3 The steps for the user to adjust the viewfinder to align with the target object include:

[0100] Selecting a framing frame based on the distance between the eyepiece and the target object, wherein each scale mark on the vertical scale line is centered to form a framing frame around it;

[0101] Adjust the position of the eyepiece so that the target object is located on the longitudinal reference plane, and then move the target object relatively along the longitudinal reference plane to the center of the viewfinder.

[0102] Each scale is surrounded by a framing box, corresponding to the distance between the target object and the eyepiece. By adjusting the framing box, the image of the object in the selected framing box can be placed in the center of the captured image.

[0103] As an example, in some embodiments, referring to Figure 4 , the positioning steps include:

[0104] S101. Photograph the target object in the viewfinder and perform feature extraction.

[0105] S102. Based on the comparison result between the preset feature information and the extracted features, identify the type of the target object.

[0106] S103. Obtain damage information of the target object based on the comparison result between the pre-recorded damage features of the recognition result and the extracted features.

[0107] S104. Determine whether the target object is a problem object based on the damage information.

[0108] The target object in the viewfinder has characteristics such as color and shape. The color specifically refers to not only the color visible to the naked eye, but also the color of infrared imaging, etc. The shape specifically refers to the proportion of each line on the target object, the degree of continuity of the lines, and other characteristics. The system extracts this type of information from the object in the image. The system's database pre-stores preset feature information for various parts, and the type of target object is determined by comparing this feature information. After identifying the target object, the pre-recorded damage features of the target object are compared with the extracted features to determine whether the object is damaged. If damaged, the target object is a problem object. If not damaged, the target object is a normal object.

[0109] Optionally, the preset feature information can be pre-stored in a local memory on the inspection device and retrieved via the controller, or it can be stored in the cloud and retrieved using a wireless communication device provided on the inspection device. The retrieval of data from the local memory and the cloud can be implemented based on corresponding strategies. For example, the controller may prioritize the local memory. When no relevant data in the local memory matches the characteristics of the target object, data is dispatched from the cloud.

[0110] In addition, in some embodiments, the features of the target object may be uploaded to a cloud server, which matches the features with the features using information in a cloud database and returns the matching results to the inspection device.

[0111] Since the storage data of the local database and the cloud database is limited, and the working conditions in different sites are also complex and changeable, in some embodiments, continue to refer to Figure 4 , the positioning step also includes the following steps:

[0112] S105. Connect to the remote assistance system based on the problem object judgment result.

[0113] If, after searching all parts, the problem object cannot be identified using the system's built-in database, the user can connect to a remote assistance system controlled remotely by a professional. This can be done automatically or manually.

[0114] S106. Photograph the target object in the viewfinder and transmit the photograph to the remote assistance system.

[0115] The professional performs calibration based on the image sent to the remote assistance system, and the calibration points are located on the objects selected by the professional in the image.

[0116] S107. Obtain a calibration instruction issued by the remote assistance system, and project a calibration point onto the eyepiece, wherein the calibration point corresponds to the object calibrated in the calibration instruction.

[0117] The remote assistance system sends the calibration instructions to the local inspection system, and the local inspection system reflects the calibration points on the eyepiece, and the calibration points just correspond to the calibrated object.

[0118] S108. Obtain a calibration image within the viewfinder and transmit it to the remote assistance system, wherein the viewfinder is selected based on the distance between the eyepiece and the calibration object, and the calibration image is an image of the calibration object that falls into the viewfinder after the viewfinder is aligned with the calibration object. The calibration object is disassembled based on the instructions of the remote assistance system.

[0119] S109. Determine whether the marked object is a problem object based on the remote assistance system.

[0120] Users can select the calibration object for secondary inspection based on the calibration points. The inspection method can include multi-angle viewing or disassembly viewing. The patent personnel on the remote assistance system use remote images to judge whether the calibration object is a problem object.

[0121] S200. Teaching step: obtaining a solution corresponding to the problem object, and projecting teaching information corresponding to the solution on the eyepiece.

[0122] After identifying the problem object, a pre-stored solution for the problem object is extracted from a cloud database or a local database, or a solution is obtained through remote manual customer service guidance. The solution can be projected on the eyepiece in the form of a dynamic image or other form. During the projection process, it can overlap with the central projection of the target object on the eyepiece to produce an interactive effect, such as forming a light spot falling on the target object, or the light spot moving on the image of the target object to indicate the operation steps, or directly playing a maintenance image on the eyepiece to guide the user to resolve the damage incident.

[0123] As an example, in some embodiments, referring to Figure 5 , the teaching steps include:

[0124] S201. Automatically or manually connect to the assistance system based on the problem object judgment result, wherein the assistance system includes a local assistance system or a remote assistance system;

[0125] S202. Generate an indicator point movement trajectory on the eyepiece based on the assistance system and the display module, wherein the problem object has a central projection image on the eyepiece relative to the wearer's eyes, the indicator point movement trajectory is located on the central projection of the problem object, and the indicator point movement trajectory corresponds to the operation steps of the problem object solution.

[0126] After obtaining the problem object, the assisting system generates a moving track of the indicator point on the eyepiece to indicate the disassembly sequence or direction. In addition, the microphone on the inspection device can also be used to provide audio instructions to assist the user in the moving track of the indicator point.

[0127] This application provides a remote assistance method based on augmented reality, which is used for remote assistance equipment. Figure 6 , including the following steps:

[0128] S301. Get remote images uploaded by the inspection device;

[0129] S302. Entering a calibration instruction, wherein the calibration instruction is input based on the position of a calibration object on the remote image, the calibration object having a central projection image on the eyepiece relative to the wearer's eye, and the calibration point corresponding to the calibration instruction is located on the calibration object on the central projection image;

[0130] S303. Output the calibration instruction to the inspection device;

[0131] S304. The remote image uploaded by the inspection device is obtained again, and it is determined whether the object corresponding to the viewfinder in the remote image is the problem object;

[0132] S305. Output the judgment result to the inspection device;

[0133] S306. Get the remote image uploaded by the inspection device again;

[0134] S307. Entering assistance instructions, wherein the object corresponding to the viewfinder on the remote image is the problem object, the problem object has a central projection image relative to the wearer's eye on the inspection device, and the assistance instructions include a movement trajectory of an indicator point on the central projection of the problem object on the inspection device, the indicator point movement trajectory corresponding to the operation steps of solving the problem object;

[0135] S308. Output the assistance instruction to the inspection device.

[0136] The inspection device takes photos of the parts on site and uploads them to the remote assistance system. The remote assistance system then enters calibration instructions, which are input by a professional based on the location of the calibration object on the remote image. After the remote assistance device transmits the calibration instructions to the inspection device, a calibration point will be displayed on the inspection device's eyepiece at the location corresponding to the calibration object. When the maintenance personnel perform an operation and the inspection device transmits the image to the remote assistance device, the professional determines whether the calibration object on the image is the problem object. If so, the professional enters assistance instructions to generate a moving trajectory of an indicator point on the inspection device's eyepiece as a solution. The solution can be projected onto the eyepiece in the form of a dynamic image or other form. The projection process can overlap with the center projection of the target object on the eyepiece, creating an interactive effect. For example, a light spot falls on the target object, or a light spot moves across the image of the target object to illustrate the operation steps, or a maintenance image is directly played on the eyepiece to guide the user in resolving the damage.

[0137] The present application also discloses an augmented reality-based remote assistance system, including:

[0138] The first image module is used to obtain remote images uploaded by the inspection device;

[0139] a calibration instruction input module, configured to input a calibration instruction, wherein the calibration instruction is input based on the position of a calibration object on a remote image, the calibration object having a central projection image on the eyepiece relative to the wearer's eye, and the calibration point corresponding to the calibration instruction is located on the calibration object on the central projection image;

[0140] A calibration instruction output module, used for outputting the calibration instruction to the inspection device;

[0141] The second image module is used to obtain the remote image uploaded by the inspection device again and determine whether the object corresponding to the viewfinder in the remote image is a problem object;

[0142] A judgment output module, configured to output the judgment result to an inspection device;

[0143] The third image module is used to obtain the remote image uploaded by the inspection device again;

[0144] an assistance instruction input module, configured to input assistance instructions, wherein the object corresponding to the viewfinder on the remote image is a problem object, the problem object has a central projection image relative to the wearer's eyes on the inspection device, and the assistance instruction includes a movement trajectory of an indicator point on the central projection of the problem object on the inspection device, the movement trajectory of the indicator point corresponding to an operation step of a solution to the problem object;

[0145] The assistance instruction output module is used to output the assistance instruction to the inspection device.

[0146] The present application also discloses a host computer comprising a memory and a processor, wherein the memory stores a computer program capable of being loaded by the processor and executing the aforementioned augmented reality-based inspection method. The method of this embodiment can be executed by a control device disposed on the host computer, which can be a Wi-Fi-enabled mobile phone, tablet computer, laptop computer, or other electronic device. The method of this embodiment can also be executed directly by the CPU (central processing unit) of the electronic device.

[0147] The present application also discloses a host computer including a memory and a processor, wherein the memory stores a computer program capable of being loaded by the processor and executing the aforementioned augmented reality-based remote assistance method. The method of this embodiment can be executed by a control device disposed on the host computer. The current device can be a mobile phone, tablet computer, laptop computer, or other electronic device with Wi-Fi capability. The method of this embodiment can also be executed directly by the CPU (central processing unit) of the electronic device.

[0148] The present application also discloses a readable storage medium, which stores a computer program that can be loaded by a processor and execute the above-mentioned inspection method based on augmented reality. Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, including a number of instructions to enable a device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0149] The embodiment of the present application also discloses a readable storage medium, which stores a computer program that can be loaded by a processor and execute the above-mentioned remote assistance method based on augmented reality. Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, an optical disk) as above, including a number of instructions to enable a device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0150] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. An inspection method based on augmented reality, characterized in that: The following steps are involved: a positioning step, obtaining a target image within a viewfinder and matching it with a preset image in a database, and determining the problem object based on the matching result, wherein the viewfinder is set on the eyepiece, and the target image is an image of the target object that falls into the viewfinder after the viewfinder is aligned with the target object; The positioning steps include: Photograph the target object in the viewfinder and perform feature extraction; Based on the comparison result between the preset feature information and the extracted features, the type of the target object is identified; Based on the comparison result of the pre-recorded damage features of the recognition result and the extracted features, damage information of the target object is obtained; Determine whether the target object is a problem object based on damage information; Teaching step: obtaining the solution corresponding to the problem object and projecting the teaching information corresponding to the solution on the eyepiece; Also included is an augmented reality-based inspection device for the above-mentioned inspection method, the inspection device comprising: A frame having a longitudinal reference surface and a transverse reference surface perpendicular to each other, wherein the transverse reference surface is used as a plane for looking straight ahead, and the longitudinal reference surface is used as a plane for looking straight ahead during pitching; The eyepiece is mounted on the frame and has longitudinal scale lines on the longitudinal reference surface. The longitudinal scale lines are provided with calibration scales on the transverse reference surface. Each longitudinal scale line is surrounded by a corresponding viewfinder, which is used by the user to align an object of interest with the longitudinal scale line when adjusting the frame posture. A display module is provided on the frame for imaging on the eyepiece; it requires data fed back by the camera so that the image of the display module on the eyepiece can match the image of an external object projected on the eyepiece relative to the center of the eyeball; A camera is arranged on the frame, the optical axis of the camera lens is parallel to the longitudinal reference plane, the projection of the camera lens optical axis on the longitudinal reference plane forms an acute angle with the projection of the eyepiece mirror optical axis on the longitudinal reference plane, the intersection position of the projection of the line of sight passing through the upper scale of the longitudinal scale line on the longitudinal reference plane and the projection of the camera lens optical axis on the longitudinal reference plane is the longitudinal landing point position, and adjacent landing point positions are equidistant.

2. The inspection method according to claim 1, characterized in that: The positioning step further includes: Connect to the remote assistance system based on the problem object judgment result; Taking pictures of the target object in the viewfinder and transmitting them to the remote assistance system; Obtaining a calibration instruction issued by the remote assistance system and projecting a calibration point onto the eyepiece, wherein the calibration point corresponds to the object calibrated in the calibration instruction; Acquire a calibration image within a viewfinder and transmit it to the remote assistance system, wherein the viewfinder is selected based on the distance between the eyepiece and the calibration object, and the calibration image is an image of the calibration object that falls within the viewfinder after the viewfinder is aligned with the calibration object. The calibration object is disassembled based on instructions from the remote assistance system; Determine whether the marked object is a problem object based on the remote assistance system.

3. The inspection method according to claim 1, characterized in that: The teaching steps include: Automatically or manually connecting to an assistance system based on the problem object determination result, wherein the assistance system includes a local assistance system or a remote assistance system; An indicator point movement trajectory is generated on the eyepiece based on the assistance system and the display module, wherein the problem object has a central projection image on the eyepiece relative to the wearer's eyes, the indicator point movement trajectory is located on the central projection of the problem object, and the indicator point movement trajectory corresponds to the operational steps of the solution to the problem object.

4. The inspection method according to claim 1, characterized in that: The steps of aligning the viewfinder with the target object include: selecting a framing frame based on the distance between the eyepiece and the target object, wherein each scale mark on the longitudinal scale line serves as a center to form a framing frame around; Adjust the position of the eyepiece so that the target object moves relatively to the center of the viewfinder.

5. The inspection method according to claim 1, characterized in that: The inspection device also includes: A gyroscope sensor is provided on the frame and is used to detect the deflection angle of the eyepiece; A microphone is arranged on the frame.

6. A remote assistance method based on augmented reality based on the inspection method according to any one of claims 1 to 5, characterized in that: The following steps are involved: Obtain remote images uploaded by inspection equipment; Entering a calibration instruction, wherein the calibration instruction is input based on the position of a calibration object on the remote image, the calibration object having a central projection image on the eyepiece relative to the wearer's eye, and the calibration point corresponding to the calibration instruction is located on the calibration object on the central projection image; outputting the calibration instruction to the inspection device; Re-acquire the remote image uploaded by the inspection device and determine whether the object corresponding to the viewfinder in the remote image is the problem object; Output the judgment result to the inspection equipment; Re-acquire the remote image uploaded by the inspection device; Entering an assistance instruction, wherein an object corresponding to the viewfinder on the remote image is a problem object, the problem object has a central projection image relative to the wearer's eyes on the inspection device, and the assistance instruction includes a movement trajectory of an indicator point located on the central projection of the problem object on the inspection device, the movement trajectory of the indicator point corresponding to an operation step of a solution to the problem object; The assistance instruction is output to the inspection device.

7. A remote assistance system based on augmented reality based on the inspection method according to any one of claims 1 to 5, characterized in that: include: The first image module is used to obtain remote images uploaded by the inspection device; a calibration instruction input module, configured to input a calibration instruction, wherein the calibration instruction is input based on the position of a calibration object on a remote image, the calibration object having a central projection image on the eyepiece relative to the wearer's eye, and the calibration point corresponding to the calibration instruction is located on the calibration object on the central projection image; A calibration instruction output module, used for outputting the calibration instruction to the inspection device; The second image module is used to obtain the remote image uploaded by the inspection device again and determine whether the object corresponding to the viewfinder in the remote image is a problem object; A judgment output module, used to output the judgment result to the inspection equipment; The third image module is used to obtain the remote image uploaded by the inspection device again; an assistance instruction input module, configured to input assistance instructions, wherein the object corresponding to the viewfinder on the remote image is a problem object, the problem object has a central projection image relative to the wearer's eyes on the inspection device, and the assistance instruction includes a movement trajectory of an indicator point on the central projection of the problem object on the inspection device, the movement trajectory of the indicator point corresponding to an operation step of a solution to the problem object; The assistance instruction output module is used to output the assistance instruction to the inspection device.

8. A host, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the inspection method as described in any one of claims 1 to 5, or stores a computer program that can be loaded by the processor and execute the remote assistance method as described in claim 6.

9. A readable storage medium, characterized in that: The computer program that can be loaded by a processor and execute the inspection method according to any one of claims 1 to 5 is stored, or the computer program that can be loaded by a processor and execute the remote assistance method according to claim 6 is stored.

Images