Hybrid tracking registration method and system based on local and global conditions, and electronic equipment

Abstract

The invention discloses a local and global-based hybrid tracking registration method and system and electronic equipment, and relates to the technical field of industrial augmented reality, and the method comprises the steps: when preset industrial equipment is a static object, if tracking of the preset industrial equipment is lost, a corresponding global pose (a first global pose) can be used for recovery. When the preset industrial equipment is a dynamic object, if the preset industrial equipment is tracked and lost, the preset industrial equipment can be re-identified and positioned by using the global pose (the second global pose) of the preset industrial equipment near the tracking loss position. Through the most basic sensors (a visual sensor and an inertial sensor), initialization, global pose estimation, local pose matching, virtual and real registration, recovery after pose tracking loss and the like of poses (local pose and global pose) are realized, and the method is more suitable for virtual and real registration in the field of industrial maintenance.

Description

Background Art

[0002] The current tracking registration methods are mainly divided into vision-based tracking registration, sensor-based tracking registration and hybrid registration methods. Among them, the vision-based registration method is further divided into tracking registration based on artificial identification and tracking registration based on natural features. The natural feature-based tracking registration is further divided into tracking registration based on natural feature points of the image, model-based tracking registration, and tracking registration based on simultaneous positioning and mapping. Tracking registration based on artificial identification requires posting of artificial identifiers, which pollutes the environment. Industrial equipment is a weak texture device with unclear feature points. The method based on natural features of the image is prone to tracking failure. Simultaneous positioning and mapping can only calculate relative poses. The model-based tracking method depends on the local area where the model is located, and large scenes in industrial environments are prone to tracking failure. Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a hybrid tracking registration method, system and electronic device based on local and global tracking, as follows:

[0004] 1) In the first aspect, the present invention provides a hybrid tracking registration method based on local and global, and the specific technical solution is as follows:

[0005] When the preset industrial equipment is a static object, determining a first rough pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to a first image frame containing the complete preset industrial equipment obtained by the visual sensor;

[0006] The model of the preset industrial equipment is used as a priori condition, and the first rough pose is optimized to obtain the first local pose;

[0007] According to the first local pose, the model of the preset industrial equipment is projected onto the first image frame to obtain a first projection contour of the preset industrial equipment;

[0008] Compare the first projection contour with the actual contour of the preset industrial equipment in the first image frame to determine the error of the first local pose;

[0009] Determine whether the error of the first local posture is less than a first preset error threshold, and obtain a first determination result;

[0010] When the first judgment result is yes, a virtual model of the preset industrial equipment is generated according to the first local pose, and the first local pose is converted into a world coordinate system to obtain a first global pose of the preset industrial equipment.

[0011] On the basis of the above scheme, the hybrid tracking and registration method based on local and global of the present invention can also be improved as follows.

[0012] Furthermore, it also includes:

[0013] When the preset industrial equipment is a static object and when the first judgment result is no, a virtual model of the preset industrial equipment is generated according to the first rough pose, and then the three-dimensional model points of the preset industrial equipment are matched with the corresponding points in the first rough pose in the real scene to obtain a first matching result, and the first rough pose is optimized according to the first matching result, and the optimized first rough pose is added to the global pose.

[0014] Furthermore, it also includes:

[0015] When the preset industrial equipment is a static object, and when two consecutive second image frames obtained by the visual sensor meet the preset conditions, the scene where the preset industrial equipment is located is perceived according to the visual features of the previous second image frame and the inertial information of the preset industrial equipment obtained by the inertial sensor, and a first virtual scene corresponding to the scene where the preset industrial equipment is located is established;

[0016] According to the first global pose, adding the virtual model of the preset industrial equipment to the first virtual scene;

[0017] The preset condition is that, in two consecutive second image frames, the first second image frame does not contain any part of the preset industrial equipment, and the second second image frame contains the complete preset industrial equipment.

[0018] Furthermore, it also includes:

[0019] When the preset industrial equipment is a static object, and when the third video frame obtained by the visual sensor contains only a part of the preset industrial equipment, a partial virtual model corresponding to the part of the preset industrial equipment is added to the first virtual scene according to the first global pose.

[0020] Furthermore, it also includes:

[0021] When the preset industrial equipment is a dynamic object, and whenever an image frame containing a complete preset industrial equipment is obtained through a visual sensor, the scene where the preset industrial equipment is located is perceived in real time based on the visual features of the image frame obtained in real time and the inertial information of the preset industrial equipment obtained in real time through an inertial sensor, and a virtual scene corresponding to the scene where the preset industrial equipment is located is established in real time;

[0022] When the preset industrial equipment is a dynamic object, the third image frame obtained by the visual sensor is matched with an image frame containing the complete preset industrial equipment before the third image frame, and it is determined whether the number of matched feature points exceeds 5, thereby obtaining a second determination result;

[0023] When the second judgment result is no, determining a second rough pose of the preset industrial equipment in the first three-dimensional coordinate system used by the visual sensor according to the third image frame;

[0024] The model of the preset industrial equipment is used as a priori condition, and the second coarse pose is optimized to obtain the second local pose;

[0025] According to the second local pose, the model of the preset industrial equipment is projected onto the third image frame to obtain a second projection contour of the preset industrial equipment;

[0026] Compare the second projection contour with the actual contour of the preset industrial equipment in the third image frame to determine the error of the second local pose;

[0027] Determine whether the error of the second local posture is less than a second preset error threshold, and obtain a third determination result;

[0028] When the third judgment result is yes, a virtual model of the preset industrial equipment is generated according to the second local pose, and the second local pose is converted into a world coordinate system to obtain a second global pose of the preset industrial equipment;

[0029] According to the second global pose, the virtual model of the preset industrial equipment is added to the corresponding virtual scene;

[0030] When the third judgment result is no, generating a virtual model of the preset industrial equipment according to the second local pose, then matching the three-dimensional model points of the preset industrial equipment with corresponding points in the second local pose in the real scene to obtain a second matching result, optimizing the second local pose according to the matching result, and adding the optimized second local pose to the global pose;

[0031] When the second judgment result is yes, a new second global pose of the preset industrial equipment is calculated based on all matched feature points and the second global pose calculated based on the previous image frame of the third image frame containing the complete preset industrial equipment, and based on the new second global pose, the virtual model of the preset industrial equipment is added to the corresponding virtual scene.

[0032] Furthermore, it also includes:

[0033] Obtain rendering images of the preset industrial equipment model at different camera rendering angles to obtain a template library;

[0034] According to a first image frame containing a complete preset industrial device obtained by a visual sensor, determining a first rough pose of the preset industrial device in a first three-dimensional coordinate system used by the visual sensor includes:

[0035] Matching the first image frame with each rendered image in the template library, and taking the pose of the preset industrial equipment in the matched rendered image as the first rough pose;

[0036] Determining a second rough pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to the third image frame includes:

[0037] The third image frame is matched with each rendered image in the template library, and the posture of the preset industrial equipment in the matched rendered image is used as the second rough posture.

[0038] Furthermore, the model of the preset industrial equipment is used as a priori condition to optimize the first rough pose, including:

[0039] The model of the preset industrial equipment is used as a priori condition, and the first rough pose is optimized using a region-based method;

[0040] The model of the preset industrial equipment is used as a priori condition to optimize the second rough pose, including:

[0041] Taking the model of preset industrial equipment as a priori condition, a region-based method is used to optimize the second coarse pose.

[0042] 2) In the second aspect, the present invention also provides a hybrid tracking and registration system based on local and global. The specific technical solution is as follows:

[0043] It includes a coarse posture acquisition module, a coarse posture optimization module, a projection contour acquisition module, an error acquisition module, a judgment module and a registration module;

[0044] The coarse posture acquisition module is used to: when the preset industrial equipment is a static object, determine a first coarse posture of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to a first image frame containing the complete preset industrial equipment obtained by the visual sensor;

[0045] The coarse pose optimization module is used to: use the preset industrial equipment model as a priori condition, optimize the first coarse pose, and obtain the first local pose;

[0046] The projection contour acquisition module is used to: project the model of the preset industrial equipment onto the first image frame according to the first local posture to obtain a first projection contour of the preset industrial equipment;

[0047] The error acquisition module is used to: compare the first projection contour with the real contour of the preset industrial equipment in the first image frame to determine the error of the first local posture;

[0048] The judgment module is used to: judge whether the error of the first local posture is less than a first preset error threshold, and obtain a first judgment result;

[0049] The registration module is used to: when the first judgment result is yes, generate a virtual model of the preset industrial equipment according to the first local pose, and convert the first local pose into a world coordinate system to obtain a first global pose of the preset industrial equipment.

[0050] Based on the above solution, the hybrid tracking and registration system based on local and global aspects of the present invention can also be improved as follows.

[0051] Furthermore, the registration module is also used for: when the preset industrial equipment is a static object, and when the first judgment result is no, generating a virtual model of the preset industrial equipment according to the first rough pose, and then matching the three-dimensional model points of the preset industrial equipment with the corresponding points in the first rough pose in the real scene to obtain a first matching result, optimizing the first rough pose according to the first matching result, and adding the optimized first rough pose to the global pose.

[0052] Furthermore, it also includes a virtual scene establishment module, which is used to: when the preset industrial equipment is a static object and when two consecutive second image frames obtained by the visual sensor meet the preset conditions, perceive the scene where the preset industrial equipment is located according to the visual features of the previous second image frame and the inertial information of the preset industrial equipment obtained by the inertial sensor, and establish a first virtual scene corresponding to the scene where the preset industrial equipment is located;

[0053] The registration module is further used to: add the virtual model of the preset industrial equipment to the first virtual scene according to the first global pose;

[0054] The preset condition is that, in two consecutive second image frames, the first second image frame does not contain any part of the preset industrial equipment, and the second second image frame contains the complete preset industrial equipment.

[0055] Furthermore, the registration module is also used to: when the preset industrial equipment is a static object, and when the third video frame obtained by the visual sensor only contains part of the preset industrial equipment, add a partial virtual model corresponding to the part of the preset industrial equipment to the first virtual scene according to the first global pose.

[0056] Furthermore, the virtual scene establishment module is also used for: when the preset industrial equipment is a dynamic object, and whenever an image frame containing a complete preset industrial equipment is obtained by a visual sensor, the scene where the preset industrial equipment is located is perceived in real time according to the visual features of the image frame obtained in real time and the inertial information of the preset industrial equipment obtained in real time by an inertial sensor, and a virtual scene corresponding to the scene where the preset industrial equipment is located is established in real time;

[0057] The judgment module is also used for: when the preset industrial equipment is a dynamic object, matching the feature points of the third image frame obtained by the visual sensor with the previous image frame of the third image frame containing the complete preset industrial equipment, judging whether the number of matched feature points exceeds 5, and obtaining a second judgment result;

[0058] The coarse posture acquisition module is also used to: when the second judgment result is no, determine a second coarse posture of the preset industrial equipment in the first three-dimensional coordinate system used by the visual sensor according to the third image frame;

[0059] The coarse pose optimization module is also used to: optimize the second coarse pose by taking the model of the preset industrial equipment as a priori condition to obtain a second local pose;

[0060] The projection contour acquisition module is also used to: project the model of the preset industrial equipment onto the third image frame according to the second local posture to obtain a second projection contour of the preset industrial equipment;

[0061] The error acquisition module is also used to: compare the second projection contour with the real contour of the preset industrial equipment in the third image frame to determine the error of the second local posture;

[0062] The judgment module is further used to: judge whether the error of the second local posture is less than a second preset error threshold, and obtain a third judgment result;

[0063] The registration module is further used for: when the third judgment result is yes, generating a virtual model of the preset industrial equipment according to the second local pose, converting the second local pose into the world coordinate system to obtain a second global pose of the preset industrial equipment, and adding the virtual model of the preset industrial equipment to the corresponding virtual scene according to the second global pose;

[0064] The registration module is also used for: when the third judgment result is no, generating a virtual model of the preset industrial equipment according to the second local pose, and then matching the three-dimensional model points of the preset industrial equipment with the corresponding points in the second local pose in the real scene to obtain a second matching result, optimizing the second local pose according to the matching result, and adding the optimized second local pose to the global pose;

[0065] The registration module is also used to: when the second judgment result is yes, calculate a new second global pose of the preset industrial equipment based on all matched feature points and the second global pose calculated based on the previous image frame of the third image frame containing the complete preset industrial equipment, and add the virtual model of the preset industrial equipment to the corresponding virtual scene based on the new second global pose.

[0066] Furthermore, it also includes a template library acquisition module, which is used to: acquire rendering images of a model of a preset industrial equipment at different camera rendering angles to obtain a template library;

[0067] The coarse pose acquisition module is specifically used to: match the first image frame with each rendered image in the template library, and use the pose of the preset industrial equipment in the matched rendered image as the first coarse pose; match the third image frame with each rendered image in the template library, and use the pose of the preset industrial equipment in the matched rendered image as the second coarse pose.

[0068] Furthermore, the coarse pose optimization module is specifically used to: use the model of the preset industrial equipment as a priori condition and use a region-based method to optimize the first coarse pose; use the model of the preset industrial equipment as a priori condition and use a region-based method to optimize the second coarse pose.

[0069] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so that the electronic device implements any of the above-mentioned local and global hybrid tracking and registration methods.

[0070] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the above-mentioned local and global hybrid tracking and registration methods is implemented.

[0071] The beneficial effects of the present invention are as follows:

[0072] When the preset industrial equipment is a static object, if the preset industrial equipment tracking is lost, the corresponding global pose (first global pose) can be used to restore it. When the preset industrial equipment is a dynamic object, if the preset industrial equipment tracking is lost, the preset industrial equipment can be re-identified and positioned using the global pose (second global pose) of the preset industrial equipment near the tracking loss position. The present invention uses the most basic sensors (visual sensors and inertial sensors) to achieve initialization of poses (local poses and global poses), global pose estimation, local pose matching, virtual-real registration, and recovery after pose tracking is lost, which is more suitable for virtual-real registration in the field of industrial maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0074] Figure 1 A schematic diagram of a flow chart of a hybrid tracking and registration method based on local and global aspects according to an embodiment of the present invention;

[0075] Figure 2 Schematic diagram of different rendering results;

[0076] Figure 3 This is a schematic diagram of the static virtual-real registration result;

[0077] Figure 4 It is a schematic diagram of the virtual and real registration results;

[0078] Figure 5 is a schematic diagram of a manual masking filter;

[0079] Figure 6 Schematic diagram of filter tracking loss due to manual occlusion;

[0080] Figure 7 A schematic diagram for realizing the recovery of tracking;

[0081] Figure 8 It is a schematic diagram of the dynamic virtual-real registration result;

[0082] Fig. 9 A structural diagram of a hybrid tracking and registration system based on local and global aspects according to an embodiment of the present invention;

[0083] Fig.10 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0085] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0086] like Figure 1 As shown, a hybrid tracking registration method based on local and global aspects of an embodiment of the present invention first determines whether a preset industrial device is a static object or a dynamic object, specifically:

[0087] The preset industrial equipment may be filters, motors, industrial robots, CNC machine tools, CNC milling machines, CNC milling machines, 3D printers and laser cutting machines, etc. Among them, after filters, motors, CNC machine tools, CNC milling machines, CNC milling machines, 3D printers and laser cutting machines are arranged in corresponding positions, their positions generally do not change, therefore, they may be considered as static objects; the positions of industrial robots often change, and they may be considered as dynamic objects; industrial robots may be welding robots, assembly robots, handling robots and spraying robots, etc.

[0088] It should be noted that the classification of static objects or dynamic objects is relative. For example, in general, the installation position of the filter is fixed, and the filter can be considered a static object. However, in some scenarios, such as scenarios where the filter needs to be moved, the position of the filter will change, and the filter can be considered a dynamic object.

[0089] When the industrial equipment is preset as a static object, perform the following steps:

[0090] S10, obtaining the first rough pose, specifically:

[0091] When the preset industrial equipment is a static object, a first rough pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor is determined according to a first image frame containing the complete preset industrial equipment obtained by the visual sensor, wherein the visual sensor may be a camera or a monocular visual sensor, etc.

[0092] The specific process of obtaining the first rough pose includes:

[0093] Obtain rendered images of a model of a preset industrial equipment at different camera rendering angles to obtain a template library; match the first image frame with each rendered image in the template library, and use the pose of the preset industrial equipment in the matched rendered image as the first rough pose.

[0094] Specifically, during rendering, ambient light rendering, diffuse reflection rendering, highlight reflection rendering and Blinn-Phong rendering can be performed. For example, by rendering the filter using different rendering methods, the ambient light rendering results, the highlight reflection rendering effects and the Blinn-Phong rendering effects can be obtained. Figure 2 shown.

[0095] S11. Obtain a first local pose. Specifically, take the model of the preset industrial equipment as a priori condition, optimize the first rough pose, and obtain the first local pose. The local pose refers to the optimized first rough pose of the preset industrial equipment. The first local pose only refers to the pose of the preset industrial equipment, and does not include the poses of other objects in the scene where the preset industrial equipment is located.

[0096] Among them, the model of the preset industrial equipment is used as a priori condition, and the region-based method is used to optimize the first rough pose to obtain the first local pose.

[0097] The model of the preset industrial equipment may be: a CAD model of the preset industrial equipment, or a three-dimensional model of the preset industrial equipment.

[0098] S12, obtaining a first projection profile, specifically:

[0099] According to the first local posture, the model of the preset industrial equipment is projected onto the first image frame to obtain a first projection outline of the preset industrial equipment.

[0100] S13, determining the error of the first local pose, specifically:

[0101] The first projection contour is compared with the actual contour of the preset industrial equipment in the first image frame to determine the error of the first local posture.

[0102] S14, determine whether the error of the first local posture is less than a first preset error threshold, and obtain a first judgment result. When the first judgment result is yes, continue to execute S15; when the first judgment result is no, continue to execute S20.

[0103] Among them, the first preset error threshold can be set according to actual conditions.

[0104] S15, generating a virtual model of a preset industrial device and obtaining a first global pose, specifically:

[0105] When the first judgment result is yes, a virtual model of the preset industrial equipment is generated according to the first local pose, and the first local pose is converted into the world coordinate system to obtain a first global pose of the preset industrial equipment, and then S16 is continued to be executed.

[0106] S16, determining whether two consecutive second image frames meet a preset condition, if so, continuing to execute S17, if not, continuing to execute S19.

[0107] The preset condition is that, in two consecutive second image frames, the first second image frame does not contain any part of the preset industrial equipment, and the second second image frame contains the complete preset industrial equipment.

[0108] S17, establishing a first virtual scene, specifically:

[0109] According to the visual features of the previous second image frame and the inertial information of the preset industrial equipment obtained by the inertial sensor, the scene where the preset industrial equipment is located is perceived, a first virtual scene corresponding to the scene where the preset industrial equipment is located is established, and then S18 is continued.

[0110] The preset inertial information of the industrial equipment includes: angular velocity and angular acceleration, and the visual features of the second image frame include ORB feature points.

[0111] S18, according to the first global posture, adding the virtual model of the preset industrial equipment to the first virtual scene, for example, adding a filter to the first virtual scene, the static virtual-real registration result is as follows: Figure 3 shown.

[0112] S19. Generate a virtual model of the preset industrial equipment according to the first rough pose, and then execute S20.

[0113] S20, optimizing the first rough pose, and adding the optimized first rough pose to the global pose, specifically:

[0114] Match the three-dimensional model points of the preset industrial equipment with the corresponding points in the first rough pose in the real scene to obtain a first matching result, optimize the first rough pose according to the first matching result, and add the optimized first rough pose to the global pose.

[0115] Taking the filter as an example, after S11 to S18, the virtual model of the filter is added to the corresponding virtual scene, and the virtual-real registration result is as follows: Figure 4 shown.

[0116] When you manually block the filter, you cannot see the filter at this time, such as Figure 5 As shown, it is obvious that the current image frame (i.e., the “previous second image frame” in the above text) does not contain any part of the preset industrial equipment, that is, the filter tracking is lost due to manual occlusion, such as Figure 6 When the next image frame of the current image frame (i.e., the "second image frame after" in the above text) contains a complete preset industrial device, the scene where the preset industrial device is located is perceived according to the visual features of the previous second image frame and the inertial information of the preset industrial device obtained by the inertial sensor, and a first virtual scene corresponding to the scene where the preset industrial device is located is established. According to the first global pose, the virtual model of the preset industrial device is added to the first virtual scene to achieve tracking recovery, as shown in FIG. Figure 7 shown.

[0117] In another embodiment, whenever an image frame containing complete preset industrial equipment is obtained through a visual sensor, the scene where the preset industrial equipment is located is perceived in real time based on the visual features of the image frame obtained in real time and the inertial information of the preset industrial equipment obtained in real time through an inertial sensor, and a virtual scene corresponding to the scene where the preset industrial equipment is located is established in real time.

[0118] Optionally, in the above technical solution, it also includes:

[0119] When the preset industrial equipment is a static object, and when the third video frame obtained by the visual sensor contains only a part of the preset industrial equipment, a partial virtual model corresponding to the part of the preset industrial equipment is added to the first virtual scene according to the first global pose.

[0120] When the industrial equipment is preset as a dynamic object, perform the following steps:

[0121] S21. Establishing a virtual scene in real time, specifically:

[0122] Whenever an image frame containing complete preset industrial equipment is obtained through the visual sensor, the scene where the preset industrial equipment is located is perceived in real time based on the visual features of the image frame obtained in real time and the inertial information of the preset industrial equipment obtained in real time through the inertial sensor, and a virtual scene corresponding to the scene where the preset industrial equipment is located is established in real time.

[0123] S22, matching feature points, and determining whether the number of matched feature points exceeds 5, specifically:

[0124] Match feature points of the third image frame obtained by the visual sensor with the previous image frame of the third image frame containing complete preset industrial equipment, and determine whether the number of matched feature points exceeds 5 to obtain a second judgment result. When the second judgment result is no, continue to execute S23; when the second judgment result is yes, continue to execute S32.

[0125] S23, obtaining a second rough pose, specifically:

[0126] According to the third image frame, a second rough pose of the preset industrial equipment in the first three-dimensional coordinate system used by the visual sensor is determined, and the specific process is as follows:

[0127] The third image frame is matched with each rendered image in the template library, and the posture of the preset industrial equipment in the matched rendered image is used as the second rough posture.

[0128] S24, obtaining a second local pose, specifically:

[0129] The model of the preset industrial equipment is used as a priori condition to optimize the second coarse pose and obtain the second local pose. Specifically:

[0130] Taking the model of preset industrial equipment as a priori condition, a region-based method is used to optimize the second coarse pose.

[0131] S25, obtaining a second projection profile, specifically:

[0132] According to the second local posture, the model of the preset industrial equipment is projected onto the third image frame to obtain a second projection contour of the preset industrial equipment.

[0133] S26, determining the error of the second local pose, specifically:

[0134] The second projection contour is compared with the actual contour of the preset industrial equipment in the third image frame to determine the error of the second local posture.

[0135] S27, determine whether the error of the second local posture is less than the second preset error threshold, and obtain a third judgment result. When the third judgment result is yes, continue to execute S28; when the third judgment result is no, continue to execute S30.

[0136] S28, generating a virtual model of the preset industrial equipment and obtaining a second global pose, specifically:

[0137] Generate a virtual model of the preset industrial equipment according to the second local pose, and transform the second local pose into a world coordinate system to obtain a second global pose of the preset industrial equipment;

[0138] S29, according to the second global pose, the virtual model of the preset industrial equipment is added to the corresponding virtual scene. For example, when the filter is moved, the filter is a dynamic object, and the virtual model of the preset industrial equipment is added to the corresponding virtual scene. The dynamic virtual-real registration result is obtained, such as Figure 8 shown.

[0139] S30, generating a virtual model of a preset industrial equipment, specifically:

[0140] A virtual model of the preset industrial equipment is generated according to the second local pose.

[0141] S31, optimizing the second local pose, and adding the optimized second local pose to the global pose, specifically:

[0142] The three-dimensional model points of the preset industrial equipment are matched with the corresponding points in the second local pose in the real scene to obtain a second matching result, the second local pose is optimized according to the matching result, and the optimized second local pose is added to the global pose.

[0143] S32, obtaining a new second global pose, specifically:

[0144] A new second global pose of the preset industrial equipment is calculated according to all matched feature points and the second global pose calculated based on the previous image frame of the third image frame containing the complete preset industrial equipment.

[0145] S33. According to the new second global pose, the virtual model of the preset industrial equipment is added to the corresponding virtual scene.

[0146] The invention discloses a hybrid tracking and registration method based on local and global. The method uses a monocular vision sensor and an inertial sensor to initialize the posture of a preset industrial device. The visual features between video frames obtained by the visual sensor are matched to obtain the posture transformation between different video frames. At the same time, the corresponding scale information is restored by integrating the inertial information obtained by the inertial sensor. The global initial posture is obtained by the monocular vision sensor and the inertial sensor. Then, the point and line features of vision and the inertial sensor are used to synchronously locate and build a map to determine the global posture of the entire scene. At the same time, an offline template library is constructed for the three-dimensional model of the device in the offline stage to obtain the projection map of the model in different postures. While estimating the global posture of the environment, a search and match is performed to determine whether there is a corresponding industrial device in the scene. According to the most matching template in the template library, the posture corresponding to the template is used as a rough estimate of the local posture of the device. The virtual posture is superimposed according to the rough estimate of the local posture, and the three-dimensional model of the device is projected into the real scene according to the rough estimate of the local posture. Then, the local posture is optimized according to the region-based method. When the local posture error is less than a certain threshold, the local posture is added to the global posture to determine the specific posture of the industrial equipment in the entire scene. When the preset industrial equipment is a static object, if the preset industrial equipment tracking is lost, the corresponding global pose (first global pose) can be used to restore it. When the preset industrial equipment is a dynamic object, if the preset industrial equipment tracking is lost, the preset industrial equipment can be re-identified and positioned using the global pose (second global pose) of the preset industrial equipment near the tracking loss position. The present invention uses the most basic sensors (visual sensors and inertial sensors) to achieve initialization of poses (local poses and global poses), global pose estimation, local pose matching, virtual-real registration, and recovery after pose tracking is lost, which is more suitable for virtual-real registration in the field of industrial maintenance.

[0147] The present invention only uses cameras and common inertial sensors, and since industrial equipment often has CAD models, it can be completed using only the input of inexpensive equipment, and the implementation is simple. Automatic initialization can be achieved based on visual features and inertial information, without the need for manual input of initial pose, and has strong adaptability. Since model-based local tracking depends on local scenes, when industrial equipment moves quickly or the camera moves quickly, tracking loss is likely to occur. Due to the existence of the global pose, the pose can be restored. The stability and accuracy are improved by using inertial sensors and visual sensors. At the same time, a combination of model-based tracking and synchronous positioning and mapping is adopted in visual tracking, taking into account local features and global features, which is more stable for large scenes such as industrial equipment tracking and registration.

[0148] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0149] like Fig. 9 As shown, a hybrid tracking and registration system 200 based on local and global aspects of an embodiment of the present invention includes a coarse pose acquisition module 201, a coarse pose optimization module 202, a projection profile acquisition module 203, an error acquisition module 204, a judgment module 205 and a registration module 206;

[0150] It includes a coarse posture acquisition module 201, a coarse posture optimization module 202, a projection profile acquisition module 203, an error acquisition module 204, a judgment module 205 and a registration module 206;

[0151] The coarse pose acquisition module 201 is used to: when the preset industrial equipment is a static object, determine a first coarse pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to a first image frame containing the complete preset industrial equipment obtained by the visual sensor;

[0152] The coarse pose optimization module 202 is used to: optimize the first coarse pose by taking the model of the preset industrial equipment as a priori condition to obtain a first local pose;

[0153] The projection contour acquisition module 203 is used to: project the model of the preset industrial equipment onto the first image frame according to the first local posture to obtain a first projection contour of the preset industrial equipment;

[0154] The error acquisition module 204 is used to: compare the first projection contour with the real contour of the preset industrial equipment in the first image frame to determine the error of the first local posture;

[0155] The judgment module 205 is used to: judge whether the error of the first local posture is less than a first preset error threshold, and obtain a first judgment result;

[0156] The registration module 206 is used to: when the first judgment result is yes, generate a virtual model of the preset industrial equipment according to the first local pose, and convert the first local pose into the world coordinate system to obtain a first global pose of the preset industrial equipment.

[0157] Optionally, in the above technical solution, the registration module 206 is also used for: when the preset industrial equipment is a static object, and when the first judgment result is no, generating a virtual model of the preset industrial equipment according to the first rough pose, and then matching the three-dimensional model points of the preset industrial equipment with the corresponding points in the first rough pose in the real scene to obtain a first matching result, optimizing the first rough pose according to the first matching result, and adding the optimized first rough pose to the global pose.

[0158] Optionally, in the above technical solution, a virtual scene establishment module is further included, and the virtual scene establishment module is used to: when the preset industrial equipment is a static object, and when two consecutive second image frames obtained by the visual sensor meet the preset conditions, the scene where the preset industrial equipment is located is perceived according to the visual features of the previous second image frame and the inertial information of the preset industrial equipment obtained by the inertial sensor, and a first virtual scene corresponding to the scene where the preset industrial equipment is located is established;

[0159] The registration module 206 is further used to: add the virtual model of the preset industrial equipment to the first virtual scene according to the first global pose;

[0160] The preset condition is that, in two consecutive second image frames, the first second image frame does not contain any part of the preset industrial equipment, and the second second image frame contains the complete preset industrial equipment.

[0161] Optionally, in the above technical solution, the registration module 206 is also used for: when the preset industrial equipment is a static object, and when the third video frame obtained by the visual sensor only contains part of the preset industrial equipment, adding a partial virtual model corresponding to the part of the preset industrial equipment to the first virtual scene according to the first global pose.

[0162] Optionally, in the above technical solution, the virtual scene establishment module is further used for: when the preset industrial equipment is a dynamic object, and whenever an image frame containing a complete preset industrial equipment is obtained by a visual sensor, the scene where the preset industrial equipment is located is perceived in real time according to the visual features of the image frame obtained in real time and the inertial information of the preset industrial equipment obtained in real time by an inertial sensor, and a virtual scene corresponding to the scene where the preset industrial equipment is located is established in real time;

[0163] The judgment module 205 is further used to: when the preset industrial equipment is a dynamic object, match the feature points of the third image frame obtained by the visual sensor with the previous image frame of the third image frame containing the complete preset industrial equipment, and determine whether the number of matched feature points exceeds 5 to obtain a second judgment result;

[0164] The coarse posture acquisition module 201 is further used to: when the second judgment result is no, determine a second coarse posture of the preset industrial equipment in the first three-dimensional coordinate system used by the visual sensor according to the third image frame;

[0165] The coarse pose optimization module 202 is further used to: optimize the second coarse pose by taking the preset industrial equipment model as a priori condition to obtain a second local pose;

[0166] The projection contour acquisition module 203 is further used to: project the model of the preset industrial equipment onto the third image frame according to the second local posture to obtain a second projection contour of the preset industrial equipment;

[0167] The error acquisition module 204 is further used to: compare the second projection contour with the real contour of the preset industrial equipment in the third image frame to determine the error of the second local posture;

[0168] The judgment module 205 is further used to: judge whether the error of the second local posture is less than a second preset error threshold, and obtain a third judgment result;

[0169] The registration module 206 is further used for: when the third judgment result is yes, generating a virtual model of the preset industrial equipment according to the second local pose, converting the second local pose into the world coordinate system to obtain a second global pose of the preset industrial equipment, and adding the virtual model of the preset industrial equipment to the corresponding virtual scene according to the second global pose;

[0170] The registration module 206 is further used for: when the third judgment result is no, generating a virtual model of the preset industrial equipment according to the second local pose, and then matching the three-dimensional model points of the preset industrial equipment with the corresponding points in the second local pose in the real scene to obtain a second matching result, optimizing the second local pose according to the matching result, and adding the optimized second local pose to the global pose;

[0171] The registration module 206 is also used to: when the second judgment result is yes, calculate a new second global pose of the preset industrial equipment based on all matched feature points and the second global pose calculated based on the previous image frame of the third image frame containing the complete preset industrial equipment, and add the virtual model of the preset industrial equipment to the corresponding virtual scene based on the new second global pose.

[0172] Optionally, in the above technical solution, a template library acquisition module is further included, and the template library acquisition module is used to: acquire rendering images of a model of a preset industrial equipment at different camera rendering angles to obtain a template library;

[0173] The coarse pose acquisition module 201 is specifically used to: match the first image frame with each rendered image in the template library, and use the pose of the preset industrial equipment in the matched rendered image as the first coarse pose; match the third image frame with each rendered image in the template library, and use the pose of the preset industrial equipment in the matched rendered image as the second coarse pose.

[0174] Optionally, in the above technical solution, the coarse pose optimization module 202 is specifically used to: take the model of the preset industrial equipment as a priori condition, and use a region-based method to optimize the first coarse pose; take the model of the preset industrial equipment as a priori condition, and use a region-based method to optimize the second coarse pose.

[0175] It should be noted that the beneficial effects of the hybrid tracking and registration system 200 based on local and global provided by the above embodiment are the same as the beneficial effects of the hybrid tracking and registration method based on local and global provided by the above embodiment, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0176] Among them, the local and global hybrid tracking and registration system of the present invention can be a computer program (including program code) running in a computer device. For example, the local and global hybrid tracking and registration system of the present invention is an application software that can be used to execute the corresponding steps in the local and global hybrid tracking and registration method of the present invention.

[0177] In some embodiments, the local and global hybrid tracking and registration system of the present invention can be implemented in a combination of software and hardware. As an example, the local and global hybrid tracking and registration system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the local and global hybrid tracking and registration method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0178] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0179] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned local and global hybrid tracking and registration methods is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the local and global hybrid tracking and registration method shown in any embodiment of the present invention by calling the computer program.

[0180] In an alternative embodiment, an electronic device is provided, such as Fig.10 As shown, Fig.10 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0181] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0182] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0183] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0184] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0185] Among them, the electronic device can also be a terminal device, and the terminal device can be any device that can install applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0186] It should be noted that Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0187] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any one of the above-mentioned local and global hybrid tracking and registration methods is implemented.

[0188] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0189] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned local and global hybrid tracking and registration methods.

[0190] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0191] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0192] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0193] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0194] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0195] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0196] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0197] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A hybrid tracking registration method based on local and global, characterized in that: include: When the preset industrial equipment is a static object, determining a first rough pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to a first image frame containing the complete preset industrial equipment obtained by the visual sensor; Taking the model of the preset industrial equipment as a priori condition, optimizing the first rough pose to obtain a first local pose; According to the first local posture, projecting the model of the preset industrial equipment onto the first image frame to obtain a first projection outline of the preset industrial equipment; Comparing the first projection outline with the actual outline of the preset industrial equipment in the first image frame to determine the error of the first local posture; Determine whether the error of the first local posture is less than a first preset error threshold, and obtain a first determination result; When the first judgment result is yes, a virtual model of the preset industrial equipment is generated according to the first local posture, and the first local posture is converted into a world coordinate system to obtain a first global posture of the preset industrial equipment.

2. A hybrid tracking and registration method based on local and global according to claim 1, characterized in that: Also includes: When the preset industrial equipment is a static object and when the first judgment result is no, a virtual model of the preset industrial equipment is generated according to the first rough pose, and then the three-dimensional model points of the preset industrial equipment are matched with the corresponding points in the first rough pose in the real scene to obtain a first matching result, and the first rough pose is optimized according to the first matching result, and the optimized first rough pose is added to the global pose.

3. A hybrid tracking and registration method based on local and global according to claim 1, characterized in that: Also includes: When the preset industrial equipment is a static object and when two consecutive second image frames obtained by the visual sensor meet a preset condition, the scene where the preset industrial equipment is located is perceived according to the visual features of the previous second image frame and the inertial information of the preset industrial equipment obtained by the inertial sensor, and a first virtual scene corresponding to the scene where the preset industrial equipment is located is established; According to the first global pose, adding the virtual model of the preset industrial equipment to the first virtual scene; The preset condition is that, of the two consecutive second image frames, the first second image frame does not contain any part of the preset industrial equipment, and the second second image frame contains the complete preset industrial equipment.

4. A hybrid tracking and registration method based on local and global according to claim 3, characterized in that: Also includes: When the preset industrial equipment is a static object and when the third video frame obtained by the visual sensor contains only part of the preset industrial equipment, a partial virtual model corresponding to the part of the preset industrial equipment is added to the first virtual scene according to the first global pose.

5. The hybrid tracking and registration method based on local and global according to claim 1, characterized in that: Also includes: When the preset industrial equipment is a dynamic object, and whenever an image frame containing a complete preset industrial equipment is obtained by a visual sensor, the scene where the preset industrial equipment is located is perceived in real time according to the visual features of the image frame obtained in real time and the inertial information of the preset industrial equipment obtained in real time by an inertial sensor, and a virtual scene corresponding to the scene where the preset industrial equipment is located is established in real time; When the preset industrial equipment is a dynamic object, matching feature points of the third image frame obtained by the visual sensor with an image frame of the third image frame that contains the complete preset industrial equipment, and determining whether the number of matched feature points exceeds 5 to obtain a second determination result; When the second judgment result is no, determining a second rough pose of the preset industrial equipment in the first three-dimensional coordinate system used by the visual sensor according to the third image frame; Taking the model of the preset industrial equipment as a priori condition, optimizing the second coarse pose to obtain a second local pose; According to the second local posture, projecting the model of the preset industrial equipment onto the third image frame to obtain a second projection contour of the preset industrial equipment; Performing a position comparison between the second projection outline and the real outline of the preset industrial equipment in the third image frame to determine an error of the second local posture; Determine whether the error of the second local posture is less than a second preset error threshold, and obtain a third determination result; When the third judgment result is yes, generating a virtual model of the preset industrial equipment according to the second local pose, and converting the second local pose into a world coordinate system to obtain a second global pose of the preset industrial equipment; According to the second global pose, adding the virtual model of the preset industrial equipment to the corresponding virtual scene; When the third judgment result is no, generating a virtual model of the preset industrial equipment according to the second local pose, then matching the three-dimensional model points of the preset industrial equipment with corresponding points in the second local pose in the real scene to obtain a second matching result, optimizing the second local pose according to the matching result, and adding the optimized second local pose to the global pose; When the second judgment result is yes, a new second global pose of the preset industrial equipment is calculated based on all matched feature points and the second global pose calculated based on the previous image frame of the third image frame containing the complete preset industrial equipment, and based on the new second global pose, the virtual model of the preset industrial equipment is added to the corresponding virtual scene.

6. A hybrid tracking and registration method based on local and global according to claim 5, characterized in that: Also includes: Obtaining rendering images of the model of the preset industrial equipment at different camera rendering angles to obtain a template library; According to a first image frame containing a complete preset industrial device obtained by a visual sensor, determining a first rough pose of the preset industrial device in a first three-dimensional coordinate system used by the visual sensor includes: Matching the first image frame with each rendered image in the template library, and taking the posture of the preset industrial equipment in the matched rendered image as the first rough posture; Determining a second rough pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to the third image frame includes: The third image frame is matched with each rendered image in the template library, and the posture of the preset industrial equipment in the matched rendered image is used as the second rough posture.

7. A hybrid tracking and registration method based on local and global according to claim 5, characterized in that: Taking the model of the preset industrial equipment as a priori condition, optimizing the first rough pose includes: Taking the model of the preset industrial equipment as a priori condition, optimizing the first rough pose using a region-based method; Taking the model of the preset industrial equipment as a priori condition, optimizing the second coarse pose includes: The model of the preset industrial equipment is used as a priori condition, and the second coarse pose is optimized using a region-based method.

8. A hybrid tracking and registration system based on local and global, characterized in that: It includes a coarse posture acquisition module, a coarse posture optimization module, a projection contour acquisition module, an error acquisition module, a judgment module and a registration module; The rough pose acquisition module is used to: when the preset industrial equipment is a static object, determine a first rough pose of the preset industrial equipment in a first three-dimensional coordinate system used by the visual sensor according to a first image frame containing the complete preset industrial equipment obtained by the visual sensor; The coarse pose optimization module is used to: use the model of the preset industrial equipment as a priori condition, optimize the first coarse pose, and obtain a first local pose; The projection contour acquisition module is used to: project the model of the preset industrial equipment onto the first image frame according to the first local posture to obtain a first projection contour of the preset industrial equipment; The error acquisition module is used to: compare the first projection contour with the real contour of the preset industrial equipment in the first image frame to determine the error of the first local posture; The judgment module is used to: judge whether the error of the first local posture is less than a first preset error threshold, and obtain a first judgment result; The registration module is used to: when the first judgment result is yes, generate a virtual model of the preset industrial equipment according to the first local posture, and convert the first local posture into a world coordinate system to obtain a first global posture of the preset industrial equipment.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a hybrid tracking and registration method based on local and global is implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the local and global hybrid tracking and registration method according to any one of claims 1 to 7 is implemented.

Images