Object tracking method and system based on implicit neural field

Abstract

The present application relates to an object tracking method based on an implicit neural field, which includes: inputting the initial pose of the current frame image into the implicit neural field to obtain a reference image and the object 3D points corresponding to the reference image, optimizing the estimated initial pose based on the hierarchical features of the object 3D points in the current frame image and the reference image, and obtaining the optimized pose of the current frame image, thereby solving the problem of complex implementation of the object's 6DOF tracking method. Only a reference image needs to be rendered through the implicit neural field for pose optimization, and no sensor or object CAD model is required, which reduces the complexity of the object tracking method and improves the tracking efficiency. In addition, the use of general feature extraction instead of traditional grayscale values ​​improves the robustness of the algorithm to changes in lighting and other factors.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to an object tracking method, system, electronic device and storage medium based on implicit neural fields. Background Art

[0002] With the rapid development of computer technology, 6DOF tracking technology for 3D objects is becoming increasingly widely used. In augmented reality applications, by obtaining the position and pose of a specific object in real time, virtual content can be anchored to the object, thereby enhancing the real environment. In industrial applications, by determining the position and pose of the target object in real time, a robotic arm can achieve precise grasping.

[0003] In related technologies, tracking objects requires the use of depth cameras, multiple sensors, or CAD models of the objects. These methods are complex to implement and have high requirements for software and hardware.

[0004] Currently, there is no effective solution to the problem of complex implementation of 6DOF tracking methods for objects in related technologies. Summary of the Invention

[0005] The embodiments of the present application provide an object tracking method, system, electronic device and storage medium based on an implicit neural field, so as to at least solve the problem of complex implementation of the 6DOF tracking method of objects in the related art.

[0006] In a first aspect, an embodiment of the present application provides an object tracking method based on an implicit neural field, the method comprising:

[0007] Obtaining a current frame image and an initial pose of the current frame image;

[0008] Based on the initial pose and the pre-constructed implicit neural field, obtaining a reference image and a segmentation mask of the reference image;

[0009] An optimized pose of the current frame image is obtained according to the initial pose, the object reference image and the segmentation mask.

[0010] In some embodiments, before obtaining the current frame image and the initial pose of the current frame image, the method further includes:

[0011] Acquiring a target image, wherein the target image includes a target object and a background;

[0012] The implicit neural field is trained based on the target image to generate an implicit 3D model of the scene.

[0013] In some embodiments, obtaining a reference image and a segmentation mask of the reference image based on the initial pose and the pre-constructed implicit neural field includes:

[0014] Inputting the initial pose into the implicit neural field to obtain a reference image, wherein the reference image includes a target object and a background;

[0015] The reference image is segmented according to a preset segmentation network to obtain a segmentation mask of the reference image, wherein the segmentation mask indicates whether each pixel in the reference image belongs to the target object or the background.

[0016] In some embodiments, obtaining the optimized pose of the current frame image according to the initial pose, the object reference image, and the segmentation mask includes:

[0017] Based on a preset feature extraction network, obtaining a hierarchical feature map of the current frame image, and a hierarchical feature map and confidence of the reference image;

[0018] The initial pose is optimized according to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image, and the confidence level to obtain an optimized pose of the current frame image.

[0019] In some embodiments, the hierarchical features include first-level features and second-level features, wherein the first level is the original image size level, and the size of the second level is smaller than the size of the first level. The step of obtaining the hierarchical feature map of the current frame image based on a preset feature extraction network, and the hierarchical feature map and confidence of the reference image include:

[0020] Performing feature extraction on the reference image through the feature extraction network to obtain a first-level feature map and a first-level confidence, as well as a second-level feature map and a second-level confidence in the reference image;

[0021] The feature extraction network is used to extract features from the current frame image to obtain a first-level feature map and a second-level feature map of the current frame image.

[0022] In some embodiments, optimizing the initial pose according to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image, and the confidence level to obtain the optimized pose of the current frame image includes:

[0023] Optimizing the initial pose according to the second-level confidence, the segmentation mask, the reference image, and the second-level feature map of the current frame image to obtain a second-level pose;

[0024] The second-level pose is optimized according to the first-level confidence, the segmentation mask, the reference image, and the first-level feature map of the current frame image to obtain an optimized pose of the current frame image.

[0025] In some embodiments, optimizing the initial pose according to the second-level confidence, the segmentation mask, the reference image, and the second-level feature map of the current frame image to obtain the second-level pose includes:

[0026] Obtaining a second target 3D point from the reference image according to the second-level confidence and the segmentation mask;

[0027] Obtaining a second-level feature of the second target 3D point from the hierarchical feature map of the reference image to obtain a second-level reference feature;

[0028] Projecting the second target 3D point onto the current frame image according to the initial pose to obtain an initial projection point;

[0029] Obtaining the second-level features of the initial projection point from the second-level feature map of the current frame image to obtain a second-level initial feature;

[0030] The initial pose is optimized according to the second-level reference features and the second-level initial features to obtain the second-level pose.

[0031] In some embodiments, optimizing the second-level pose according to the first-level confidence, the segmentation mask, the reference image, and the first-level feature map of the current frame image to obtain the optimized pose of the current frame image includes:

[0032] Obtaining a first target 3D point from the reference image according to the first-level confidence and the segmentation mask;

[0033] Obtaining a first-level feature of the first target 3D point from the hierarchical feature map of the reference image to obtain a first-level reference feature;

[0034] Projecting the first target 3D point onto the current frame image according to the second-level pose to obtain an updated projection point;

[0035] Obtaining the first-level features of the updated projection point from the first-level feature map of the current frame image to obtain a first-level updated feature;

[0036] The second-level pose is optimized according to the first-level reference features and the first-level updated features to obtain an optimized pose of the current frame image.

[0037] In a second aspect, an embodiment of the present application provides an object tracking system based on an implicit neural field, the system comprising: an information acquisition module, an information matching module, and a posture optimization module:

[0038] The information acquisition module is used to acquire the current frame image and the initial posture of the current frame image;

[0039] The information matching module is used to obtain a reference image and a segmentation mask of the reference image based on the initial pose and the pre-constructed implicit neural field;

[0040] The posture optimization module is used to obtain the optimized posture of the current frame image according to the initial posture, the object reference image and the segmentation mask.

[0041] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the object tracking method based on implicit neural fields as described in the first aspect above is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the object tracking method based on implicit neural fields as described in the first aspect above.

[0043] Compared to related technologies, the object tracking method based on implicit neural fields provided in the embodiments of the present application obtains a reference image and a segmentation mask of the reference image by inputting the initial pose of the current frame image into the implicit neural field, determines the target 3D point by the segmentation mask and confidence of the reference image, optimizes the estimated initial pose based on the hierarchical features of the target 3D point in the current frame image and the reference image, and obtains the optimized pose of the current frame image, thereby solving the problem of complex implementation of the 6DOF tracking method of the object. By obtaining the reference image through the implicit neural field and performing pose optimization, no sensor or CAD model of the object is required, which reduces the complexity of the object tracking method and improves the tracking efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 is a schematic diagram of an application environment of an object tracking method based on an implicit neural field according to an embodiment of the present application;

[0046] Figure 2is a flow chart of an object tracking method based on implicit neural field according to an embodiment of the present application;

[0047] Figure 3 is a flow chart of a feature extraction method according to an embodiment of the present application;

[0048] Figure 4 is a structural block diagram of an object tracking system based on implicit neural field according to an embodiment of the present application;

[0049] Figure 5 Schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0051] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0052] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0053] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0054] The object tracking method based on implicit neural field provided in this application can be applied to Figure 1 In the application environment shown, Figure 1 is a schematic diagram of an application environment of an object tracking method based on an implicit neural field according to an embodiment of the present application, such as Figure 1 As shown. The server 10 and the terminal 11 communicate via a network. The terminal 11 captures the current frame image and sends the current frame image to the server 10. The server 10 obtains the initial pose of the current frame image based on the pose of the previous image, and matches the current frame image with the implicit neural field to obtain a reference image and the object 3D points of the reference image. The server 10 determines the optimized pose of the current frame image based on the initial pose, the reference image, and the object 3D points.

[0055] This embodiment also provides an object tracking method based on implicit neural field. Figure 2 is a flow chart of an object tracking method based on implicit neural field according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0056] Step S201: Obtain the current frame image and the initial pose of the current frame image. It should be noted that the initial pose of the current frame image can be the pose of the previous frame image, or it can be an estimated pose obtained by transforming the poses of the previous frames.

[0057] Step S202, based on the initial pose and the pre-constructed implicit neural field, obtain the reference image and the segmentation mask of the reference image. It should be noted that the segmentation mask is a binary image output by the segmentation network, in which each pixel is marked as an object or a background, and the formed mask can accurately describe the position and shape of different areas in the original image. In this embodiment, the initial pose of the current frame image is input into the pre-constructed implicit neural field, and the implicit neural field is used to render an object image as a reference image. According to the preset segmentation network, the reference image is segmented to obtain the segmentation mask of the reference image, thereby determining the pixel points representing the target object.

[0058] Step S203, obtains the optimized pose of the current frame image according to the initial pose, the reference image and the segmentation mask. In this embodiment, the features and confidences of the current frame image and the reference image are extracted through the feature extraction network, and the 2D points representing the object in the reference image are determined according to the segmentation mask, and then, according to the confidence of the reference image, the target 3D points for optimizing the pose are selected from the 3D points corresponding to these 2D points. Furthermore, according to the initial pose, the target 3D points are projected to the current frame image to obtain the hierarchical features of the target 3D points in the current frame image, and the feature residuals of the target 3D points in the reference image and the current frame image are calculated. The pose of the current frame image is optimized according to the feature residuals, and the pose corresponding to the minimum feature residual is found, and the pose is used as the optimized pose of the current frame. Because the implicit neural field represents the 3D structural information of the object, the 3D points corresponding to the 2D points in the reference image can be obtained.

[0059] Through the above steps S201 to S203, the initial pose of the current frame image is input into the implicit neural field, the reference image and the segmentation mask of the reference image are obtained, the target 3D point is determined by the segmentation mask and confidence of the reference image, and the estimated initial pose is optimized according to the hierarchical features of the target 3D point in the current frame image and the reference image to obtain the optimized pose of the current frame image, thereby solving the problem of complex implementation of the 6DOF tracking method of the object. Pose optimization through the implicit neural field does not require sensors or CAD models of objects, which reduces the complexity of the object tracking method and improves tracking efficiency. In addition, this solution optimizes the pose based on the feature residuals of the reference image and the current image. Therefore, it is only necessary to use the implicit neural field rendering once to obtain the reference image, further improving tracking efficiency.

[0060] In some embodiments, before obtaining the current frame image and the initial pose of the current frame image, the method further includes:

[0061] A target image is collected, where the target image includes a target object and a background. An implicit neural field is trained based on the target image to generate an implicit 3D model of the scene.

[0062] In this embodiment, an implicit neural field is pre-constructed. During the object tracking process, a reference image and a segmentation mask of the reference image are obtained based on the initial pose and the pre-constructed implicit neural field, specifically including: inputting the initial pose into the implicit neural field to obtain a reference image, wherein the reference image includes a target object and a background; segmenting the reference image according to a preset segmentation network to obtain a segmentation mask of the reference image, wherein the segmentation mask indicates whether each pixel in the reference image belongs to the target object or the background.

[0063] It should be noted that the implicit neural field uses a neural network to learn the field value function (Field Function) to map the three-dimensional scene to a two-dimensional image, thereby achieving high-quality image rendering. The implicit neural field can render complex three-dimensional geometries and does not require special segmentation or processing of the three-dimensional body. This method avoids the large number of calculation steps and tedious ray tracing processes in traditional rendering methods, while having higher expressiveness and faster rendering speed. Taking into account factors such as rendering speed, when training the implicit neural field of an object, you can use methods such as instant-ngp or mobile-nerf to maintain the highest possible rendering quality while reducing computing and storage requirements to adapt to the resource limitations of mobile devices.

[0064] In some embodiments, obtaining an optimized pose of a current frame image according to the initial pose, the object reference image, and the segmentation mask includes:

[0065] Based on the pre-set feature extraction network, the hierarchical feature map of the current frame image, the hierarchical feature map of the reference image and the confidence level are obtained. According to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image and the confidence level, the initial pose is optimized to obtain the optimized pose of the current frame image.

[0066] In this embodiment, before object tracking is performed, a general feature extraction network needs to be trained first. The network input is an RGB image, which is compressed to 32 or 64 by downsampling and then restored to the required size by upsampling. Image compression can adopt the structure of VGG or resnet, and restoration can adopt multi-scale fusion strategies such as FPN. The ultimate goal of training is to find two images, in which the features corresponding to the same physical point in the two images should be close, and the feature distances of different points should be far. Optionally, a superpoint training method is used, in which a reference image is obtained after a known transformation of a picture, and the correspondence between the feature points in the two pictures is determined by the known transformation, and then the features output by the two pictures are constrained in terms of distance, and finally multiple levels of features and confidence are output.

[0067] Figure 3This is a flow chart of a feature extraction method according to an embodiment of the present application. Figure 3 As shown, the process includes the following steps:

[0068] S31, input reference image;

[0069] S32, compress the reference image based on downsampling technology to obtain an image 1 / 32 of the original image size, and obtain hierarchical features of 1 / 32 size;

[0070] In S33, by downsampling and size fusion of the original image, hierarchical features and confidence levels of 1 / 16, 1 / 4, and 1 / 1 sizes are obtained based on the hierarchical features of 1 / 32 size.

[0071] It should be noted that traditional machine learning requires manual feature extraction, while the feature extraction network can automatically learn the features of the data, avoiding the tedious process of manual feature extraction; the feature extraction network uses convolutional neural networks (CNN), which can process input images of any size, so the feature extraction network can adapt to inputs of different scales; the feature extraction network can be shared between different tasks and can be adapted to new tasks through fine-tuning, which makes the feature extraction network highly transferable and can be used in various application scenarios; at the same time, since the feature extraction network uses convolution operations, it can utilize the local connectivity and parameter sharing properties of convolution, reducing the number of parameters and computational complexity in the network and improving the efficiency of the network.

[0072] In some embodiments, the hierarchical features include first-level features and second-level features, wherein the first level is the original image size level, and the size of the second level is smaller than the size of the first level. Based on a pre-set feature extraction network, obtaining the hierarchical feature map of the current frame image, and the hierarchical feature map and confidence of the reference image include:

[0073] The feature extraction network is used to extract features from the reference image to obtain the first-level feature map and the first-level confidence in the reference image, as well as the second-level feature map and the second-level confidence. The feature extraction network is used to extract features from the current frame image to obtain the first-level feature map and the second-level feature map of the current frame image.

[0074] It should be noted that both the first and second levels are obtained by downsampling and compressing the original image, and then restoring it through upsampling, but the sizes of their restorations are different. The feature level can also include a third level, and the size of the third level is smaller than that of the second level. In this embodiment, the features and confidence of the current frame image are first extracted to obtain its corresponding feature map and confidence. During the optimization process, after each pose update, the target 3D point is reprojected, and the corresponding features are obtained on the feature map through the reprojected points, and the next step of optimization is performed based on the features. When performing pose optimization, start the optimization from the level with the smallest size, and the optimization results of each layer are given to the next layer as the initial value. Finally, the original image size is optimized. In each pyramid level, the optimization can be iterated N times, or conditions can be set to terminate the optimization early.

[0075] In some embodiments, optimizing the initial pose according to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image, and the confidence level to obtain the optimized pose of the current frame image includes:

[0076] The initial pose is optimized based on the second-level confidence, segmentation mask, reference image, and second-level feature map of the current frame image to obtain the second-level pose. Specifically, the process includes: obtaining the second target 3D point from the reference image based on the second-level confidence and segmentation mask; obtaining the second-level features of the second target 3D point from the level feature map of the reference image to obtain the second-level reference features; projecting the second target 3D point onto the current frame image based on the initial pose to obtain the initial projected point; obtaining the second-level features of the initial projected point from the second-level feature map of the current frame image to obtain the second-level initial features; optimizing the initial pose based on the second-level reference features and the second-level initial features to obtain the second-level pose.

[0077] According to the first-level confidence, segmentation mask, reference image and first-level feature map of the current frame image, the second-level pose is optimized to obtain the optimized pose of the current frame image. Specifically, the method includes: obtaining the first target 3D point from the reference image according to the first-level confidence and segmentation mask, obtaining the first-level features of the first target 3D point from the hierarchical feature map of the reference image, and obtaining the first-level reference features; according to the second-level pose, the first target 3D point is projected to the current frame image to obtain the updated projection point; from the first-level feature map of the current frame image, the first-level features of the updated projection point are obtained to obtain the first-level updated features; according to the first-level reference features and the first-level updated features, the second-level pose is optimized to obtain the optimized pose of the current frame image.

[0078] In this embodiment, the segmentation mask is used to determine the pixels representing the target object in the reference image. A preset confidence threshold is then applied to the pixels in the reference image with a confidence score above the threshold, and the corresponding 3D points are used as the target 3D points. The pose is then optimized based on the feature residuals of the target 3D points in the current frame and the reference image.

[0079] For example, the feature extraction network extracts three levels of features, each at 1 / 16, 1 / 4, and 1 / 1 of the original image size, to construct a feature pyramid. Before pose optimization, the three-level feature maps of the current frame, along with the three-level feature maps and confidence scores of the reference image, are obtained. During pose optimization, optimization begins at the lowest-resolution feature level, the 1 / 16 feature level, and each level is iterated N times.

[0080] Based on the segmentation mask of the reference image, the 3D points representing the object are determined. Then, based on the 1 / 16 level confidence, the 3D points corresponding to the 1 / 16 level are obtained from the 3D points representing the object. The reference features corresponding to the 1 / 16 level of these 3D points are obtained from the level feature map of the reference image. Then, based on the initial pose, these 3D points are projected onto the current frame image to obtain the corresponding 2D points. The features of these 2D points are obtained from the 1 / 16 level feature map of the current frame image. Based on the features of these 2D points and the 1 / 16 level reference features, a first optimization of the 1 / 16 level is performed to obtain a new pose. Further, projection is performed based on the new pose to re-determine the 2D point corresponding to the target 3D point on the current image and obtain new 2D point features. Based on the new 2D features and the features of the target 3D point, optimization is continued to update the pose.

[0081] After N optimizations, the optimization process moves to the 1 / 4 level. Based on the confidence level of the 1 / 4 level, the 3D points for optimization in the 1 / 4 level are obtained. The reference features of the 1 / 4 level corresponding to these 3D points are obtained from the hierarchical feature map of the reference image. The 3D points of the 1 / 4 level are projected onto the current frame image through the pose projection of the previous level to obtain the 1 / 4 level features of the current frame image. The 1 / 4 level features are obtained by projecting the target 3D points through the pose projection of the previous level. Based on the 1 / 4 level features of the current frame image and the 1 / 4 level reference features, the pose obtained at the previous level is optimized. The process is also iterated N times. Finally, the original image size is optimized. The optimization process is similar to the optimization process of the 1 / 16 level and will not be repeated here.

[0082] In this embodiment, extracting multi-scale features through feature pyramid can improve the scale adaptability, accuracy and robustness of pose estimation:

[0083] Different objects or scenes may have different scale variations, and feature extraction at a single scale may not be able to effectively capture these variations. Extracting multi-scale features through a feature pyramid can adapt to the variations of objects at different scales and provide a more comprehensive feature representation.

[0084] When images are blurred, have low resolution, or are blocked, features at a single scale may be affected, leading to inaccurate pose estimation. By using multi-scale features, feature information at other scales can be used to compensate for the shortcomings of a single scale, thereby improving the robustness of the algorithm.

[0085] Due to the scale variation of the target, features at a single scale may not be able to accurately match the corresponding points of the target. By extracting multi-scale features, matching points can be searched at different scales, increasing the accuracy and reliability of the matching.

[0086] It should be noted that feature pyramids can extract features at different scales, but this does not necessarily mean that feature extraction needs to be performed on the entire image. Typically, higher-level pyramid features can be calculated from lower-level features through downsampling or pooling operations, thereby reducing the amount of computation and improving the efficiency of the algorithm.

[0087] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0088] This embodiment also provides an object tracking system based on an implicit neural field, which is used to implement the above-mentioned embodiments and preferred implementation methods, and will not be repeated here. As used below, the terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0089] Figure 4 is a structural block diagram of an object tracking system based on implicit neural field according to an embodiment of the present application, such as Figure 4 As shown, the system includes: an information acquisition module 41, an information matching module 42 and a posture optimization module 43:

[0090] An information acquisition module 41 is used to acquire the current frame image and the initial position and posture of the current frame image;

[0091] An information matching module 42 is used to obtain a reference image and a segmentation mask of the reference image based on the initial pose and the pre-constructed implicit neural field;

[0092] The pose optimization module 43 is used to obtain the optimized pose of the current frame image according to the initial pose, the reference image and the segmentation mask.

[0093] In this embodiment, the information acquisition module 41 acquires the current frame image and the initial pose of the current frame image. The information matching module 42 matches the initial pose with the implicit neural field to obtain a reference image and a segmentation mask of the reference image. The pose optimization module 43 obtains the optimized pose of the current frame image based on the initial pose, reference image, and segmentation mask, thereby resolving the complexity of the 6DOF tracking method for objects. Only a reference image needs to be rendered using the implicit neural field for pose optimization, without the need for sensors or CAD models of objects. This reduces the complexity of the object tracking method and improves tracking efficiency.

[0094] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0095] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0096] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0097] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0098] S1, obtain the current frame image and the initial pose of the current frame image.

[0099] S2, based on the initial pose and the pre-built implicit neural field, obtains the reference image and the segmentation mask of the reference image.

[0100] S3, obtain the optimized pose of the current frame image according to the initial pose, reference image and segmentation mask.

[0101] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0102] In one embodiment, Figure 5 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 5As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an object tracking method based on an implicit neural field is implemented.

[0103] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0104] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0105] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An object tracking method based on implicit neural field, characterized in that: The method comprises: Obtaining a current frame image and an initial pose of the current frame image; Based on the initial pose and the pre-constructed implicit neural field, obtaining a reference image and a segmentation mask of the reference image; Obtaining an optimized pose of the current frame image according to the initial pose, the reference image, and the segmentation mask, comprising: Based on a preset feature extraction network, obtaining a hierarchical feature map of the current frame image, and a hierarchical feature map and confidence of the reference image; The initial pose is optimized according to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image, and the confidence level to obtain an optimized pose of the current frame image.

2. The method according to claim 1, characterized in that Before acquiring the current frame image and the initial pose of the current frame image, the method further includes: Acquiring a target image, wherein the target image includes a target object and a background; The implicit neural field is trained based on the target image to generate an implicit 3D model of the scene.

3. The method according to claim 1, characterized in that Based on the initial pose and the pre-constructed implicit neural field, obtaining a reference image and a segmentation mask of the reference image includes: Inputting the initial pose into the implicit neural field to obtain a reference image, wherein the reference image includes a target object and a background; The reference image is segmented according to a preset segmentation network to obtain a segmentation mask of the reference image, wherein the segmentation mask indicates whether each pixel in the reference image belongs to the target object or the background.

4. The method according to claim 1, wherein The hierarchical features include first-level features and second-level features, wherein the first level is the original image size level, and the size of the second level is smaller than the size of the first level. The hierarchical feature map of the current frame image, the hierarchical feature map of the reference image, and the confidence level based on the preset feature extraction network include: Performing feature extraction on the reference image through the feature extraction network to obtain a first-level feature map and a first-level confidence, as well as a second-level feature map and a second-level confidence in the reference image; The feature extraction network is used to extract features from the current frame image to obtain a first-level feature map and a second-level feature map of the current frame image.

5. The method according to claim 4, characterized in that Optimizing the initial pose according to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image, and the confidence level to obtain the optimized pose of the current frame image includes: Optimizing the initial pose according to the second-level confidence, the segmentation mask, the reference image, and the second-level feature map of the current frame image to obtain a second-level pose; The second-level pose is optimized according to the first-level confidence, the segmentation mask, the reference image, and the first-level feature map of the current frame image to obtain an optimized pose of the current frame image.

6. The method according to claim 5, characterized in that Optimizing the initial pose according to the second-level confidence, the segmentation mask, the reference image, and the second-level feature map of the current frame image to obtain the second-level pose includes: Obtaining a second target 3D point from the reference image according to the second-level confidence and the segmentation mask; Obtaining a second-level feature of the second target 3D point from the hierarchical feature map of the reference image to obtain a second-level reference feature; Projecting the second target 3D point onto the current frame image according to the initial pose to obtain an initial projection point; Obtaining the second-level features of the initial projection point from the second-level feature map of the current frame image to obtain a second-level initial feature; The initial pose is optimized according to the second-level reference features and the second-level initial features to obtain the second-level pose.

7. The method according to claim 5, characterized in that Optimizing the second-level pose according to the first-level confidence, the segmentation mask, the reference image, and the first-level feature map of the current frame image to obtain the optimized pose of the current frame image includes: Obtaining a first target 3D point from the reference image according to the first-level confidence and the segmentation mask; Obtaining a first-level feature of the first target 3D point from the hierarchical feature map of the reference image to obtain a first-level reference feature; Projecting the first target 3D point onto the current frame image according to the second-level pose to obtain an updated projection point; Obtaining the first-level features of the updated projection point from the first-level feature map of the current frame image to obtain a first-level updated feature; The second-level pose is optimized according to the first-level reference features and the first-level updated features to obtain an optimized pose of the current frame image.

8. An object tracking system based on implicit neural field, characterized in that: The system includes: an information acquisition module, an information matching module and a posture optimization module: The information acquisition module is used to acquire the current frame image and the initial posture of the current frame image; The information matching module is used to obtain a reference image and a segmentation mask of the reference image based on the initial pose and the pre-constructed implicit neural field; The pose optimization module is configured to obtain an optimized pose of the current frame image based on the initial pose, the reference image, and the segmentation mask, comprising: Based on a preset feature extraction network, obtaining a hierarchical feature map of the current frame image, and a hierarchical feature map and confidence of the reference image; The initial pose is optimized according to the hierarchical feature map of the current frame image, the segmentation mask, the hierarchical feature map of the reference image, and the confidence level to obtain an optimized pose of the current frame image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the object tracking method based on implicit neural field according to any one of claims 1 to 7 is implemented.

Images