Augmented reality anchor point placement method, device, electronic device and readable storage medium
Through the conversion relationship between the NeRF model and the initial AR point cloud, the first pose data of the target image is converted into the second pose data, and anchor points are placed in the AR point cloud, which solves the flexibility of AR point cloud anchor points in the existing technology. The problem of poor performance enables remote placement of anchor points, improving the flexibility and accuracy of AR point clouds.
Patent Information
- Application Number
- CN202410221234.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-02-28
AI Technical Summary
In the prior art, the placement flexibility of augmented reality AR point cloud anchor points is poor and cannot be placed remotely, resulting in limited application of anchor points in different scenarios.
By obtaining the neural radiation field NeRF model and initial AR point cloud of the scene to be created, the first pose data of the target image is obtained based on the NeRF model, and it is converted into the second digit through the conversion relationship between the NeRF model and the initial AR point cloud. Pose data, place anchor points in the initial AR point cloud to generate the target AR point cloud.
It realizes remote placement of anchor points without actually placing anchor points, which improves the flexibility of AR point cloud anchor points, so that the target AR point cloud contains anchor points corresponding to the target image, enhancing the flexibility of AR point cloud and Accuracy.
Smart Images

Figure CN118135156B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning technology, and in particular to an augmented reality anchor point placement method, device, electronic device, and readable storage medium. Background Art
[0002] Augmented Reality (AR) is one of the key technologies for virtual experience. Through AR mapping, users can browse different three-dimensional (3D) scenes through terminals. In related technologies, AR point clouds are built through images of the scene to be mapped, and anchor points are placed to display virtual objects corresponding to the anchor points in the AR point cloud. However, in related technologies, anchor points must be placed in the actual scene, and anchor points cannot be placed remotely, resulting in poor flexibility in placing anchor points in AR point clouds.
[0003] It can be seen that the related technology has the problem of poor flexibility in placing anchor points of AR point clouds. Summary of the invention
[0004] The embodiments of the present application provide an augmented reality anchor point placement method, device, electronic device and readable storage medium to solve the problem of poor flexibility in placing anchor points of AR point clouds in the related art.
[0005] To solve the above problems, this application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides an augmented reality anchor point placement method, comprising:
[0007] Obtain the Neural Radiation Field (NeRF) model and initial augmented reality (AR) point cloud of the scene to be mapped;
[0008] Acquire M target images and first position data of the M target images based on the NeRF model, where M is an integer greater than or equal to 1;
[0009] Based on the conversion relationship between the NeRF model and the initial AR point cloud, converting the first pose data of the M target images into second pose data of the M target images;
[0010] Anchor points are placed in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud.
[0011] In a second aspect, an embodiment of the present application further provides an augmented reality anchor point placement device, comprising:
[0012] The first acquisition module is used to obtain the NeRF model and initial augmented reality AR point cloud of the scene to be mapped;
[0013] A second acquisition module is used to acquire M target images and first position data of the M target images based on the NeRF model, where M is an integer greater than or equal to 1;
[0014] A conversion module, configured to convert the first pose data of the M target images into second pose data of the M target images based on a conversion relationship between the NeRF model and the initial AR point cloud;
[0015] A generation module is used to place anchor points in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud.
[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps in the augmented reality anchor point placement method described in the first aspect above are implemented.
[0017] In a fourth aspect, an embodiment of the present application further provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the augmented reality anchor point placement method as described in the first aspect above.
[0018] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the augmented reality anchor point placement method as described in the first aspect above.
[0019] In the embodiment of the present application, the NeRF model and the initial AR point cloud of the scene to be mapped are obtained; based on the NeRF model, M target images and the first pose data of the M target images are obtained, where M is an integer greater than or equal to 1; based on the conversion relationship between the NeRF model and the initial AR point cloud, the first pose data of the M target images are converted into the second pose data of the M target images; anchor points are placed in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud. In this way, the NeRF model generates M target images, the M target images correspond to anchor points, and the first pose data of the target images are converted into the second pose data to generate a target AR point cloud, so that the target AR point cloud includes the anchor points corresponding to the target images, so that the anchor points are not actually placed, but are placed remotely, which improves the flexibility of the AR point cloud anchor point placement. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 is a flow chart of an augmented reality anchor point placement method provided in an embodiment of the present application;
[0022] Figure 2 is a schematic diagram of posture data of N images provided in an embodiment of the present application;
[0023] Figure 3 is a schematic diagram of constructing an initial AR point cloud provided in an embodiment of the present application;
[0024] Figure 4 is a schematic diagram of constructing a target AR point cloud provided in an embodiment of the present application;
[0025] Figure 5 This is a schematic diagram of the interaction between the cloud and the client provided in an embodiment of the present application;
[0026] Figure 6 is a structural diagram of an augmented reality anchor point placement device provided in an embodiment of the present application;
[0027] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0029] See also Figure 1 , Figure 1 is a flow chart of an augmented reality anchor point placement method provided by an embodiment of the present application, such as Figure 1 As shown, the following steps are included:
[0030] Step 101: Obtain a Neural Radiance Fields (NeRF) point cloud and an initial augmented reality (AR) point cloud of the scene to be mapped.
[0031] Step 102: acquiring M target images and first position data of the M target images based on the NeRF model, where M is an integer greater than or equal to 1;
[0032] Step 103: based on the conversion relationship between the NeRF model and the initial AR point cloud, convert the first pose data of the M target images into the second pose data of the M target images;
[0033] Step 104: placing anchor points in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud.
[0034] The above NeRF model and the initial AR point cloud are models / point clouds of the same scene, wherein the NeRF model is a model trained based on the NeRF point cloud, and is used to render images of the scene to be constructed under different pose data conditions; the initial AR point cloud is a three-dimensional point cloud model, and the initial AR point cloud is a point cloud that has not been constructed through anchor points. In an embodiment of the present application, a target image corresponding to the second pose data is generated by the NeRF model, and the anchor points corresponding to the M target images are placed in the initial AR point cloud to generate a target AR point cloud, so that the target AR point cloud includes the anchor points corresponding to the M target images.
[0035] The above-mentioned M target images are images rendered by the NeRF model. The first position data corresponding to each target image can be the same or different. By placing the anchor points corresponding to the M target images in the initial AR point cloud, a target AR point cloud is generated so that the target AR point cloud includes the anchor points corresponding to the M target images.
[0036] The conversion relationship between the above NeRF model and the initial AR point cloud is the conversion relationship between the pose data of the image at the same position in the scene in the NeRF model and the initial AR point cloud. It should be noted that the first pose data is the pose data of the target image in the NeRF model, which cannot be directly used to generate the target AR point cloud. It is necessary to convert the first pose data of the target image in the NeRF model into the second pose data of the target image in the initial AR point cloud based on the conversion relationship between the NeRF model and the initial AR point cloud, so as to realize the generation of the target AR point cloud by combining the second pose data of the target image with the initial AR point cloud.
[0037] The above-mentioned target AR point cloud is an AR point cloud generated after placing anchor points in the initial AR point cloud based on M target images and the second pose data of the M target images. The anchor points corresponding to the M target images are placed in the initial AR point cloud through the M target images.
[0038] In an embodiment of the present application, a NeRF model and an initial AR point cloud of a scene to be mapped are obtained; based on the NeRF model, M target images and the first pose data of the M target images are obtained, where M is an integer greater than or equal to 1; based on the conversion relationship between the NeRF model and the initial AR point cloud, the first pose data of the M target images are converted into the second pose data of the M target images; anchor points are placed in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud. In this way, the NeRF model generates M target images, the M target images correspond to anchor points, and the first pose data of the target images are converted into the second pose data to generate a target AR point cloud, so that the target AR point cloud includes the anchor points corresponding to the M target images, so that the anchor points are not actually placed, but are placed remotely, thereby improving the flexibility of the AR point cloud anchor point placement.
[0039] In one embodiment, the step of obtaining a neural radiation field NeRF model and an initial augmented reality AR point cloud of a scene to be mapped includes:
[0040] Obtain an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1;
[0041] The NeRF model is constructed based on the image set and the camera intrinsic parameters, and the initial AR point cloud is constructed based on the image set and the camera intrinsic parameters.
[0042] The N images in the above image set are all images of the scene to be mapped. The N images can be obtained by extracting frames from a video of the scene to be mapped. For example, the N images are obtained by extracting frames from the video at intervals of 1 second. Figure 2 As shown in the figure, the N images are images from different perspectives. By analyzing each image and the intrinsic parameters of the camera that takes the image, the pose data of each image is calculated to construct the NeRF model and AR point cloud.
[0043] It should be noted that there are differences between the process of building the NeRF model and building the initial AR point cloud, and the pose of the same image in different point clouds is different.
[0044] The process of building the NeRF model is as follows: calculate the pose data of each image in N images based on colmap and camera intrinsic parameters; form the translation and rotation matrix of each image based on the pose data, and generate the training file transforms.json based on the translation and rotation matrix. The training file is used as the training data of the NeRF model to train and generate the NeRF model of the scene to be built.
[0045] In addition, if Figure 3As shown in FIG. 1 , the process of constructing the initial AR point cloud is specifically as follows: extracting two-dimensional feature points from N images, matching the two-dimensional feature points between every two images, achieving the correspondence between the two-dimensional (2D) feature points between every two images, and then achieving the correspondence between the two-dimensional feature points in the image and the point cloud of the three-dimensional scene, calculating the pose data of each image in combination with the camera intrinsic parameters, and calculating the coordinates of the three-dimensional point cloud to generate the initial AR point cloud, which does not include anchor points.
[0046] Furthermore, if Figure 4 As shown, ARFoundation can also be used to determine the pose data of M target images, and generate a target AR point cloud based on the M target images, the second pose data of the M target images, and the image data. At this time, there is no need to calculate the pose data through two-dimensional feature points. Instead, the coordinates of the three-dimensional point cloud are calculated through the existing second pose data to generate the target AR point cloud. At this time, the target AR point cloud is positioned through the target image, thereby effectively improving the accuracy of the target AR point cloud.
[0047] In one embodiment, before converting the first pose data of the M target images into the second pose data of the M target images based on the conversion relationship between the NeRF model and the initial AR point cloud, the method includes:
[0048] Calculating a first Euclidean distance of a plurality of co-viewed images based on the NeRF model, and calculating a second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud, wherein the plurality of co-viewed images are images having a common area in the N images;
[0049] The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance, or the conversion relationship includes a conversion matrix, which is a matrix constructed based on a difference between the first Euclidean distance and the second Euclidean distance.
[0050] It should be noted that after the NeRF model and the initial AR point cloud are constructed by the image set and the camera intrinsic parameters, the first pose data of the target image rendered by the NeRF model cannot be directly used as the anchor point because the pose data of the same image in the NeRF model and in the initial AR point cloud are different. It is necessary to convert the first pose data of the target image into the second pose data based on the conversion relationship. The second pose data is the pose data of the target image in the initial AR point cloud. The target image, the second pose data of the target image and the initial AR point cloud are used to generate the target AR point cloud to improve the accuracy of the target AR point cloud.
[0051] The above-mentioned multiple co-viewing images are images screened from M target images based on the common viewing algorithm. There are common areas in the multiple co-viewing images. The first Euclidean distance of the multiple co-viewing images is calculated based on the NeRF model, and the first Euclidean distance is used to characterize the position data of the target image in the NeRF model; the second Euclidean distance of the multiple co-viewing images is calculated based on the initial AR point cloud, and the second Euclidean distance is used to characterize the position data of the target image in the AR point cloud. The conversion relationship between the NeRF model and the initial AR point cloud is determined by the first Euclidean distance and the second Euclidean distance.
[0052] Exemplarily, after the first Euclidean distance and the second Euclidean distance are calculated, the first pose data of the target image can be converted into the second pose data by the ratio of the first Euclidean distance and the second Euclidean distance, for example, the quotient of the first pose data of the target image and the ratio is set as the second pose data of the target image. Optionally, the conversion relationship can also include the ratio of the second Euclidean distance to the first Euclidean distance, and the first pose data of the target image is converted into the second pose data by the ratio of the second Euclidean distance to the first Euclidean distance.
[0053] In addition, the conversion relationship may further include a conversion matrix, which is a matrix constructed based on the difference between the first Euclidean distance and the second Euclidean distance, and the first pose data of the target image is converted into the second pose data through the conversion matrix.
[0054] In an embodiment of the present application, the first Euclidean distance of multiple co-visible images is calculated based on the NeRF model, and the second Euclidean distance of multiple co-visible images is calculated based on the initial AR point cloud, so that the conversion relationship between the NeRF model and the initial AR point cloud is determined by the ratio of the first Euclidean distance to the second Euclidean distance, or the conversion relationship between the NeRF model and the initial AR point cloud is determined by the difference between the first Euclidean distance and the second Euclidean distance, thereby realizing the conversion of the first pose data of the target image into the second pose data based on the conversion relationship between the NeRF model and the initial AR point cloud.
[0055] In one embodiment, the first Euclidean distance includes a plurality of first sub-distances, the second Euclidean distance includes a plurality of second sub-distances, and the first Euclidean distance of the plurality of co-viewing images is calculated based on the NeRF model, comprising:
[0056] Calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the NeRF model to obtain a plurality of first sub-distances;
[0057] The computing the second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud comprises:
[0058] Computing the Euclidean distance between every two images in the plurality of co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances;
[0059] The conversion matrix is obtained as follows:
[0060] Get the initial matrix;
[0061] Calculate the difference between the first sub-distance and the second sub-distance of every two images to obtain multiple difference values;
[0062] The multiple difference values are input into the initial matrix for iteration to obtain the conversion matrix.
[0063] The difference between the first sub-distance and the second sub-distance is used to characterize the difference between the Euclidean distance of the two images in the NeRF model and in the initial AR point cloud. A plurality of differences are obtained by calculating the difference between the first sub-distance and the second sub-distance of each two images. The plurality of differences are then input into the initial matrix for iteration to obtain a conversion matrix, so as to convert the first pose data into the second pose data through the conversion matrix.
[0064] The multiple difference values are input into the initial matrix for iteration, and the iteration can be based on the least square method to obtain the conversion matrix.
[0065] Specifically, an initial matrix is initialized, which is a unit matrix. Then, based on the NeRF model and the initial AR point cloud computing target image A1...target image An to target image A, the difference between the NeRF model and the initial AR point Euclidean distance is calculated, which is recorded as ΔD A , and calculate ΔD repeatedly B , ΔD C The Euclidean distance difference between every two target images is equal. The difference is input into the initial matrix, and finally the initial matrix is continuously optimized and iterated based on the least squares method to obtain the conversion matrix.
[0066] In one embodiment, calculating the first Euclidean distance of the plurality of co-viewing images based on the NeRF model includes:
[0067] Calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the NeRF model to obtain a plurality of first sub-distances;
[0068] Setting the sum of the multiple first sub-distances to be the first Euclidean distance;
[0069] The computing the second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud comprises:
[0070] Computing the Euclidean distance between every two images in the plurality of co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances;
[0071] The sum of the multiple second sub-distances is set as the second Euclidean distance.
[0072] There is a Euclidean distance between every two co-viewing images in the above-mentioned multiple co-viewing images, and the multiple co-viewing images may include more than two images, that is, there are multiple Euclidean distances. It is necessary to combine the Euclidean distance between every two co-viewing images in all the co-viewing images to obtain the first Euclidean distance or the second Euclidean distance.
[0073] Specifically, the Euclidean distance between each two images in a plurality of co-viewed images is calculated based on the NeRF model to obtain a plurality of first sub-distances, and then the sum of the plurality of first sub-distances is set as the first Euclidean distance; based on the initial AR point cloud, the Euclidean distance between each two images in a plurality of co-viewed images is calculated to obtain a plurality of second sub-distances, and then the sum of the plurality of second sub-distances is set as the second Euclidean distance, so as to realize the conversion relationship between the NeRF model and the initial AR point cloud through the first Euclidean distance and the second Euclidean distance.
[0074] In one embodiment, the acquiring M target images and the first pose data of the M target images based on the NeRF model includes:
[0075] Obtain multiple first position data;
[0076] Rendering the plurality of first pose data based on the NeRF model to obtain an image corresponding to each first pose data in the plurality of first pose data;
[0077] Calculate the peak signal-to-noise ratio of the image corresponding to each first pose data, and screen the images whose peak signal-to-noise ratio is greater than a set value among the images corresponding to each first pose data to obtain the M target images and the first pose data of the M target images.
[0078] The above-mentioned Peak Signal-to-Noise Ratio (PSNR) is used to characterize the rendering quality of the image. In the process of image rendering by the NeRF model, the PSNR of the rendered image is low, the rendering quality of the image is poor, and the accuracy of the first pose data corresponding to the image in the NeRF model is low. The image cannot be used as a target image for subsequent target AR point cloud generation. It is necessary to screen and obtain M target images with higher image rendering quality, thereby improving the accuracy of the target point cloud data.
[0079] Specifically, multiple first pose data are first obtained, and the image corresponding to each first pose data is obtained by rendering based on the NeRF model. Then, based on the PSNR of the image corresponding to each first pose data, M target images are screened. The PSNR of the M target images is greater than the set value. At this time, the first pose data corresponding to each target image in the M target images has a high accuracy. The first pose data of the M target images are converted into second pose data. The accuracy of the second pose data is high, so that the accuracy of the anchor point in the target AR point cloud is higher.
[0080] In one embodiment, the obtaining of the plurality of first pose data includes:
[0081] Sending an initial image and initial pose data corresponding to the initial image to the client;
[0082] Receive multiple first pose data, where the multiple first pose data are obtained by the client based on multiple displacements superimposed on the initial pose data, and the multiple displacements are multiple displacements obtained by the client in response to user operations after receiving the initial image and the initial pose data corresponding to the initial image.
[0083] The above-mentioned multiple displacements are displacements generated by the client roaming in the three-dimensional image provided by the NeRF model. The above-mentioned client is a client that generates multiple first-position data. Before generating multiple first-position data, the client receives the initial image and the initial pose data corresponding to the initial image. The initial image is an image rendered by the NeRF model, and the initial pose data is the pose data of the initial image in the NeRF model. After the client receives the initial image and the initial pose data corresponding to the initial image, it can move based on the initial position data, and superimpose the displacement based on the movement on the initial pose data to obtain multiple first-position data.
[0084] For example, Figure 5 As shown in the figure, after the NeRF model is trained in the cloud, the cloud receives the first pose data sent by the terminal through the server, and then renders the image corresponding to each first pose data based on the NeRF model, and sends the image to the client through the server to calculate the PSNR of the image, and then screen out M target images.
[0085] In an embodiment of the present application, an initial image and initial pose data corresponding to the initial image are sent to a client; multiple first pose data are received, where the multiple first pose data are obtained by the client based on multiple displacements superimposed on the initial pose data, and the multiple displacements are multiple displacements obtained by the client in response to user operations after receiving the initial image and the initial pose data corresponding to the initial image. In this way, multiple first pose data are obtained through interaction between the cloud and the client, and M target images are screened.
[0086] See also Figure 6 , Figure 6 is a structural diagram of an augmented reality anchor point placement device provided in an embodiment of the present application, such as Figure 6 As shown, the augmented reality anchor point placement device 600 includes:
[0087] The first acquisition module 601 is used to acquire a neural radiation field NeRF model and an initial augmented reality AR point cloud of the scene to be mapped;
[0088] A second acquisition module 602 is used to acquire M target images and first position data of the M target images based on the NeRF model, where M is an integer greater than or equal to 1;
[0089] A conversion module 603 is used to convert the first pose data of the M target images into second pose data of the M target images based on the conversion relationship between the NeRF model and the initial AR point cloud;
[0090] The generating module 604 is configured to place anchor points in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud.
[0091] In one embodiment, the first acquisition module 601 includes:
[0092] A first acquisition unit is used to acquire an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1;
[0093] A construction unit is used to construct the NeRF model based on the image set and the camera intrinsic parameters, and to construct the initial AR point cloud based on the image set and the camera intrinsic parameters.
[0094] In one embodiment, before the conversion module 603, the augmented reality anchor point placement device 600 includes:
[0095] a calculation module, configured to calculate a first Euclidean distance of a plurality of co-viewed images based on the NeRF model, and to calculate a second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud, wherein the plurality of co-viewed images are images having a common area in the N images;
[0096] The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance.
[0097] In one embodiment, the first Euclidean distance includes a plurality of first sub-distances, the second Euclidean distance includes a plurality of second sub-distances, and the calculation module includes:
[0098] A first calculation unit, configured to calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the NeRF model to obtain a plurality of first sub-distances;
[0099] A second calculation unit is configured to calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances;
[0100] The conversion matrix is obtained as follows:
[0101] Get the initial matrix;
[0102] Calculate the difference between the first sub-distance and the second sub-distance of every two images to obtain multiple difference values;
[0103] The multiple difference values are input into the initial matrix for iteration to obtain the conversion matrix.
[0104] In one embodiment, the calculation module includes:
[0105] A third calculation unit is used to calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the NeRF model to obtain a plurality of first sub-distances;
[0106] a fourth calculation unit, configured to set the sum of the plurality of first sub-distances as the first Euclidean distance;
[0107] A fifth calculation unit, configured to calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances;
[0108] A sixth calculation unit is configured to set the sum of the multiple second sub-distances as the second Euclidean distance.
[0109] In one embodiment, the second acquisition module 602 includes:
[0110] A second acquisition unit, used for acquiring a plurality of first pose data;
[0111] A rendering unit, used for rendering the plurality of first pose data based on the NeRF model to obtain an image corresponding to each first pose data in the plurality of first pose data;
[0112] A screening unit is used to calculate the peak signal-to-noise ratio of the image corresponding to each first position data, and screen the images corresponding to each first position data whose peak signal-to-noise ratio is greater than a set value to obtain the M target images and the first position data of the M target images.
[0113] In one embodiment, the second acquiring unit includes:
[0114] A sending subunit, used for sending an initial image and initial posture data corresponding to the initial image to a client;
[0115] A receiving subunit is used to receive multiple first pose data, where the multiple first pose data are obtained by the client based on multiple displacements superimposed on the initial pose data, and the multiple displacements are multiple displacements obtained by the client in response to user operations after receiving the initial image and the initial pose data corresponding to the initial image.
[0116] The augmented reality anchor point placement device provided in the embodiment of the present application is capable of implementing each process of each embodiment of the above-mentioned augmented reality anchor point placement method. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.
[0117] It should be noted that the augmented reality anchor point placement device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in an electronic device.
[0118] The present application also provides an electronic device, see Figure 7 , Figure 7 701, a processor 702, and a program or instruction stored in the memory 701 and executed by the processor 702. Figure 1 Any steps in the corresponding method embodiments and achieving the same beneficial effects will not be repeated here.
[0119] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a graphics processing unit (GPU).
[0120] Those skilled in the art will appreciate that all or part of the steps of implementing the above-mentioned embodiment method can be completed by hardware associated with program instructions, and the program can be stored in a readable medium.
[0121] The present application also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned Figure 1Any steps in the corresponding method embodiments can achieve the same technical effect, and will not be repeated here to avoid repetition. The readable storage medium is a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0122] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 Any steps in the corresponding method embodiments can achieve the same technical effect and will not be repeated here to avoid repetition.
[0123] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing comprising independent A, independent B, independent C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations that A, B and C all exist.
[0124] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a second terminal device, etc.) to execute the methods of each embodiment of the present application.
[0126] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for placing an augmented reality anchor point, characterized in that: include: Obtain the Neural Radiation Field (NeRF) model and initial augmented reality (AR) point cloud of the scene to be mapped; Acquire M target images and first position data of the M target images based on the NeRF model, where M is an integer greater than or equal to 1; Based on the conversion relationship between the NeRF model and the initial AR point cloud, converting the first pose data of the M target images into second pose data of the M target images; Placing anchor points in the initial AR point cloud based on the M target images and second pose data of the M target images to generate a target AR point cloud; The step of obtaining the NeRF model and the augmented reality point cloud of the scene to be mapped includes: Obtain an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1; Constructing the NeRF model based on the image set and the camera intrinsic parameters, and constructing the initial AR point cloud based on the image set and the camera intrinsic parameters; Before converting the first pose data of the M target images into the second pose data of the M target images based on the conversion relationship between the NeRF model and the initial AR point cloud, the method includes: Calculating a first Euclidean distance of a plurality of co-viewed images based on the NeRF model, and calculating a second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud, wherein the plurality of co-viewed images are images having a common area in the N images; The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance, or the conversion relationship includes a conversion matrix, which is a matrix constructed based on a difference between the first Euclidean distance and the second Euclidean distance.
2. The method according to claim 1, characterized in that The first Euclidean distance includes a plurality of first sub-distances, the second Euclidean distance includes a plurality of second sub-distances, and the first Euclidean distance of the plurality of co-viewing images calculated based on the NeRF model includes: Calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the NeRF model to obtain a plurality of first sub-distances; The computing the second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud comprises: Computing the Euclidean distance between every two images in the plurality of co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances; The conversion matrix is obtained as follows: Get the initial matrix; Calculate the difference between the first sub-distance and the second sub-distance of every two images to obtain multiple difference values; The multiple difference values are input into the initial matrix for iteration to obtain the conversion matrix.
3. The method according to claim 1, characterized in that The calculating the first Euclidean distance of the plurality of co-viewed images based on the NeRF model comprises: Calculate the Euclidean distance between every two images in the plurality of co-viewed images based on the NeRF model to obtain a plurality of first sub-distances; Setting the sum of the multiple first sub-distances to be the first Euclidean distance; The computing the second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud comprises: Computing the Euclidean distance between every two images in the plurality of co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances; The sum of the multiple second sub-distances is set as the second Euclidean distance.
4. The method according to claim 1, characterized in that The acquiring M target images and first position data of the M target images based on the NeRF model includes: Obtain multiple first position data; Rendering the plurality of first pose data based on the NeRF model to obtain an image corresponding to each first pose data in the plurality of first pose data; Calculate the peak signal-to-noise ratio of the image corresponding to each first pose data, and screen the images whose peak signal-to-noise ratio is greater than a set value among the images corresponding to each first pose data to obtain the M target images and the first pose data of the M target images.
5. The method according to claim 4, characterized in that The obtaining of a plurality of first pose data comprises: Sending an initial image and initial pose data corresponding to the initial image to the client; Receive multiple first pose data, where the multiple first pose data are obtained by the client based on multiple displacements superimposed on the initial pose data, and the multiple displacements are multiple displacements obtained by the client in response to user operations after receiving the initial image and the initial pose data corresponding to the initial image.
6. An augmented reality anchor point placement device, characterized in that: include: The first acquisition module is used to obtain the NeRF model and initial augmented reality AR point cloud of the scene to be mapped; A second acquisition module is used to acquire M target images and first position data of the M target images based on the NeRF model, where M is an integer greater than or equal to 1; A conversion module, configured to convert the first pose data of the M target images into second pose data of the M target images based on a conversion relationship between the NeRF model and the initial AR point cloud; A generating module, configured to place anchor points in the initial AR point cloud based on the M target images and the second pose data of the M target images to generate a target AR point cloud; The first acquisition module includes: A first acquisition unit is used to acquire an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1; A construction unit, configured to construct the NeRF model based on the image set and the camera intrinsic parameters, and to construct the initial AR point cloud based on the image set and the camera intrinsic parameters; Before the conversion module, the augmented reality anchor point placement device includes: a calculation module, configured to calculate a first Euclidean distance of a plurality of co-viewed images based on the NeRF model, and to calculate a second Euclidean distance of the plurality of co-viewed images based on the initial AR point cloud, wherein the plurality of co-viewed images are images having a common area in the N images; The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance, or the conversion relationship includes a conversion matrix, which is a matrix constructed based on a difference between the first Euclidean distance and the second Euclidean distance.
7. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the method for placing an augmented reality anchor point as claimed in any one of claims 1 to 5.
8. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the augmented reality anchor point placement method according to any one of claims 1 to 5 are implemented.
9. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the augmented reality anchor point placement method as described in any one of claims 1 to 5.
