Augmented reality anchor point placement method, device, electronic device and readable storage medium
By using NeRF model and target image pose data placement anchor points in AR point cloud, the problem of poor flexibility in placement of AR point cloud is solved, and better display effect is achieved.
Patent Information
- Application Number
- CN202410221287.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-02-28
AI Technical Summary
In the prior art, the flexibility of placing anchor points in augmented reality (AR) point clouds is poor, and the anchor points cannot be placed remotely, resulting in poor display effect of AR point clouds.
By obtaining the neural radiation field NeRF model and initial AR point cloud of the scene to be created, the characteristic points of the image are calculated and the target image is determined. An anchor point is placed in the AR point cloud based on the pose data of the target image, and the target AR point cloud is generated, and the target AR point cloud is realized. Build an AR point cloud including anchor points without actually placing anchor points.
It improves the flexibility of AR point cloud placement anchor points, achieves better display effects, and enhances the flexibility of AR point cloud placement.
Smart Images

Figure CN118071965B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning technology, and in particular to an augmented reality anchor point placement method, device, electronic device, and readable storage medium. Background Art
[0002] Augmented Reality (AR) is one of the key technologies for virtual experience. AR mapping enables users to browse different three-dimensional (3D) scenes through terminals. In related technologies, after establishing an AR point cloud, it is necessary to pre-place anchor points in the AR mapping scene to display the virtual objects corresponding to the anchor points in the AR point cloud. However, in related technologies, the anchor points must be actually placed in the scene to be mapped, and the anchor points cannot be placed remotely, resulting in the AR point cloud being unable to display the virtual objects corresponding to the anchor points, resulting in poor flexibility in placing anchor points in the AR point cloud.
[0003] It can be seen that the related technology has the problem of poor flexibility in placing anchor points of AR point clouds. Summary of the invention
[0004] The embodiments of the present application provide an augmented reality anchor point placement method, device, electronic device and readable storage medium to solve the problem of poor flexibility in placing anchor points of AR point clouds in the related art.
[0005] To solve the above problems, this application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides an augmented reality anchor point placement method, comprising:
[0007] Obtain the Neural Radiation Field (NeRF) model and initial augmented reality (AR) point cloud of the scene to be mapped;
[0008] Based on the NeRF model, M images and first position data of the M images are obtained, where the M images are images rendered by the NeRF model, and M is an integer greater than 1;
[0009] Calculating the number of feature points corresponding to each of the M images, and determining a target image among the M images, wherein the number of feature points corresponding to the target image meets a preset condition;
[0010] Anchor points are placed in the initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud.
[0011] In a second aspect, an embodiment of the present application further provides an augmented reality anchor point placement device, comprising:
[0012] The first acquisition module is used to obtain the NeRF model and initial augmented reality AR point cloud of the scene to be mapped;
[0013] A second acquisition module is used to acquire M images and first position data of the M images based on the NeRF model, wherein the M images are images rendered by the NeRF model, and M is an integer greater than 1;
[0014] A first calculation module, used to calculate the number of feature points corresponding to each of the M images, and determine a target image among the M images, wherein the number of feature points corresponding to the target image meets a preset condition;
[0015] A generation module is used to place anchor points in the initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud.
[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps in the augmented reality anchor point placement method described in the first aspect above are implemented.
[0017] In a fourth aspect, an embodiment of the present application further provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the augmented reality anchor point placement method as described in the first aspect above.
[0018] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the augmented reality anchor point placement method as described in the first aspect above.
[0019] In the embodiment of the present application, the NeRF model of the neural radiation field and the initial augmented reality AR point cloud of the scene to be mapped are obtained; based on the NeRF model, M images and the first position data of the M images are obtained, the M images are images rendered by the NeRF model, and M is an integer greater than 1; the number of feature points corresponding to each of the M images is calculated, and the target image in the M images is determined, and the number of feature points corresponding to the target image meets the preset conditions; the target AR point cloud is generated based on the target image, the first position data of the target image, and the initial AR point cloud. In this way, M images are obtained through the NeRF model, and then the target image that meets the preset conditions is determined from the M images, and the target image is used as an anchor point. A better display effect can be achieved through the target image to generate a target AR point cloud, and the target AR point cloud including the anchor point is constructed without actually placing the anchor point, thereby improving the flexibility of placing the anchor point of the AR point cloud. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 is a flow chart of an augmented reality anchor point placement method provided in an embodiment of the present application;
[0022] Figure 2 This is a schematic diagram of the interaction between the cloud and the client provided in an embodiment of the present application;
[0023] Figure 3 is a schematic diagram of posture data of N images provided in an embodiment of the present application;
[0024] Figure 4 is a structural diagram of an augmented reality anchor point placement device provided in an embodiment of the present application;
[0025] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0027] See also Figure 1 , Figure 1 is a flow chart of an augmented reality anchor point placement method provided by an embodiment of the present application, such as Figure 1 As shown, the following steps are included:
[0028] Step 101: Obtain a neural radiation field NeRF model and an initial augmented reality AR point cloud of the scene to be mapped;
[0029] Step 102: acquiring M images and first position data of the M images based on the NeRF model, wherein the M images are images rendered by the NeRF model, and M is an integer greater than 1;
[0030] Step 103, calculating the number of feature points corresponding to each of the M images, and determining a target image among the M images, wherein the number of feature points corresponding to the target image meets a preset condition;
[0031] Step 104: placing anchor points in the initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud.
[0032] The above NeRF model and the initial AR point cloud are the model / point cloud of the same scene to be mapped, wherein the NeRF model is a point cloud model, specifically a three-dimensional point cloud model, which is used to render images of the scene to be mapped under different pose data conditions; the initial AR point cloud is a three-dimensional point cloud, and the initial AR point cloud is a point cloud that has not been constructed through anchor points, and the initial AR point cloud does not include virtual objects corresponding to the anchor points. In the embodiment of the present application, the anchor point (i.e., the target image) is generated by the NeRF model, and then the target AR point cloud is generated in combination with the initial AR point cloud, so that the target AR point cloud includes the target image.
[0033] The above M images are images rendered by the NeRF model. The first position data corresponding to each image can be the same or different. By determining the target image among the M images, the number of feature points corresponding to the target image meets the preset conditions, and the target image is used as the anchor point, and the target AR point cloud is generated through the target image and the first position number of the target image and the initial AR point, so that the target AR point cloud includes the target image.
[0034] Among them, M images can be obtained as follows:
[0035] Obtain multiple first pose data, and render each first pose data based on the NeRF model to obtain M images. The multiple first pose data can be obtained through the client. For example, Figure 2 As shown, after the NeRF model is trained in the cloud, the cloud receives the first position data sent by the terminal through the server. Further, after receiving multiple first position data in the cloud, the image corresponding to each first position data is rendered, and then the image is sent to the client to obtain M images.
[0036] It should be noted that all M images can be used as anchor points to generate the target AR point cloud, but different images in the M images have different effects as anchor points. When the image meets the preset conditions, the effect of using the image as an anchor point is better, so that the display effect of the anchor point in the target AR point cloud established based on the first pose data of the image is better. In this application, the images are screened by the number of feature points of the image to obtain a target AR point cloud with better anchor point display effect.
[0037] The number of feature points corresponding to the target image meeting the preset conditions includes at least one of the following:
[0038] The number of feature points corresponding to the target image is greater than a set number threshold;
[0039] The proportion of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-viewed image to the number of feature points corresponding to the target image is greater than a first set ratio threshold, the first co-viewed image is a common area between the M images and the target image, and the first position data of the first co-viewed image matches the position data of the target image.
[0040] The above number of feature points is the number of feature points of the scene to be mapped in the image. It should be noted that when the number of feature points is less than or equal to the set number threshold, the accuracy of pure visual positioning will be reduced, and the image will be displayed as an anchor point less effectively; while when the number of feature points is greater than the set number threshold, the accuracy of pure visual positioning will be improved, and the image will be displayed as an anchor point more effectively in this case.
[0041] The first co-viewing image is an image in the M images that has a common area with the target image, and the pose data of the first co-viewing image and the target image match. The first co-viewing image can be one image or multiple images. In the case where the first co-viewing image is one image, the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-viewing image accounts for a ratio of feature points corresponding to the target image that is greater than a first set ratio threshold; in the case where the first co-viewing image is multiple images, the number of feature points corresponding to the target image that matches the number of feature points corresponding to each first co-viewing image accounts for a ratio of feature points corresponding to the target image that is greater than a first set ratio threshold. It should be noted that the greater the ratio of the number of matched feature points to the number of feature points corresponding to the target image, the more feature points each image in the multiple images (including the target image and the first co-viewing image) corresponding to the first pose data of the target image has about the scene to be mapped, and in this case, the image is better displayed as an anchor point.
[0042] Among them, the pose data of the first co-viewing image and the target image match, which can be that the pose data of the first co-viewing image and the pose data of the target image are the same, or the difference between the pose data of the first co-viewing image and the pose data of the target image is within a set range.
[0043] The above-mentioned target image can be one image or multiple images that meet preset conditions. When the target image is multiple images, the target AR point cloud is generated based on each target image, the first position data corresponding to each image and the initial AR point cloud.
[0044] In the embodiment of the present application, the NeRF model of the neural radiation field and the initial augmented reality AR point cloud of the scene to be mapped are obtained; based on the NeRF model, M images and the first position data of the M images are obtained, the M images are images rendered by the NeRF model, and M is an integer greater than 1; the number of feature points corresponding to each of the M images is calculated, and the target image in the M images is determined, and the number of feature points corresponding to the target image meets the preset conditions; the target AR point cloud is generated based on the target image, the first position data of the target image, and the initial AR point cloud. In this way, M images are obtained through the NeRF model, and then the target image that meets the preset conditions is determined from the M images, and the target image is used as an anchor point. A better display effect can be achieved through the target image to generate a target AR point cloud, and the target AR point cloud including the anchor point is constructed without actually placing the anchor point, thereby improving the flexibility of placing the anchor point of the AR point cloud.
[0045] In one embodiment, after calculating the number of feature points corresponding to each of the M images, the method further includes:
[0046] Calculating a peak signal-to-noise ratio of each of the M images and an average peak signal-to-noise ratio of the M images;
[0047] The preset condition includes at least one of the following:
[0048] The peak signal-to-noise ratio of the target image is greater than a set peak signal-to-noise ratio threshold;
[0049] A weighted sum of the first ratio and the second ratio is greater than a second set ratio threshold, the first ratio being the ratio of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-visual image to the number of feature points corresponding to the target image, and the second ratio being the ratio of the peak signal-to-noise ratio of the target image to the mean of the peak signal-to-noise ratio.
[0050] It should be noted that the image quality of the target image used as an anchor point also affects the anchor point display effect. When the target image quality is higher, the target AR point cloud obtained by using the target image as an anchor point will have a better effect. Therefore, in the embodiment of the present application, target images with higher image quality are screened through preset conditions to further improve the anchor point display effect of the target AR point cloud.
[0051] Furthermore, the embodiment of the present application also confirms the target image by combining the image quality and the feature points of the image, that is, by whether the weighted sum of the first ratio and the second ratio is greater than the second set ratio threshold, to determine the target image from M images. When the weighted sum of the first ratio and the second ratio is greater than the second set ratio threshold, it is considered that the first position data of the target image has a better display effect, and the target image is used as an anchor point to generate a target AR point cloud.
[0052] For example, the second set threshold is 0.93, the first ratio is k1, the second ratio is k2, the weight coefficient corresponding to the first ratio is 0.4, the weight coefficient corresponding to the second ratio is 0.6, and the target image is determined from M images, and the target image is an image with 0.4k1+0.6k2>0.93.
[0053] Optionally, the number of corresponding feature points of the target image is greater than a set number threshold, the peak signal-to-noise ratio of the target image is greater than a set peak signal-to-noise ratio threshold, and the weighted sum of the first ratio and the second ratio is greater than a second set ratio threshold.
[0054] In an embodiment of the present application, by calculating the peak signal-to-noise ratio of each image in M images and the average peak signal-to-noise ratio of the M images, the target image is determined from the M images, the peak signal-to-noise ratio of the target image is greater than the set peak signal-to-noise ratio threshold, and / or the weighted sum of the first ratio and the second ratio is greater than the second set ratio threshold, the first ratio is the ratio of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-visual image to the number of feature points corresponding to the target image, and the second ratio is the ratio of the peak signal-to-noise of the target image to the average peak signal-to-noise ratio. In this way, the image quality of the target image is higher, and the target image is used as an anchor point to further improve the anchor point display effect of the target AR point cloud.
[0055] In one embodiment, the step of obtaining a neural radiation field NeRF model and an initial augmented reality AR point cloud of a scene to be mapped includes:
[0056] Obtain an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1;
[0057] The NeRF model is constructed based on the image set and the camera intrinsic parameters, and the initial AR point cloud is constructed based on the image set and the camera intrinsic parameters.
[0058] The N images in the above image set are all images of the scene to be mapped. The N images can be obtained by extracting frames from a video of the scene to be mapped. For example, the N images are obtained by extracting frames from the video at intervals of 1 second. Figure 3 As shown in the figure, the N images are images from different perspectives. By analyzing each image and the intrinsic parameters of the camera that takes the image, the pose data of each image is calculated to construct the NeRF model and AR point cloud.
[0059] The process of building the NeRF model is as follows: calculate the pose data of each image in N images based on colmap and camera intrinsic parameters; form the translation and rotation matrix of each image based on the pose data, and generate the training file transforms.json based on the translation and rotation matrix. The training file is used as the training data of the NeRF model to train and generate the NeRF model of the scene to be built.
[0060] In addition, the process of constructing the initial AR point cloud is specifically as follows: extracting two-dimensional feature points from N images, matching the two-dimensional feature points between every two images, achieving the correspondence between the two-dimensional (2D) feature points between every two images, and then achieving the correspondence between the two-dimensional feature points in the image and the point cloud of the three-dimensional scene, so as to calculate the pose data of each image in combination with the camera intrinsic parameters, and calculate the coordinates of the three-dimensional point cloud to generate the initial AR point cloud.
[0061] In one embodiment, generating a target AR point cloud based on the target image, the first pose data of the target image and the initial AR point cloud includes:
[0062] Based on the conversion relationship between the NeRF model and the initial AR point cloud, converting the first pose data of the target image into the second pose data of the target image;
[0063] A target AR point cloud is generated based on the target image, the second pose data of the target image, and the initial AR point cloud.
[0064] The conversion relationship between the above NeRF model and the initial AR point cloud is the conversion relationship between the pose data of the image at the same position in the scene in the NeRF model and the initial AR point cloud. It should be noted that there is a difference between the process of building the NeRF model and the process of building the initial AR point cloud. The poses of the same image in different point clouds are different. It is necessary to convert the first pose data of the target image in the NeRF model into the second pose data in the initial AR point cloud, and then generate the target AR point cloud based on the target image, the second pose data of the target image and the initial AR point cloud, so as to improve the anchor point display effect of the target AR point cloud.
[0065] In an embodiment of the present application, the first pose data of the target image is converted into the second pose data of the target image through the conversion relationship based on the NeRF model and the initial AR point cloud; the target AR point cloud is generated based on the target image, the second pose data of the target image and the initial AR point cloud. In this way, the first pose data of the target image in the NeRF model is converted into the second pose data in the initial AR point cloud. The second pose data of the target image has higher accuracy than the first pose data, thereby further improving the anchor point display effect of the target AR point cloud.
[0066] In one embodiment, before converting the first pose data of the target image into the second pose data of the target image based on the conversion relationship between the NeRF model and the initial AR point cloud, the method includes:
[0067] Calculating first Euclidean distances of a plurality of second co-viewed images based on the NeRF model, and calculating second Euclidean distances of the plurality of second co-viewed images based on the initial AR point cloud, wherein the plurality of second co-viewed images are images having a common area in the N images;
[0068] The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance.
[0069] It should be noted that after the NeRF model and the initial AR point cloud are constructed by the image set and the camera intrinsic parameters, the first pose data of the target image rendered by the NeRF model cannot be directly used as the anchor point because the pose data of the same image in the NeRF model and in the initial AR point cloud are different. It is necessary to convert the first pose data of the target image into the second pose data based on the conversion relationship. The second pose data is the pose data of the target image in the initial AR point cloud. The target image, the second pose data of the target image and the initial AR point cloud are used to generate the target AR point cloud to improve the anchor point display effect of the target AR point cloud.
[0070] The above-mentioned multiple second co-view images are images screened from M target images based on the common view algorithm. There are common areas in the multiple second co-view images. The first Euclidean distances of the multiple second co-view images are calculated based on the NeRF model, and the first Euclidean distance is used to characterize the position data of the target image in the NeRF model; the second Euclidean distances of the multiple second co-view images are calculated based on the initial AR point cloud, and the second Euclidean distance is used to characterize the position data of the target image in the AR point cloud. The conversion relationship between the NeRF model and the initial AR point cloud is determined by the first Euclidean distance and the second Euclidean distance.
[0071] Exemplarily, after the first Euclidean distance and the second Euclidean distance are calculated, the first pose data of the target image can be converted into the second pose data by the ratio of the first Euclidean distance and the second Euclidean distance, for example, the quotient of the first pose data of the target image and the ratio is set as the second pose data of the target image. Optionally, the conversion relationship can also include the ratio of the second Euclidean distance to the first Euclidean distance, and the first pose data of the target image is converted into the second pose data by the ratio of the second Euclidean distance to the first Euclidean distance.
[0072] In an embodiment of the present application, the first Euclidean distance of multiple second co-viewed images is calculated based on the NeRF model, and the second Euclidean distance of multiple second co-viewed images is calculated based on the initial AR point cloud, so that the conversion relationship between the NeRF model and the initial AR point cloud is determined by the first Euclidean distance and the second Euclidean distance, thereby realizing the conversion of the first pose data of the target image into the second pose data based on the conversion relationship between the NeRF model and the initial AR point cloud.
[0073] In one embodiment, calculating the first Euclidean distance of the plurality of second co-viewed images based on the NeRF model includes:
[0074] Calculate the Euclidean distance between every two images in the plurality of second co-viewed images based on the NeRF model to obtain a plurality of first sub-distances;
[0075] Setting the sum of the multiple first sub-distances to be the first Euclidean distance;
[0076] The computing the second Euclidean distance of the plurality of second co-visible images based on the initial AR point cloud comprises:
[0077] Computing the Euclidean distance between every two images in the plurality of second co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances;
[0078] The sum of the multiple second sub-distances is set as the second Euclidean distance.
[0079] There is a Euclidean distance between every two second co-viewed images in the above-mentioned multiple second co-viewed images, and the multiple second co-viewed images may include more than two images, that is, there are multiple Euclidean distances. It is necessary to combine the Euclidean distance between every two second co-viewed images in all the second co-viewed images to obtain the first Euclidean distance or the second Euclidean distance.
[0080] Specifically, the Euclidean distance between each two images in the multiple second co-viewed images is calculated based on the NeRF model to obtain multiple first sub-distances, and then the sum of the multiple first sub-distances is set as the first Euclidean distance; based on the initial AR point cloud, the Euclidean distance between each two images in the multiple second co-viewed images is calculated to obtain multiple second sub-distances, and then the sum of the multiple second sub-distances is set as the second Euclidean distance, so as to realize the conversion relationship between the NeRF model and the initial AR point cloud through the first Euclidean distance and the second Euclidean distance.
[0081] See also Figure 4 , Figure 4 is a structural diagram of an augmented reality anchor point placement device provided in an embodiment of the present application, such as Figure 4 As shown, the augmented reality anchor point placement device 400 includes:
[0082] The first acquisition module 401 is used to acquire a neural radiation field NeRF model and an initial augmented reality AR point cloud of the scene to be mapped;
[0083] A second acquisition module 402 is used to acquire M images and first position data of the M images based on the NeRF model, wherein the M images are images rendered by the NeRF model, and M is an integer greater than 1;
[0084] A first calculation module 403 is used to calculate the number of feature points corresponding to each of the M images, and determine a target image among the M images, wherein the number of feature points corresponding to the target image meets a preset condition;
[0085] The generating module 404 is configured to generate a target AR point cloud based on the target image, the first pose data of the target image and the initial AR point cloud.
[0086] In one embodiment, the number of feature points corresponding to the target image meets the preset condition including at least one of the following:
[0087] The number of feature points corresponding to the target image is greater than a set number threshold;
[0088] The proportion of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-viewed image to the number of feature points corresponding to the target image is greater than a first set ratio threshold, the first co-viewed image is a common area between the M images and the target image, and the first position data of the first co-viewed image matches the position data of the target image.
[0089] In one embodiment, after the first calculation module 403, the augmented reality anchor point placement device 400 further includes:
[0090] A second calculation module, used for calculating the peak signal-to-noise ratio of each image in the M images and the average peak signal-to-noise ratio of the M images;
[0091] The preset condition includes at least one of the following:
[0092] The peak signal-to-noise ratio of the target image is greater than a set peak signal-to-noise ratio threshold;
[0093] A weighted sum of the first ratio and the second ratio is greater than a second set ratio threshold, the first ratio being the ratio of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-visual image to the number of feature points corresponding to the target image, and the second ratio being the ratio of the peak signal-to-noise ratio of the target image to the mean of the peak signal-to-noise ratio.
[0094] In one embodiment, the first acquisition module 401 includes:
[0095] An acquisition unit, used to acquire an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1;
[0096] A construction unit is used to construct the NeRF model based on the image set and the camera intrinsic parameters, and to construct the initial AR point cloud based on the image set and the camera intrinsic parameters.
[0097] In one embodiment, the generating module 404 includes:
[0098] A conversion submodule, configured to convert the first pose data of the target image into the second pose data of the target image based on the conversion relationship between the NeRF model and the initial AR point cloud;
[0099] A generating submodule is used to generate a target AR point cloud based on the target image, the second pose data of the target image and the initial AR point cloud.
[0100] In one embodiment, before the conversion submodule, the augmented reality anchor point placement device 400 includes:
[0101] a calculation submodule, configured to calculate a first Euclidean distance of a plurality of second co-viewed images based on the NeRF model, and calculate a second Euclidean distance of the plurality of second co-viewed images based on the initial AR point cloud, wherein the plurality of second co-viewed images are images having a common area in the N images;
[0102] The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance.
[0103] In one embodiment, the calculation submodule includes:
[0104] A first calculation unit, configured to calculate the Euclidean distance between every two images in the plurality of second co-viewed images based on the NeRF model to obtain a plurality of first sub-distances;
[0105] a second calculation unit, configured to set the sum of the plurality of first sub-distances as the first Euclidean distance;
[0106] a third calculating unit, configured to calculate the Euclidean distance between every two images in the plurality of second co-viewing images based on the initial AR point cloud to obtain a plurality of second sub-distances;
[0107] The fourth calculation unit is used to set the sum of the multiple second sub-distances as the second Euclidean distance.
[0108] The augmented reality anchor point placement device provided in the embodiment of the present application is capable of implementing each process of each embodiment of the above-mentioned augmented reality anchor point placement method. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.
[0109] It should be noted that the augmented reality anchor point placement device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in an electronic device.
[0110] The present application also provides an electronic device, see Figure 5 , Figure 5 is a schematic diagram of the structure of an electronic device provided by the present application, the electronic device includes a memory 501, a processor 502 and a program or instruction stored in the memory 501 and running thereon, and when the program or instruction is executed by the processor 502, it can be realized Figure 1 Any steps in the corresponding method embodiments and achieving the same beneficial effects will not be repeated here.
[0111] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a graphics processing unit (GPU).
[0112] Those skilled in the art will appreciate that all or part of the steps of implementing the above-mentioned embodiment method can be completed by hardware associated with program instructions, and the program can be stored in a readable medium.
[0113] The present application also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned Figure 1 Any steps in the corresponding method embodiments can achieve the same technical effect, so they will not be repeated here to avoid repetition. The storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0114] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 Any steps in the corresponding method embodiments can achieve the same technical effect and will not be repeated here to avoid repetition.
[0115] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing comprising independent A, independent B, independent C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations that A, B and C all exist.
[0116] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a second terminal device, etc.) to execute the methods of each embodiment of the present application.
[0118] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for placing an augmented reality anchor point, characterized in that: include: Obtain the Neural Radiation Field (NeRF) model and initial augmented reality (AR) point cloud of the scene to be mapped; Based on the NeRF model, M images and first position data of the M images are obtained, where the M images are images rendered by the NeRF model, and M is an integer greater than 1; Calculating the number of feature points corresponding to each of the M images, and determining a target image among the M images, wherein the number of feature points corresponding to the target image meets a preset condition; Placing anchor points in an initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud; The step of placing an anchor point in the initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud includes: The target image is used as the anchor point, and a target AR point cloud is generated through the target image, the first pose number of the target image, and the initial AR point.
2. The method according to claim 1, characterized in that The number of feature points corresponding to the target image meets the preset condition including at least one of the following: The number of feature points corresponding to the target image is greater than a set number threshold; The proportion of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-viewed image to the number of feature points corresponding to the target image is greater than a first set ratio threshold, the first co-viewed image is a common area between the M images and the target image, and the first position data of the first co-viewed image matches the position data of the target image.
3. The method according to claim 1, characterized in that After calculating the number of feature points corresponding to each of the M images, the method further includes: Calculating a peak signal-to-noise ratio of each of the M images and an average peak signal-to-noise ratio of the M images; The preset condition includes at least one of the following: The peak signal-to-noise ratio of the target image is greater than a set peak signal-to-noise ratio threshold; A weighted sum of the first ratio and the second ratio is greater than a second set ratio threshold, the first ratio being the ratio of the number of feature points corresponding to the target image that matches the number of feature points corresponding to the first co-visual image to the number of feature points corresponding to the target image, and the second ratio being the ratio of the peak signal-to-noise ratio of the target image to the mean of the peak signal-to-noise ratio.
4. The method according to any one of claims 1 to 3, characterized in that The step of obtaining the NeRF model of the scene to be mapped and the initial augmented reality point cloud comprises: Obtain an image set and camera internal parameters, wherein the image set includes N images of the scene to be mapped taken at different viewing angles, where N is an integer greater than 1; The NeRF model is constructed based on the image set and the camera intrinsic parameters, and the initial AR point cloud is constructed based on the image set and the camera intrinsic parameters.
5. The method according to claim 4, characterized in that The generating a target AR point cloud based on the target image, the first pose data of the target image and the initial AR point cloud comprises: Based on the conversion relationship between the NeRF model and the initial AR point cloud, converting the first pose data of the target image into the second pose data of the target image; A target AR point cloud is generated based on the target image, the second pose data of the target image, and the initial AR point cloud.
6. The method according to claim 5, characterized in that Before converting the first pose data of the target image into the second pose data of the target image based on the conversion relationship between the NeRF model and the initial AR point cloud, the method includes: Calculating first Euclidean distances of a plurality of second co-viewed images based on the NeRF model, and calculating second Euclidean distances of the plurality of second co-viewed images based on the initial AR point cloud, wherein the plurality of second co-viewed images are images having a common area in the N images; The conversion relationship includes a ratio of the first Euclidean distance to the second Euclidean distance.
7. The method according to claim 6, characterized in that The calculating the first Euclidean distance of the plurality of second co-viewed images based on the NeRF model comprises: Calculate the Euclidean distance between every two images in the plurality of second co-viewed images based on the NeRF model to obtain a plurality of first sub-distances; Setting the sum of the multiple first sub-distances to be the first Euclidean distance; The computing the second Euclidean distance of the plurality of second co-visible images based on the initial AR point cloud comprises: Computing the Euclidean distance between every two images in the plurality of second co-viewed images based on the initial AR point cloud to obtain a plurality of second sub-distances; The sum of the multiple second sub-distances is set as the second Euclidean distance.
8. An augmented reality anchor point placement device, characterized in that: include: The first acquisition module is used to obtain the NeRF model and initial augmented reality AR point cloud of the scene to be mapped; A second acquisition module is used to acquire M images and first position data of the M images based on the NeRF model, wherein the M images are images rendered by the NeRF model, and M is an integer greater than 1; A first calculation module, used to calculate the number of feature points corresponding to each of the M images, and determine a target image among the M images, wherein the number of feature points corresponding to the target image meets a preset condition; A generating module, configured to place anchor points in an initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud; The step of placing an anchor point in the initial AR point cloud based on the target image and the first pose data of the target image to generate a target AR point cloud includes: The target image is used as the anchor point, and a target AR point cloud is generated through the target image, the first pose number of the target image, and the initial AR point.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the method for placing an augmented reality anchor point as claimed in any one of claims 1 to 7.
10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the augmented reality anchor point placement method according to any one of claims 1 to 7 are implemented.
11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the augmented reality anchor point placement method as described in any one of claims 1 to 7.
