Image set processing method and device

By evaluating and optimizing the integrity of the target image set, the problem that the quality of the light field reconstruction model depends on human-computer interaction in the prior art is solved, and scientific evaluation and improvement of the integrity and quality of the light field reconstruction model is achieved.

CN119006706APending Publication Date: 2024-11-22LENOVO SHANGHAI INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411040635.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the existing three-dimensional reconstruction technology, the quality of the light field reconstruction model depends on human-computer interaction, lack of scientific basis, resulting in incomplete image information and ineffective model quality.

Method used

By obtaining the target structure information of the target space, the integrity index value of the target image set is determined based on multiple target poses, the integrity of the image set is evaluated, and the new pose is determined based on the evaluation results, and new images are added to improve the image set. Completeness.

Benefits of technology

The scientific evaluation and improvement of the quality of the light field reconstruction model is achieved, and the problems of inefficient reconstruction efficiency and poor quality caused by incomplete image information are avoided, ensuring the high integrity and high quality of the light field reconstruction model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006706A_ABST
    Figure CN119006706A_ABST
Patent Text Reader

Abstract

The invention discloses an image set processing method and device, and relates to the technical field of three-dimensional reconstruction. The method comprises the following steps: obtaining target structure information of a target space, wherein the target structure information is obtained based on a surface reconstruction result corresponding to the target space; determining an integrity index value of a target image set based on the target structure information and the plurality of target poses; wherein the plurality of target poses are poses of image acquisition devices corresponding to a plurality of images in the target image set, and the target image set is used for performing light field reconstruction on the target space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to an image set processing method and device. Background Art

[0002] Three-dimensional reconstruction is currently widely used in aerospace, automobile, medical, scientific research, games, film and television, and its commercial value is becoming increasingly prominent with the emergence of technologies such as "digital twins" and "metaverse".

[0003] In computer vision, light field reconstruction technology has emerged to realistically reconstruct three-dimensional objects and improve the fidelity of three-dimensional object rendering. This technology constructs a camera array to collect multi-view images and lighting information for three-dimensional reconstruction of the volume of objects contained in the target space. However, currently, image information from different perspectives is usually collected based on the historical experience of staff for light field reconstruction, but due to the lack of scientific basis, the quality of the model reconstructed based on the image information cannot be effectively guaranteed due to incomplete collected image information. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application provide an image set processing method and device.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image set processing method, including:

[0007] Obtaining target structure information of the target space, wherein the target structure information is obtained based on a surface reconstruction result corresponding to the target space;

[0008] Determining a completeness index value of a target image set based on the target structure information and a plurality of target poses;

[0009] Among them, the multiple target postures are the postures of the image acquisition devices corresponding to the multiple images in the target image set, and the target image set is used to reconstruct the light field of the target space.

[0010] In some embodiments, determining the integrity index value of the target image set based on the target structure information and the multiple target poses includes:

[0011] Based on the target structure information, determining a target point in the target space as a reference point;

[0012] Based on the relationship between the reference point and the plurality of target poses, a completeness index value of the target image set is determined.

[0013] In some embodiments, determining the completeness index value of the target image set based on the relationship between the reference point and the plurality of target poses includes:

[0014] Determine first information corresponding to the reference point, where the first information is used to represent a proportion of target postures in the multiple target postures that can observe the reference point;

[0015] Based on the first information, a completeness index value of the target image set is determined.

[0016] In some embodiments, determining the completeness index value of the target image set based on the relationship between the reference point and the plurality of target poses includes:

[0017] Determining second information corresponding to the reference point, the second information being used to characterize a distribution uniformity of angles between a line connecting the reference point and the positions of the multiple target postures and an observation direction of the multiple target postures;

[0018] Based on the second information, a completeness index value of the target image set is determined.

[0019] In some embodiments, determining the integrity index value of the target image set based on the target structure information and the multiple target poses includes:

[0020] Based on the target structure information, determining distribution information of the image acquisition points corresponding to the multiple target postures in the target space;

[0021] Based on the distribution information, a completeness index value of the target image set is determined.

[0022] In some embodiments, the method further comprises:

[0023] Based on the integrity index value of the target image set and the target structure information, a newly added posture is determined, and the newly added posture is used to collect a new image of the target space and add it to the target image set.

[0024] In some embodiments, the method further comprises:

[0025] The newly added posture is output to a control unit of the image acquisition device, and the control unit is used to control the image acquisition device to acquire the new image of the target space based on the newly added posture.

[0026] In some embodiments, the method further comprises:

[0027] Based on the newly added posture, an image acquisition guiding animation is generated, and the image acquisition guiding animation is used to guide the user to acquire the new image of the target space through the image acquisition device according to the newly added posture.

[0028] In some embodiments, the method further comprises:

[0029] The integrity information corresponding to the integrity index value is displayed on the display interface of the image acquisition device.

[0030] In a second aspect, an embodiment of the present application provides an image set processing device, including:

[0031] An acquisition module, used for acquiring target structure information of a target space, wherein the target structure information is obtained based on a surface reconstruction result corresponding to the target space;

[0032] A determination module, used to determine the integrity index value of the target image set based on the target structure information and multiple target postures;

[0033] Among them, the multiple target postures are the postures of the image acquisition devices corresponding to the multiple images in the target image set, and the target image set is used to reconstruct the light field of the target space. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of image acquisition for surface reconstruction provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of image acquisition for NERF reconstruction provided in an embodiment of the present application;

[0037] Figure 3 One of the flowcharts of an image set processing method provided in an embodiment of the present application;

[0038] Figure 4 A second flowchart of an image set processing method provided in an embodiment of the present application;

[0039] Figure 5 A third flowchart of an image set processing method provided in an embodiment of the present application;

[0040] Figure 6A fourth flowchart of an image set processing method provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of observing reference points from the perspectives of various target positions provided in an embodiment of the present application;

[0042] Figure 8 A fifth flowchart of an image set processing method provided in an embodiment of the present application;

[0043] Fig. 9 A sixth flowchart of an image set processing method provided in an embodiment of the present application;

[0044] Fig.10 One of the schematic diagrams of a light field reconstruction process provided in an embodiment of the present application;

[0045] Fig.11 A second schematic diagram of a light field reconstruction process provided in an embodiment of the present application;

[0046] Fig.12 A third schematic diagram of a light field reconstruction process provided in an embodiment of the present application;

[0047] Fig.13 A schematic diagram of the structure of an image set processing device provided in an embodiment of the present application;

[0048] Fig.14 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0050] It should be noted that in the description of the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0051] In order to facilitate a clearer understanding of the embodiments of the present application, some related technologies are first introduced as follows.

[0052] Traditional 3D reconstruction technology is surface reconstruction. Surface reconstruction is a 3D reconstruction technology that uses image data or point cloud data to accurately restore the 3D surface shape of an object. The reconstruction result is usually a point cloud or a triangular mesh. The completeness of the reconstructed model depends on the judgment of the completeness of the mesh in the human-computer interaction experience. Figure 1 A schematic diagram of image acquisition for surface reconstruction provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, images of objects included in the target space are captured by cameras with different viewing angles (eg, Cam_1, Cam_2, and Cam_3), and the captured images can be used for surface reconstruction.

[0053] Neural Radiance Fields (NERF) reconstruction is an emerging technology in light field reconstruction. It uses a neural network to learn image data collected by cameras from different perspectives, so that the neural network stores the 3D scene information corresponding to the images collected by the camera in the weights of the neural network. After learning is completed, the image information corresponding to the 3D scene observed from each perspective will be recorded, and finally a 3D model with more restored details and authenticity will be generated. Figure 2 A schematic diagram of image acquisition for NERF reconstruction provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, it also uses cameras with different viewing angles (such as Cam_1, Cam_2 and Cam_3) to collect images of objects included in the target space, and the collected images can be used for NERF reconstruction.

[0054] It should be noted that Figure 1 In the method, ordinary cameras with different viewing angles are used to collect images of objects included in the target space to obtain the collected two-dimensional images; Figure 2 In this method, light field cameras with light field information collection function at different viewing angles are used to collect light field images of objects included in the target space to obtain two-dimensional images carrying light field information.

[0055] In the following, in conjunction with the drawings in the embodiments of the present application, an image set processing method and device provided in the embodiments of the present application are exemplarily introduced.

[0056] Figure 3 One of the flow charts of an image set processing method provided in an embodiment of the present application is as follows: Figure 3 As shown, the method includes:

[0057] S301 : Obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space.

[0058] It should be noted that the target space can be any physical space, including indoor space and outdoor space. The indoor space is, for example, an office, a conference room, or a study room, and the outdoor space is, for example, a playground, a stadium, or a leisure square.

[0059] It should be noted that the target structure information may include three-dimensional point cloud information of the target space, or three-dimensional grid information of the target space, or voxel information of the target space, etc., and the embodiments of the present application do not specifically limit this.

[0060] In some embodiments, the surface of the target space may be reconstructed based on any surface reconstruction algorithm to obtain a surface reconstruction result, and then the target structure information of the target space may be obtained based on the surface reconstruction result.

[0061] In some embodiments, the target structure information of the target space may be obtained from the target system or target device, wherein the target system or target device may be a system or device for performing surface reconstruction on the target space, the system or device stores a surface reconstruction result of performing surface reconstruction on the target space, and stores the target structure information of the target space obtained based on the surface reconstruction result.

[0062] S302: Determine a completeness index value of a target image set based on the target structure information and multiple target postures.

[0063] Among them, the multiple target postures are the postures of the image acquisition devices corresponding to the multiple images in the target image set, and the target image set is used to reconstruct the light field of the target space.

[0064] In an embodiment of the present application, a target image set for reconstructing the light field of the target space can be first determined, and then the postures of the image acquisition devices corresponding to the multiple images in the target image set can be determined, and the postures of the image acquisition devices corresponding to the multiple images can be used as multiple target postures, and then based on the target structure information of the target space and the multiple target postures, the completeness index value of the target image set can be determined.

[0065] It should be noted that the image acquisition device can be an intelligent device capable of image acquisition, such as a smart phone, a tablet computer (PAD) or AR glasses, or it can be a digital camera, a sports camera or a panoramic camera, etc. The embodiments of the present application do not specifically limit this.

[0066] It should be noted that, according to the completeness index value of the target image set, the completeness of the light field reconstruction model or the quality of the light field reconstruction model obtained by performing light field reconstruction based on the target image set can be evaluated.

[0067] It can be understood that the image set processing method provided in the embodiment of the present application evaluates the integrity of the target image set based on the target structure information of the target space and the posture of the image acquisition device corresponding to the multiple images in the target image set for light field reconstruction, and obtains the integrity index value of the target image set. According to the integrity index value, the integrity or reconstruction quality of the light field reconstruction model obtained by light field reconstruction based on the target image set can be estimated, so that the estimation result is supported by a scientific basis, avoiding relying solely on the experience of the staff and being unable to guarantee the quality of the light field reconstruction model.

[0068] In some embodiments, determining the integrity index value of the target image set based on the target structure information and the multiple target poses includes:

[0069] Based on the target structure information, determining a target point in the target space as a reference point;

[0070] Based on the relationship between the reference point and the plurality of target poses, a completeness index value of the target image set is determined.

[0071] In an embodiment of the present application, after obtaining the target structure information of the target space, the target point in the target space can be determined as a reference point based on the target structure information, and then the completeness index value of the target image set can be determined based on the relationship between the reference point and multiple target postures.

[0072] It should be noted that the target point may be any one or more representative points that can reflect the target spatial structure.

[0073] It should be noted that the relationship between the reference point and the multiple target postures may include a relationship based on whether the reference point can be observed by the multiple target postures.

[0074] For example, Figure 4 A second flow chart of an image set processing method provided in an embodiment of the present application is as follows: Figure 4 As shown, the method includes:

[0075] S401 : Obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space.

[0076] S402: Based on the target structure information, determine a target point in the target space as a reference point.

[0077] S403: Determine a completeness index value of a target image set based on a relationship between the reference point and a plurality of target postures.

[0078] It can be understood that the embodiment of the present application determines the target point in the target space as a reference point based on the target structure information of the target space, and then determines the completeness index value of the target image set based on the relationship between the reference point and multiple target postures. That is, the completeness index value of the target image set can be calculated based on the relationship between the target point in the target space and multiple target postures, which has the advantages of simple calculation, saving calculation time and calculation resources, etc.

[0079] In some embodiments, determining the completeness index value of the target image set based on the relationship between the reference point and the plurality of target poses includes:

[0080] Determine first information corresponding to the reference point, where the first information is used to represent a proportion of target postures in the multiple target postures that can observe the reference point;

[0081] Based on the first information, a completeness index value of the target image set is determined.

[0082] In an embodiment of the present application, after determining that a target point in the target space is a reference point, the first information corresponding to the reference point can be determined, and the first information can represent the proportion of target postures in which the reference point can be observed among multiple target postures, and then based on the first information corresponding to the reference point, the completeness index value of the target image set can be determined.

[0083] For example, for a certain reference point p, the first information corresponding to the reference point p can be determined based on the following formula (1):

[0084]

[0085] Where N represents the total number of target poses; K i Indicates the i-th target pose; Contain obs (K i ,p) indicates whether the reference point p can be represented by the i-th target pose K i Observe that if so, then Contain obs (K i ,p)=1, if not, then Contain obs (K i ,p)=0;S freq (p) represents the proportion of target poses in which the reference point p can be observed among multiple target poses.

[0086] It can be understood that for a reference point p, the more target poses it can be observed by, the smaller its corresponding Sfreq The higher the (p) value.

[0087] For example, Figure 5 A third flow chart of an image set processing method provided in an embodiment of the present application is as follows: Figure 5 As shown, the method includes:

[0088] S501 : Obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space.

[0089] S502: Based on the target structure information, determine a target point in the target space as a reference point.

[0090] S503: Determine first information corresponding to the reference point, where the first information is used to represent a proportion of target postures in which the reference point can be observed among multiple target postures.

[0091] S504: Determine a completeness index value of a target image set based on the first information.

[0092] It should be noted that, the higher the proportion of target poses in which reference points can be observed among multiple target poses, the higher the completeness index value of the target image set is determined.

[0093] It can be understood that the embodiment of the present application can effectively determine the completeness index value of the target image set based on the first information corresponding to the reference point in the target space, that is, based on the proportion of target postures in which the reference point can be observed among multiple target postures, and has the advantages of simple calculation, saving calculation time and computing resources, etc.

[0094] In some embodiments, determining the completeness index value of the target image set based on the relationship between the reference point and the plurality of target poses includes:

[0095] Determining second information corresponding to the reference point, the second information being used to characterize a distribution uniformity of angles between a line connecting the reference point and the positions of the multiple target postures and an observation direction of the multiple target postures;

[0096] Based on the second information, a completeness index value of the target image set is determined.

[0097] In an embodiment of the present application, after determining that a target point in the target space is a reference point, second information corresponding to the reference point can be determined, and the second information can characterize the distribution uniformity of the angles between the lines connecting the reference point and the positions of multiple target postures and the observation directions of the multiple target postures, and then based on the second information corresponding to the reference point, the completeness index value of the target image set can be determined.

[0098] For example, for a certain reference point p, the second information corresponding to the reference point p may be determined based on the following formula (2):

[0099]

[0100] Among them, f PDF represents the probability density function (PDF); u represents the uniform probability density function; θ represents the angle between the line connecting the reference point p and the position of the target posture and the observation direction of the target posture; represents the target posture observation interval; Ω represents the target space; S TV It represents the distribution uniformity of the angle between the line connecting the reference point p and the positions of multiple target postures and the observation directions of multiple target postures.

[0101] It can be understood that the more uniform the distribution of the angles between the lines connecting the reference point p and the positions of the multiple target postures and the observation directions of the multiple target postures, the smaller the corresponding S TV The higher the value.

[0102] For example, Figure 6 A fourth flow chart of an image set processing method provided in an embodiment of the present application is as follows: Figure 6 As shown, the method includes:

[0103] S601 : Obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space.

[0104] S602: Based on the target structure information, determine a target point in the target space as a reference point.

[0105] S603, determining first information and second information corresponding to the reference point, wherein the first information is used to characterize the proportion of target postures in which the reference point can be observed among multiple target postures, and the second information is used to characterize the distribution uniformity of the angle between the line connecting the reference point and the positions of the multiple target postures and the observation directions of the multiple target postures.

[0106] S604: Determine a completeness index value of a target image set based on the first information and the second information.

[0107] In some embodiments, the completeness index value of the target image set may be determined based on an average of a value corresponding to the first information and a value corresponding to the second information.

[0108] For example, Figure 7 A schematic diagram of observing reference points from the perspective of each target posture is provided in an embodiment of the present application, such as Figure 7As shown, the more target postures the reference point p is observed by, the more evenly the angles between the lines connecting the reference point p and the positions of the multiple target postures and the observation directions of the multiple target postures are distributed.

[0109] It can be understood that the embodiment of the present application can effectively and accurately determine the integrity index value of the target image set based on the second information corresponding to the reference point in the target space, that is, based on the distribution uniformity of the angles between the lines connecting the reference point and the positions of multiple target postures and the observation directions of the multiple target postures, and has the advantages of simple calculation, saving calculation time and computing resources, etc.

[0110] In some embodiments, determining the integrity index value of the target image set based on the target structure information and the multiple target poses includes:

[0111] Based on the target structure information, determining distribution information of the image acquisition points corresponding to the multiple target postures in the target space;

[0112] Based on the distribution information, a completeness index value of the target image set is determined.

[0113] In an embodiment of the present application, after obtaining the target structure information of the target space, the distribution information of the image acquisition points corresponding to multiple target postures in the target space can be determined based on the target structure information, and then based on the distribution information, the completeness index value of the target image set can be determined.

[0114] In some embodiments, based on the target structure information of the target space, the distribution information of the image acquisition points corresponding to the multiple target postures in the target space can be determined by using the distribution function in mathematical statistics.

[0115] It should be noted that the distribution information may include distribution density information and distribution uniformity information.

[0116] It can be understood that the more uniformly the image acquisition points corresponding to the multiple target postures are distributed in the target space and the higher the distribution density is, the higher the completeness index value of the target image set can be determined.

[0117] For example, Figure 8 A fifth flow chart of an image set processing method provided in an embodiment of the present application is as follows: Figure 8 As shown, the method includes:

[0118] S801. Obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space.

[0119] S802: Based on the target structure information, determine distribution information of image acquisition points corresponding to multiple target postures in the target space.

[0120] S803: Determine a completeness index value of the target image set based on the distribution information.

[0121] It can be understood that the embodiments of the present application determine the distribution information of image acquisition points corresponding to multiple target postures in the target space based on the target structure information of the target space, and then determine the completeness index value of the target image set based on the distribution information. That is, by calculating the distribution information of image acquisition points corresponding to multiple target postures in the target space, the completeness index value of the target image set can be determined, which has the advantages of simple calculation, saving calculation time and calculation resources, etc.

[0122] In some embodiments, the image set processing method further comprises:

[0123] Based on the integrity index value of the target image set and the target structure information, a newly added posture is determined, and the newly added posture is used to collect a new image of the target space and add it to the target image set.

[0124] In an embodiment of the present application, after obtaining the completeness index value of the target image set, a new posture can be determined based on the completeness index value of the target image set and the target structure information, and then a new image captured of the target space can be obtained based on the new posture and added to the target image set.

[0125] It can be understood that after adding the new image to the target image set, the completeness index value of the target image set increases, thereby improving the completeness and quality of the light field reconstruction model obtained by performing light field reconstruction based on the target image set.

[0126] For example, Fig. 9 A sixth flow chart of an image set processing method provided in an embodiment of the present application is as follows: Fig. 9 As shown, the method includes:

[0127] S901. Obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space.

[0128] S902: Determine a completeness index value of a target image set based on the target structure information and multiple target postures.

[0129] S903: Determine a newly added posture based on the integrity index value and the target structure information, where the newly added posture is used to collect a new image of the target space and add it to the target image set.

[0130] It should be noted that in the related art, the completeness of the light field reconstruction model purely depends on the intuitive judgment of the operator. Once the reconstruction result does not meet expectations, the entire reconstruction process needs to be repeated, resulting in low reconstruction efficiency and failure to guarantee reconstruction quality. The embodiment of the present application evaluates the completeness index value of the target image set, and then determines the posture that needs to be added based on the completeness index value of the target image set, and adds a new image to the target image set based on the newly added posture, so that the image in the target image set is more complete for the target space, thereby ensuring that the light field reconstruction model obtained by light field reconstruction based on the target image set has a higher degree of completeness or better reconstruction quality, and can avoid repeating the entire reconstruction process multiple times.

[0131] In some embodiments, the image set processing method further comprises:

[0132] The newly added posture is output to a control unit of the image acquisition device, and the control unit is used to control the image acquisition device to acquire the new image of the target space based on the newly added posture.

[0133] In an embodiment of the present application, after the newly added posture is obtained, the newly added posture can be output to the control unit of the image acquisition device, so that the control unit can control the image acquisition device to capture a new image of the target space based on the newly added posture.

[0134] It should be noted that the image acquisition device in the embodiment of the present application may be an image acquisition device mounted on automated equipment such as a drone or a robot.

[0135] It can be understood that, by outputting the newly added posture to the control unit of the image acquisition device, the embodiment of the present application can enable the control unit to realize automatic image acquisition of the target space based on the newly added posture.

[0136] In some embodiments, the image set processing method further comprises:

[0137] Based on the newly added posture, an image acquisition guiding animation is generated, and the image acquisition guiding animation is used to guide the user to acquire the new image of the target space through the image acquisition device according to the newly added posture.

[0138] In an embodiment of the present application, after obtaining the newly added posture, an image acquisition guidance animation can be generated based on the newly added posture, so that the user can capture a new image of the target space through the image acquisition device according to the guidance of the image acquisition guidance animation.

[0139] In some embodiments, after the image acquisition guidance animation is generated based on the newly added posture, the image acquisition guidance animation can be displayed on a display interface of the image acquisition device.

[0140] It can be understood that the embodiment of the present application generates an image acquisition guidance animation based on the newly added posture. The image acquisition guidance animation helps to guide the user to acquire a new image of the target space through the image acquisition device, which can effectively improve the user experience.

[0141] In some embodiments, the image set processing method further comprises:

[0142] The integrity information corresponding to the integrity index value is displayed on the display interface of the image acquisition device.

[0143] In an embodiment of the present application, after obtaining the integrity index value of the target image set based on the target structure information and multiple target postures, the integrity information corresponding to the integrity index value of the target image set can be displayed on the display interface of the image acquisition device.

[0144] It can be understood that the embodiment of the present application helps the user to judge whether the current target image set is suitable for light field reconstruction of the target space based on the displayed completeness information and in combination with his own experience by displaying the completeness information corresponding to the completeness index value of the target image set on the display interface of the image acquisition device; if the completeness index value of the current target image set does not reach the set value, a new posture can be determined to acquire new images of the target space based on the new posture and add them to the target image set, until it is determined that the completeness index value of the target image set reaches the set value, and then the target image set is used to reconstruct the light field of the target space, so as to avoid poor quality of the light field reconstruction model obtained by light field reconstruction due to incomplete images in the target image set.

[0145] It should be noted that, based on the image set processing method described above, a target image set with a high integrity index value can be obtained, and the target image set can be used to reconstruct the light field of the target space, ensuring that the obtained light field reconstruction model has high integrity and good quality. The light field reconstruction process provided in the embodiment of the present application is introduced below.

[0146] For example, Fig.10 One of the flowcharts of a light field reconstruction provided in an embodiment of the present application is as follows: Fig.10As shown, firstly, a smart phone, PAD or AR glasses is used to perform a Mesh scan on the surface reconstruction result of the target space to obtain the target structure information of the target space, and then the integrity of the image set used for light field reconstruction is evaluated based on the target structure information, so as to obtain a new camera pose based on the evaluation result, and perform light field data acquisition based on the new camera pose, and add the acquired new image to the image set, and finally the server obtains the image set from the smart phone, PAD or AR glasses, and performs light field reconstruction based on the images in the image set. Among them, the server can realize light field reconstruction based on including but not limited to NERF three-dimensional reconstruction technology or 3D Gaussian three-dimensional reconstruction technology.

[0147] It should be noted that when the data volume of the Mesh scanning result is small (equal to or less than a specific value), the image set completeness evaluation function (i.e., the image set processing method described above) can be implemented using smart devices such as smartphones, PADs, or AR glasses.

[0148] For example, Fig.11 A second schematic diagram of a light field reconstruction process provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, firstly, a smart phone or PAD or AR glasses is used to perform a Mesh scan on the surface reconstruction result of the target space to obtain the target structure information of the target space, and the target structure information is transmitted to the server. Then, based on the target structure information, the server evaluates the completeness of the image set used for light field reconstruction, and based on the evaluation result, transmits the obtained newly added camera pose to the smart phone or PAD or AR glasses, so that the smart phone or PAD or AR glasses can collect light field data based on the newly added camera pose, and add the collected new image to the image set. Finally, the server obtains the image set from the smart phone or PAD or AR glasses, and reconstructs the light field based on the images in the image set. Among them, the server can realize light field reconstruction based on, but not limited to, NERF three-dimensional reconstruction technology or 3D Gaussian three-dimensional reconstruction technology.

[0149] It should be noted that when the data volume of the Mesh scanning results is large (greater than a specific value), the image set completeness assessment task can be migrated to the server. Through the interaction between the smart device (such as a smart phone or PAD or AR glasses) and the server, the final image set for light field reconstruction is obtained, and finally the light field reconstruction is realized based on the images in the image set by the server.

[0150] It should be noted that the amount of data in the Mesh scan result can be used to determine the Fig.10 or Fig.11 The light field reconstruction process shown implements light field reconstruction, thereby achieving a balance between light field reconstruction efficiency and user experience.

[0151] For example, Fig.12 A third schematic diagram of a light field reconstruction process provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, first, a smart phone or PAD or AR glasses is used to perform a Mesh scan on the surface reconstruction result of the target space to obtain the target structure information of the target space, and the target structure information is transmitted to the server. Then, based on the target structure information, the server evaluates the completeness of the image set used for light field reconstruction, and based on the evaluation result, transmits the obtained newly added camera pose to the smart phone or PAD or AR glasses, so that the digital camera or sports camera or panoramic camera that has been optically aligned with the smart phone or PAD or AR glasses can collect light field data based on the newly added camera pose, and add the collected new image to the image set and cache it in the smart phone or PAD or AR glasses. Finally, the server obtains the image set from the smart phone or PAD or AR glasses, and reconstructs the light field based on the images in the image set. Among them, the server can realize light field reconstruction based on, but not limited to, NERF three-dimensional reconstruction technology or 3D Gaussian three-dimensional reconstruction technology.

[0152] It should be noted that, for some users, the image quality captured by smart devices (such as smartphones or PADs and AR glasses) may not meet their actual needs, so the embodiments of the present application use digital cameras, sports cameras or panoramic cameras to capture images. In addition, since digital cameras, sports cameras or panoramic cameras do not have the function of communicating with the server, the embodiments of the present application optically align the smartphone or PAD or AR glasses with the digital camera or sports camera or panoramic camera, and cache the images captured by the digital camera or sports camera or panoramic camera in the smartphone or PAD or AR glasses.

[0153] It should be noted that in order to be compatible with more practical application scenarios, for the implementation of the Mesh scanning function, the pre-established Mesh scanning data can be imported into a smartphone, PAD, AR glasses, or personal computer for caching; or a 3D scanner can be used to perform Mesh scanning on the surface reconstruction results of the target space, and then the Mesh scanning data can be imported into a smartphone, PAD, AR glasses, or personal computer for caching; or a digital camera, sports camera, or panoramic camera that has been optically aligned with a smartphone, PAD, or AR glasses can be used to perform Mesh scanning on the surface reconstruction results of the target space, and then the Mesh scanning data can be imported into a smartphone, PAD, AR glasses, or personal computer for caching.

[0154] The image set processing device provided in an embodiment of the present application is described below. The image set processing device described below and the image set processing method described above can be referenced to each other.

[0155] Fig.13 A schematic diagram of the structure of an image set processing device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, the device includes: an obtaining module 1310 and a determining module 1320; wherein:

[0156] An acquisition module 1310 is used to obtain target structure information of a target space, where the target structure information is obtained based on a surface reconstruction result corresponding to the target space;

[0157] A determination module 1320, configured to determine a completeness index value of a target image set based on the target structure information and a plurality of target poses;

[0158] Among them, the multiple target postures are the postures of the image acquisition devices corresponding to the multiple images in the target image set, and the target image set is used to reconstruct the light field of the target space.

[0159] The image set processing device provided in the embodiment of the present application evaluates the integrity of the target image set based on the target structure information of the target space and the postures of the image acquisition devices corresponding to the multiple images in the target image set for light field reconstruction, and obtains a completeness index value of the target image set. According to the completeness index value, the completeness or reconstruction quality of the light field reconstruction model obtained by light field reconstruction based on the target image set can be estimated, so that the estimation result is supported by a scientific basis, avoiding relying solely on the experience of the staff and being unable to guarantee the quality of the light field reconstruction model.

[0160] In some embodiments, the determining module 1320 includes:

[0161] A first determining unit, configured to determine a target point in the target space as a reference point based on the target structure information;

[0162] The second determining unit is used to determine the integrity index value of the target image set based on the relationship between the reference point and the multiple target postures.

[0163] In some embodiments, the second determining unit is further configured to:

[0164] Determine first information corresponding to the reference point, where the first information is used to represent a proportion of target postures in the multiple target postures that can observe the reference point;

[0165] Based on the first information, a completeness index value of the target image set is determined.

[0166] In some embodiments, the second determining unit is further configured to:

[0167] Determining second information corresponding to the reference point, the second information being used to characterize a distribution uniformity of angles between a line connecting the reference point and the positions of the multiple target postures and an observation direction of the multiple target postures;

[0168] Based on the second information, a completeness index value of the target image set is determined.

[0169] In some embodiments, the determining module 1320 includes:

[0170] A third determining unit is used to determine distribution information of the image acquisition points corresponding to the multiple target postures in the target space based on the target structure information;

[0171] A fourth determining unit is used to determine a completeness index value of the target image set based on the distribution information.

[0172] In some embodiments, the determining module 1320 is further configured to:

[0173] Based on the integrity index value of the target image set and the target structure information, a newly added posture is determined, and the newly added posture is used to collect a new image of the target space and add it to the target image set.

[0174] In some embodiments, the apparatus further comprises:

[0175] An output module is used to output the newly added posture to a control unit of the image acquisition device, and the control unit is used to control the image acquisition device to acquire the new image of the target space based on the newly added posture.

[0176] In some embodiments, the apparatus further comprises:

[0177] The animation generation module is used to generate an image acquisition guidance animation based on the newly added posture, and the image acquisition guidance animation is used to guide the user to acquire the new image of the target space through the image acquisition device according to the newly added posture.

[0178] In some embodiments, the apparatus further comprises:

[0179] A display module is used to display the integrity information corresponding to the integrity index value on the display interface of the image acquisition device.

[0180] It should be noted here that the above-mentioned image set processing device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned image set processing method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0181] Fig.14 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Fig.14 As shown, the electronic device may include: a processor 1410, a communication interface 1420, a memory 1430 and a communication bus 1440, wherein the processor 1410, the communication interface 1420 and the memory 1430 communicate with each other through the communication bus 1440. The processor 1410 may call the executable data instructions stored in the memory 1430 to execute part or all of the steps in the image set processing method provided in the above embodiments.

[0182] In addition, the executable data instructions stored in the above-mentioned memory 1430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0183] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, part or all of the steps in the image set processing method provided in the above embodiments are implemented.

[0184] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute some or all of the steps in the image set processing method provided in the above embodiments.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0186] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0187] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0188] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0190] The above description is merely an optional embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for processing an image set, comprising: Obtaining target structure information of the target space, wherein the target structure information is obtained based on a surface reconstruction result corresponding to the target space; Determining a completeness index value of a target image set based on the target structure information and a plurality of target poses; Among them, the multiple target postures are the postures of the image acquisition devices corresponding to the multiple images in the target image set, and the target image set is used to reconstruct the light field of the target space.

2. The image set processing method according to claim 1, wherein determining the integrity index value of the target image set based on the target structure information and multiple target poses comprises: Based on the target structure information, determining a target point in the target space as a reference point; Based on the relationship between the reference point and the plurality of target poses, a completeness index value of the target image set is determined.

3. The image set processing method according to claim 2, wherein determining the integrity index value of the target image set based on the relationship between the reference point and the plurality of target poses comprises: Determine first information corresponding to the reference point, where the first information is used to represent a proportion of target postures in the multiple target postures that can observe the reference point; Based on the first information, a completeness index value of the target image set is determined.

4. The image set processing method according to claim 2 or 3, wherein determining the integrity index value of the target image set based on the relationship between the reference point and the plurality of target poses comprises: Determining second information corresponding to the reference point, the second information being used to characterize a distribution uniformity of angles between a line connecting the reference point and the positions of the multiple target postures and an observation direction of the multiple target postures; Based on the second information, a completeness index value of the target image set is determined.

5. The image set processing method according to claim 1, wherein determining the integrity index value of the target image set based on the target structure information and multiple target poses comprises: Based on the target structure information, determining distribution information of the image acquisition points corresponding to the multiple target postures in the target space; Based on the distribution information, a completeness index value of the target image set is determined.

6. The image set processing method according to any one of claims 1 to 3 or 5, further comprising: Based on the integrity index value of the target image set and the target structure information, a newly added posture is determined, and the newly added posture is used to collect a new image of the target space and add it to the target image set.

7. The image set processing method according to claim 6, further comprising: The newly added posture is output to a control unit of the image acquisition device, and the control unit is used to control the image acquisition device to acquire the new image of the target space based on the newly added posture.

8. The image set processing method according to claim 6, further comprising: Based on the newly added posture, an image acquisition guiding animation is generated, and the image acquisition guiding animation is used to guide the user to acquire the new image of the target space through the image acquisition device according to the newly added posture.

9. The image set processing method according to any one of claims 1 to 3 or 5, further comprising: The integrity information corresponding to the integrity index value is displayed on the display interface of the image acquisition device.

10. An image set processing device, comprising: An acquisition module, used for acquiring target structure information of a target space, wherein the target structure information is obtained based on a surface reconstruction result corresponding to the target space; A determination module, used to determine the integrity index value of the target image set based on the target structure information and multiple target postures; Among them, the multiple target postures are the postures of the image acquisition devices corresponding to the multiple images in the target image set, and the target image set is used to reconstruct the light field of the target space.