Test method and device of model reconstruction module, equipment, medium

Abstract

The application discloses a model reconstruction module test method and device, equipment and medium. The model reconstruction module test method comprises the following steps: obtaining a test reconstruction result of a model reconstruction module, wherein the test reconstruction result comprises a target model obtained by reconstructing a target object and / or performance loss data of a reconstruction process; and determining a performance characteristic parameter of the model reconstruction module by using the test reconstruction result. The above scheme can test the performance of the model reconstruction module.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a model reconstruction module testing method and device, equipment and medium. BACKGROUND

[0002] In the current field of computer vision, the real physical world needs to be reconstructed and restored by relying on data and algorithms. The accuracy and precision of the reconstructed model and the reconstruction algorithm can only be evaluated by using the completed model in the corresponding application program and then subjectively feeling during use. However, this method is more subjective, resulting in inaccurate evaluation results of the algorithm accuracy. SUMMARY

[0003] The present application provides at least a model reconstruction module testing method and device, equipment and medium.

[0004] The present application provides a model reconstruction module testing method, which comprises: obtaining a test reconstruction result of a model reconstruction module, the test reconstruction result comprising: a target model obtained by reconstructing a target object and / or performance loss data in a reconstruction process; and determining a performance characterization parameter of the model reconstruction module by using the test reconstruction result.

[0005] Therefore, by obtaining the target model and / or performance loss data in the reconstruction process of the model reconstruction module based on the visual data of the target object, the performance characterization parameter of the model reconstruction module can be determined by using the corresponding target model and / or performance loss data, the testing of the model reconstruction module is realized, subjective feeling during use of the target model is not required, the method provided by the present application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0006] The performance characterization parameter of the model reconstruction module is determined by using the test reconstruction result, comprising: obtaining target data of the target model; and comparing the true value data with the target data to obtain a reconstruction accuracy parameter of the model reconstruction module, wherein the true value data represents appearance data of the target object, and the target data comprises appearance data of the target model.

[0007] Therefore, by comparing the target data of the target model with the true value data, the difference between the appearance data of the target model and the appearance data of the target object can be obtained, and the reconstruction accuracy parameter of the model reconstruction module can be obtained.

[0008] The ground truth data includes ground truth geometric data of the target object, the target data includes target geometric data of the target model, the reconstruction accuracy parameter includes a first reconstruction accuracy parameter, and the first reconstruction accuracy parameter represents a difference between the ground truth geometric data and the target geometric data; and / or, the ground truth data includes a ground truth pose of a surface plane of the target object, the target data includes a target pose of a corresponding target plane on the target model, the reconstruction accuracy parameter includes a second reconstruction accuracy parameter, and the second reconstruction accuracy parameter represents a difference between the ground truth pose of the surface plane and the target pose of the corresponding target plane on the target model.

[0009] Therefore, by obtaining the geometric parameter data of the target object and the target model, the first reconstruction accuracy of the model reconstruction module can be determined. In addition, the second reconstruction accuracy of the model reconstruction module is obtained by the difference between the surface plane of the target object and the target pose of the corresponding target plane on the target model. Thus, the reconstruction accuracy of the model reconstruction module can be obtained.

[0010] The model reconstruction module includes a dense reconstruction submodule, and the dense reconstruction submodule is used to perform the step of model reconstruction on the target object; the reconstruction accuracy parameter of the model reconstruction module is obtained by comparing the ground truth data with the target data, including: obtaining the first reconstruction accuracy parameter based on the difference between the ground truth geometric data and the target geometric data; and / or, the model reconstruction module includes a plane detection submodule, and the plane detection submodule is used to perform the step of model reconstruction on the target object; the reconstruction accuracy parameter of the model reconstruction module is obtained by comparing the ground truth data with the target data, including: obtaining the second reconstruction accuracy parameter based on the difference between the ground truth pose of the surface plane and the target pose of the corresponding target plane on the target model.

[0011] Therefore, by using the difference between the pose of each surface plane and the pose of the target plane to determine the second reconstruction accuracy parameter in the case of using the plane detection submodule to perform model reconstruction on the target object, or by obtaining the first reconstruction accuracy parameter based on the difference between the ground truth geometric data and the target geometric data in the case of using the dense reconstruction submodule to perform model reconstruction on the target object. Thus, the corresponding test method is used for different model reconstruction algorithms, thereby improving the accuracy of the obtained reconstruction accuracy parameter of the model reconstruction module.

[0012] Before the ground truth data and the target data are compared to obtain the reconstruction accuracy parameter of the model reconstruction module, the method further includes: measuring the target object to obtain the ground truth data; or, establishing a ground truth model by using the measurement data of the target object, obtaining appearance data of the ground truth model, and obtaining the ground truth data.

[0013] Therefore, by measuring the target object or establishing a true value model by using the measurement data of the target object, corresponding true value data is obtained, so that the true value data can intuitively reflect the appearance of the target object.

[0014] The test reconstruction result of the model reconstruction module is obtained by: obtaining visual data of the target object, and running the model reconstruction module to reconstruct a model of the target object based on the visual data to obtain the test reconstruction result; or receiving the test reconstruction result sent by a test device in which the model reconstruction module is installed.

[0015] Therefore, by providing multiple ways of obtaining the test reconstruction result, the ways of obtaining the test reconstruction result are diversified.

[0016] The performance loss data includes at least one of a processing time consumption and a resource occupation in the reconstruction process; and / or the performance characterization parameter of the model reconstruction module is determined by using the reconstruction result, including: taking the performance loss data as the performance loss parameter of the model reconstruction module, or statistically analyzing the performance loss data to obtain the performance loss parameter of the model reconstruction module.

[0017] Therefore, by using the processing time consumption and the resource occupation in the reconstruction process, the performance characterization parameter of the model reconstruction module can be determined to test the performance of the model reconstruction module. In addition, by directly taking the performance loss data as the performance loss parameter of the model reconstruction module, or statistically analyzing the performance loss data to obtain the performance loss parameter of the model reconstruction module, the performance loss parameter of the model reconstruction module is diversified.

[0018] The processing time consumption includes single-frame processing time consumption and plane processing time consumption in the reconstruction process, the single-frame processing time consumption includes at least one of average single-frame processing time consumption and each-frame processing time consumption in the reconstruction process, and the plane processing time consumption includes average single-plane reconstruction time consumption and each-plane reconstruction time consumption in the reconstruction process; and the resource occupation includes at least one of central processing unit occupation, graphics processing unit occupation, and memory occupation.

[0019] Therefore, by obtaining the processing time consumption and the plane processing time consumption in the reconstruction process, the performance characterization parameter determined is more accurate.

[0020] After the performance characterization parameter of the model reconstruction module is determined by using the reconstruction result, the method further includes: for each performance characterization parameter, determining a performance type corresponding to the performance characterization parameter according to a numerical interval in which the performance characterization parameter is located.

[0021] Therefore, by determining the performance type corresponding to the performance characterization parameter, the performance of the model reconstruction module can be intuitively observed.

[0022] The test method of the model reconstruction module is performed before the model reconstruction module is integrated into the application program.

[0023] Therefore, by performing the test method of the model reconstruction module before the model reconstruction module is integrated into the application program, the model reconstruction module can be decoupled from other modules of the application program. Compared with testing the performance of the model reconstruction module in the application program by using the integrated application program, only the model reconstruction module is tested, which can reduce the influence of other modules of the application program on the reconstructed model, thereby accurately determining the performance of the model reconstruction module itself and reducing the positioning analysis of the problems obtained by testing. In addition, performing the test method of the model reconstruction module before the model reconstruction module is integrated into the application program can also reduce the probability of integrating the model reconstruction module with low accuracy and precision into the application program.

[0024] The model reconstruction module is integrated into an augmented reality application program.

[0025] Therefore, by integrating the model reconstruction module into the augmented reality application program, the performance of the model reconstruction module for the augmented reality application program can be accurately evaluated, so that the model reconstruction module can be accurately improved based on the performance, and the accuracy of the model reconstruction of the augmented reality application program can be improved subsequently.

[0026] The application provides a test device of a model reconstruction module, which comprises a data acquisition unit configured to acquire a test reconstruction result of the model reconstruction module, the test reconstruction result comprising a target model obtained by reconstructing a target object and / or performance loss data of a reconstruction process; and an evaluation unit configured to determine a performance characterization parameter of the model reconstruction module by using the test reconstruction result.

[0027] The application provides an electronic device comprising a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the test method of the model reconstruction module.

[0028] The application provides a test system of a model reconstruction module, which comprises a test device and an evaluation device, wherein the test device is configured to perform model reconstruction on a target object by using the model reconstruction module to obtain a test reconstruction result, and the evaluation device is configured to perform the test method of the model reconstruction module.

[0029] The application provides a computer-readable storage medium having program instructions stored thereon, wherein the program instructions are executed by a processor to implement the test method of the model reconstruction module.

[0030] The above scheme can determine the performance characterization parameter of the model reconstruction module through the corresponding target model and / or performance loss data after the model reconstruction module based on the visual data of the target object reconstructs the target object to obtain the target model and / or performance loss data in the reconstruction process, realize the test of the model reconstruction module, and does not need to perform subjective feeling in the use process of the target model. The method provided in the application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0031] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0033] Figure 1 is a flowchart of an embodiment of the test method of the model reconstruction module of the present application;

[0034] Figure 2 is a flowchart of step S12 in an embodiment of the test method of the model reconstruction module of the present application;

[0035] Figure 3 is a structural schematic diagram of an embodiment of the test device of the model reconstruction module of the present application;

[0036] Figure 4 is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0037] Figure 5 is a structural schematic diagram of an embodiment of the test system of the model reconstruction module of the present application;

[0038] Figure 6 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0039] The scheme of the embodiments of the present application will be described in detail below in conjunction with the drawings of the specification.

[0040] In the following description, specific details are set forth in order to provide a thorough understanding of the present application, such as particular system structures, interfaces, techniques, etc., but the present application is not limited to the specific details.

[0041] The term "and / or", merely describes an associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship. In addition, "multiple" in this paper means two or more than two. In addition, the term "at least one" in this paper means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C, which can mean including any one or more elements selected from the set consisting of A, B and C.

[0042] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of a test method of a model reconstruction module of the present application. Specifically, it can include the following steps:

[0043] Step S11: Obtain the test reconstruction result of the model reconstruction module, and the test reconstruction result includes: the target model obtained by reconstructing the target object and / or the performance loss data in the reconstruction process.

[0044] Wherein, the target object can be a real physical entity, or a virtual object. The target model obtained by model reconstruction of the target object can be a three-dimensional model. The performance loss data in the reconstruction process can be the loss of resources of the device for model reconstruction, such as the loss of time resources, the loss of storage resources, etc. It can be understood that the step of model reconstruction of the target object by the model reconstruction module can be executed by the execution device of the present test method, or by other test devices, and the test reconstruction result is sent to the execution device of the present test method by other test devices, so the acquisition method of the test reconstruction result is not limited here.

[0045] Step S12: Determine the performance characterization parameter of the model reconstruction module by using the test reconstruction result.

[0046] In some disclosed embodiments, the performance characterization parameter of the model reconstruction module is determined by using the target model obtained by model reconstruction of the target object.

[0047] In some disclosed embodiments, the performance characterization parameter of the model reconstruction module is determined by using the performance loss data in the process of model reconstruction of the target object.

[0048] In some disclosed embodiments, the performance characterization parameter of the model reconstruction module is determined by using the target model obtained by model reconstruction of the target object and the performance loss data in the model reconstruction process.

[0049] The above solution obtains the target model and / or performance loss data during the reconstruction process by acquiring the visual data of the target object through the model reconstruction module. Then, the performance characterization parameters of the model reconstruction module can be determined through the corresponding target model and / or performance loss data, thus enabling the testing of the model reconstruction module. There is no need to make subjective judgments during the use of the target model. The method provided by this application is more objective, making the evaluation results of the model reconstruction module more accurate.

[0050] Please also see Figure 2 , Figure 2 This is a schematic diagram of a sub-process of step S12, illustrating an embodiment of the test method for the model reconstruction module of this application. Figure 2 As shown, step S12 above may include the following steps:

[0051] Step S121: Obtain the target data of the target model.

[0052] The target data of the target model may include the appearance data of the target model. For example, the appearance data may include geometric data, pose data, etc.

[0053] In some disclosed embodiments, the target data includes the target geometric data of the target model. The target geometric data can be the dimensions of the target model in at least one direction, such as the dimensions of multiple parts of the target model in the vertical direction, the horizontal direction, etc.

[0054] In some disclosed embodiments, the target data includes the target pose of the target plane corresponding to the target model. Generally, the target model is a three-dimensional model, which can be composed of multiple planes. For example, in the process of constructing a three-dimensional model of a target object based on a plane detection algorithm, multiple planes are first constructed based on the captured image, and then the three-dimensional model is reconstructed based on the constructed multiple planes. The multiple planes constituting the target model can be considered as the target planes described in the embodiments of this disclosure. The target pose of the target plane can be considered as the pose of the target plane in the world coordinate system, or the relative pose with respect to a preset reference object. The pose includes position and orientation, etc. The orientation can be the angle between the plane and the preset plane. For example, if a target model of a building in a scene is obtained by model reconstruction, multiple target planes constitute the facade of the target model, and the pose of each target plane can be the pose relative to a preset reference object (e.g., the base of the building), or the pose of each target plane in the world coordinate system.

[0055] Step S122: Compare the true data with the target data to obtain the reconstruction accuracy parameters of the model reconstruction module.

[0056] The true value data represents appearance data of the target object.

[0057] In some disclosed embodiments, the true value data comprises true value geometry data of the target object. The true value geometry data can be a dimension of the target object in at least one direction, for example, a dimension of each part of the target object in a vertical direction, a dimension of each part of the target object in a horizontal direction, and the like.

[0058] In some disclosed embodiments, the true value data comprises a true value pose of a surface plane of the target object. The true value pose of the surface plane of the target object can be a pose of the surface plane in a world coordinate system or a relative pose between the surface plane and a preset reference object. The pose comprises a position and an orientation, and the like. The orientation can be an included angle between the surface plane and a preset plane. The target object can have multiple surface planes, each of which corresponds to a target plane of the target model.

[0059] Before step S122 is performed, the following steps can also be performed:

[0060] The true value data is obtained by measuring the target object or by establishing a true value model using measurement data of the target object. The true value geometry data of the target object can be obtained by measuring the target object and using the measurement results as the true value geometry data of the target object. Alternatively, the true value geometry data of the target object can be obtained by scanning the target object and using the geometry data of the three-dimensional model obtained by scanning as the true value geometry data. The surface plane of the target object can be obtained by measuring the target object and using modeling software (for example, 3dsmax or Maya) to establish multiple plane models as true value models for each target plane. The true value model can be a three-dimensional model of the target object, and the true value data of the target object can be obtained based on the three-dimensional model. The true value model can also be multiple three-dimensional planes used to constitute the three-dimensional model, and the three-dimensional planes are used as the surface planes of the target object. The poses of the surface planes are marked as the true value poses.

[0061] The true value data is obtained by measuring the target object or by establishing a true value model using measurement data of the target object, so that the true value data can intuitively reflect the appearance of the target object.

[0062] In some application scenarios, the true value data comprises true value geometry data of the target object, and the target data comprises target geometry data of the target model. The reconstruction accuracy parameter comprises a first reconstruction accuracy parameter, wherein the first reconstruction accuracy parameter represents a difference between the true value geometry data and the target geometry data. The first reconstruction accuracy parameter can be a size difference.

[0063] In some application scenarios, the ground truth data includes ground truth data of a surface of the target object, and the target data includes a target pose of a corresponding target plane on the target model. The reconstruction accuracy parameter includes a second reconstruction accuracy parameter. The second reconstruction accuracy parameter can be used to represent a difference between the ground truth pose of the surface plane and the target pose of the corresponding target plane on the target model.

[0064] By obtaining the geometric parameter data of the target object and the target model, the first reconstruction accuracy of the model reconstruction module can be determined. In addition, the second reconstruction accuracy of the model reconstruction module can be obtained by the difference between the surface plane of the target object and the target pose of the corresponding target plane on the target model. Thus, the reconstruction accuracy of the model reconstruction module can be obtained.

[0065] In some disclosed embodiments, the model reconstruction module includes a dense reconstruction sub-module. The dense reconstruction sub-module is configured to perform the step of model reconstruction of the target object. The dense reconstruction sub-module can perform the step of model reconstruction of the target object by using a dense reconstruction algorithm to perform three-dimensional reconstruction of the target object. For example, the target model can be obtained by performing dense mesh or point cloud reconstruction of the target object. In the case where the model reconstruction module includes the dense reconstruction sub-module, the step S122 includes the following step: obtaining the first reconstruction accuracy parameter based on the difference between the ground truth geometric data and the target geometric data. For example, if the ground truth geometric data includes the size of the target object in several directions, such as the height in the vertical direction, and the target geometric data includes the size of the target model in the corresponding several directions, such as the height in the vertical direction, the difference between the ground truth geometric data and the target geometric data includes the size difference of the target object and the target model in the several directions. Specifically, it can be the average of the size difference in each direction. The several directions refer to one or more directions.

[0066] In some disclosed embodiments, the model reconstruction module comprises a plane detection sub-module. The plane detection sub-module is configured to perform the step of reconstructing the target object. In some embodiments, the plane detection sub-module is configured to perform the step of reconstructing the target object by using a plane detection algorithm to reconstruct the target object in three dimensions. In some embodiments, when the model reconstruction module comprises the plane detection sub-module, the step S122 comprises the following step: obtaining a second reconstruction accuracy parameter based on a difference between the ground truth pose of the table plane and the target pose of the corresponding target plane on the target model. Specifically, the difference between the ground truth pose of the table plane and the target pose of the corresponding target plane on the target model is obtained. In some embodiments, the second reconstruction accuracy parameter is obtained based on the difference between the poses of at least some of the target planes and the corresponding table planes. In some embodiments, the at least some of the target planes and the corresponding table planes can be one or more. For example, the second reconstruction accuracy parameter can be obtained based on an average of the differences between the poses of a plurality of target planes and the corresponding table planes.

[0067] By comparing the target data of the target model with the ground truth data, the difference between the appearance data of the reconstructed target model and the appearance data of the target object can be obtained, and thus the reconstruction accuracy parameter of the model reconstruction module can be obtained.

[0068] In addition, in some embodiments, the second reconstruction accuracy parameter is obtained based on the difference between the poses of each table plane and the corresponding target plane when the plane detection sub-module is used to reconstruct the target object. In some embodiments, the first reconstruction accuracy parameter is obtained based on the difference between the ground truth geometry data and the target geometry data when the dense reconstruction sub-module is used to reconstruct the target object. In some embodiments, different model reconstruction algorithms are tested using corresponding test methods, and thus the accuracy of the obtained reconstruction accuracy parameter of the model reconstruction module is improved.

[0069] In some disclosed embodiments, the step S11 can comprise the following steps:

[0070] The visual data of the target object is acquired, and a model reconstruction module is run to perform model reconstruction on the target object based on the visual data to obtain a test reconstruction result. Alternatively, the test reconstruction result sent by a test device in which the model reconstruction module is installed is received. The visual data of the target object can be obtained in the manner that a plurality of image data of the target object in different dimensions are captured by a shooting module provided in an execution device of a test method of the model reconstruction module according to the embodiments of the present disclosure. Then, the image data is processed to obtain the visual data of the target object. The visual data can be any data for representing the appearance of the target object, such as the pose, external structure size, and the like of the target object. The model reconstruction module can be run to perform model reconstruction on the target object based on the visual data to obtain the test reconstruction result in the manner that a dense reconstruction submodule is used to perform model reconstruction on the target object based on the visual data to obtain the test reconstruction result, or a plane detection submodule is used to perform model reconstruction on the target object based on the visual data to obtain the test reconstruction result. The specific manner of performing three-dimensional reconstruction on the target object according to the plane detection algorithm or the specific manner of performing three-dimensional reconstruction on the target object according to the dense reconstruction algorithm can be referred to general technology, and will not be described in detail here.

[0071] That is, the device for performing model reconstruction on the target object based on the visual data to obtain the test reconstruction result can be the same device as the execution device of the test method of the model reconstruction module according to the embodiments of the present disclosure, or can not be the same device. The model reconstruction module in the test device is run to perform model reconstruction on the target object based on the visual data to obtain the test reconstruction result, and then the test reconstruction result sent by the test device is received to determine the performance characterization parameter of the model reconstruction module in the test device.

[0072] By providing a plurality of ways of obtaining the test reconstruction result, the ways of obtaining the test reconstruction result are diversified.

[0073] In some embodiments of the present disclosure, the performance consumption data includes at least one of the processing time consumption and resource occupation in the reconstruction process. The performance consumption data specifically includes performance consumption data in the entire model reconstruction process. The performance consumption data in the model reconstruction process can be stored in a log of the test device. In the case that the test device is the execution device, the performance consumption data is obtained by reading the log. In the case that the test device is not the execution device, the performance consumption data is obtained by receiving the performance consumption data sent by the test device.

[0074] The processing time consumption includes single-frame processing time consumption in the reconstruction process and plane processing time consumption. The single-frame processing time consumption includes at least one of single-frame average processing time consumption and each-frame processing time consumption in the reconstruction process. The plane processing time consumption includes average reconstruction time consumption of a single plane and reconstruction time consumption of each plane in the reconstruction process. The resource occupation condition includes at least one of central processor occupation condition, graphic processor occupation condition and memory occupation condition. As described above, the model reconstruction module can include a plane detection sub-module or a dense reconstruction sub-module. The single-frame processing time consumption and the plane processing time consumption can be single-frame processing time consumption and plane processing time consumption of the plane detection sub-module in the model reconstruction process, or single-frame processing time consumption and plane processing time consumption of the dense reconstruction sub-module in the model reconstruction process.

[0075] As described above, the model reconstruction process of the target object can be based on processing of a plurality of images taken of the target object, for example, feature extraction and feature matching of the plurality of images, and then obtaining a plurality of target planes according to the image processing result, and then constructing a target model of the target object based on the plurality of planes. The each-frame processing time consumption can be the time consumed for processing each frame of image. The single-frame average processing time consumption is the average of each-frame processing time consumption. The reconstruction time consumption of a single plane in the reconstruction process refers to the time consumed for reconstructing the plane. The average reconstruction time consumption of a single plane in the reconstruction process refers to the average of the reconstruction time consumption of each plane.

[0076] Exemplarily, for the plane detection algorithm, a plurality of plane models are built as true values according to the actual measured size of the target object by using modeling software (such as 3dsmax or Maya); then the position accuracy error between each target plane of the target model and the true value plane is evaluated. And the execution time of the plane detection algorithm for each frame of image in the reconstruction process of the target object is recorded in the log, and the average single-frame time consumption of the plane detection is counted through the execution time of not less than 5 min, which includes the average time consumption of single plane detection and 5 plane detections.

[0077] Exemplarily, for the dense point cloud / dense mesh algorithm, the scanned three-dimensional model of the target object is taken as a true value, the dense mesh / point cloud reconstruction is performed on the scene, and the geometric accuracy error between the reconstructed dense mesh / dense point cloud and the true value three-dimensional model is evaluated. And the average single-frame time consumption of the dense mesh / dense point cloud reconstruction is counted through the execution time of not less than 5 min.

[0078] By means of the processing time consumption and the resource occupation condition in the reconstruction process, the performance characterization parameter of the model reconstruction module can be determined, and the performance of the model reconstruction module can be tested. In addition, by means of the multi-aspect processing time consumption and the plane processing time consumption in the reconstruction process, the performance characterization parameter determined is more accurate.

[0079] In some disclosed embodiments, the step S12 can further include the following steps:

[0080] The performance loss data is taken as the performance loss parameter of the model reconstruction module. Alternatively, the performance loss data is statistically analyzed to obtain the performance loss parameter of the model reconstruction module. The statistical analysis can be statistical average, variance, median, mode, etc. By directly taking the performance loss data as the performance loss parameter of the model reconstruction module, or statistically analyzing the performance loss data to obtain the performance loss parameter of the model reconstruction module, the performance loss parameter of the model reconstruction module is more diversified.

[0081] In some disclosed embodiments, after the step S12 is performed, the following steps can be further performed:

[0082] For each performance characterization parameter, a performance type corresponding to the performance characterization parameter is determined according to the numerical interval in which the performance characterization parameter is located. Optionally, the performance characterization parameter includes a performance characterization parameter related to reconstruction accuracy and a performance characterization parameter related to performance loss data.

[0083] For example, for the single-frame processing time, if the single-frame processing time is less than 200 ms, the performance type corresponding to the performance loss data is determined to be excellent, if the single-frame processing time is greater than or equal to 200 ms and less than or equal to 1 s, the performance type corresponding to the performance loss data is determined to be general, and if the single-frame processing time is greater than 1 s, the performance type corresponding to the performance loss data is determined to be poor.

[0084] For example, for the reconstruction accuracy, if the reconstruction accuracy is less than 0.5 m, the performance type corresponding to the reconstruction accuracy is determined to be excellent, if the reconstruction accuracy is greater than or equal to 0.5 m and less than or equal to 1 m, the performance type corresponding to the reconstruction accuracy is determined to be general, and if the reconstruction accuracy is greater than 1 m, the performance type corresponding to the reconstruction accuracy is determined to be poor.

[0085] By determining the performance type corresponding to the performance characterization parameter, the performance of the model reconstruction module can be intuitively observed.

[0086] In some disclosed embodiments, the model reconstruction module can be integrated in an augmented reality application. For example, the model reconstruction module can be integrated in an AR game, an AR navigation, an AR tour guide, and an AR large screen. By integrating the model reconstruction module in the augmented reality application, the performance of the model reconstruction module for the augmented reality application can be accurately evaluated, so that the model reconstruction module can be accurately improved based on the performance, and thus the accuracy of the model reconstruction of the augmented reality application can be improved. The test method of the model reconstruction module is performed before the model reconstruction module is integrated in the application. In some application scenarios, when it is needed to evaluate whether the accuracy of a three-dimensional reconstruction algorithm meets the project requirements, the test method can be used to evaluate the algorithm.

[0087] By performing the test method of the model reconstruction module before the model reconstruction module is integrated in the application, the model reconstruction module can be decoupled from other modules of the application. Compared with testing the performance of the model reconstruction module in the application by using the integrated application, only the model reconstruction module is tested, which can reduce the influence of other modules of the application on the reconstructed model, and thus the performance of the model reconstruction module itself can be accurately determined, and the positioning analysis of the problems obtained by the test can be reduced. In addition, by performing the test method of the model reconstruction module before the model reconstruction module is integrated in the application, the probability of integrating the model reconstruction module with low accuracy and precision into the application can be reduced. In addition, by integrating the model reconstruction module in the augmented reality application, the model detection accuracy of the augmented reality application can be improved.

[0088] The above scheme can determine the performance characterization parameter of the model reconstruction module by the corresponding target model and / or performance loss data after the model reconstruction module reconstructs the target object based on the visual data of the target object to obtain the target model and / or the performance loss data in the reconstruction process, and thus the model reconstruction module can be tested without subjective feeling in the use process of the target model. The method provided in the present application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0089] The execution subject of the test method of the model reconstruction module can be a test device of the model reconstruction module, for example, the test device of the model reconstruction module can be a terminal device or a server or other processing device. The terminal device can be an AR device (for example, AR glasses, and any terminal device installed with an AR application), a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like. In some possible implementation manners, the saliency detection method can be implemented by a processor invoking computer readable instructions stored in a memory.

[0090] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of an embodiment of the test device of the model reconstruction module. The test device of the model reconstruction module 30 includes a data acquisition module 31 and an evaluation module 32. The data acquisition unit 31 is configured to acquire a test reconstruction result of the model reconstruction module, and the test reconstruction result includes a target model obtained by reconstructing a target object and / or performance loss data of a reconstruction process. The evaluation unit 32 is configured to determine a performance characterization parameter of the model reconstruction module by using the test reconstruction result.

[0091] The above scheme can determine the performance characterization parameter of the model reconstruction module by using the corresponding target model and / or performance loss data after the model reconstruction module reconstructs the target object based on the visual data of the target object to obtain the target model and / or the performance loss data in the reconstruction process, thereby testing the model reconstruction module without subjective feeling in the use process of the target model. The method provided in the application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0092] In some disclosed embodiments, the evaluation unit 32 determines the performance characterization parameter of the model reconstruction module by using the test reconstruction result, including: acquiring target data of the target model; and comparing the true value data with the target data to obtain a reconstruction accuracy parameter of the model reconstruction module, wherein the true value data represents appearance data of the target object, and the target data includes appearance data of the target model.

[0093] The above scheme can obtain the difference between the appearance data of the target model and the appearance data of the target object by comparing the target data of the target model with the true value data, thereby obtaining the reconstruction accuracy parameter of the model reconstruction module.

[0094] In some disclosed embodiments, the ground truth data includes ground truth geometry data of the target object, the target data includes target geometry data of the target model, the reconstruction accuracy parameter includes a first reconstruction accuracy parameter, and the first reconstruction accuracy parameter represents a difference between the ground truth geometry data and the target geometry data; and / or, the ground truth data includes a ground truth pose of a surface plane of the target object, the target data includes a target pose of a corresponding target plane on the target model, the reconstruction accuracy parameter includes a second reconstruction accuracy parameter, and the second reconstruction accuracy parameter represents a difference between the ground truth pose of the surface plane and the target pose of the corresponding target plane on the target model.

[0095] The above scheme can determine the first reconstruction accuracy of the model reconstruction module by obtaining the geometry parameter data of the target object and the target model. In addition, the second reconstruction accuracy of the model reconstruction module can be obtained by the difference between the surface plane of the target object and the target pose of the corresponding target plane on the target model. Thus, the reconstruction accuracy of the model reconstruction module can be obtained.

[0096] In some disclosed embodiments, the model reconstruction module includes a dense reconstruction sub-module, and the dense reconstruction sub-module is configured to perform the step of model reconstruction on the target object; the evaluation unit 32 obtains the reconstruction accuracy parameter of the model reconstruction module by comparing the ground truth data with the target data, including: obtaining the first reconstruction accuracy parameter based on the difference between the ground truth geometry data and the target geometry data; and / or, the model reconstruction module includes a plane detection sub-module, and the plane detection sub-module is configured to perform the step of model reconstruction on the target object; the evaluation unit 32 obtains the reconstruction accuracy parameter of the model reconstruction module by comparing the ground truth data with the target data, including: obtaining the second reconstruction accuracy parameter based on the difference between the ground truth pose of the surface plane and the target pose of the corresponding target plane on the target model.

[0097] The above scheme can determine the second reconstruction accuracy parameter by using the difference between the pose of each surface plane and the pose of the target plane in the case of using the plane detection sub-module to perform model reconstruction on the target object. Or, the first reconstruction accuracy parameter can be obtained based on the difference between the ground truth geometry data and the target geometry data in the case of using the dense reconstruction sub-module to perform model reconstruction on the target object. Thus, the corresponding test method can be used for different model reconstruction algorithms, thereby improving the accuracy of the obtained reconstruction accuracy parameter of the model reconstruction module.

[0098] In some disclosed embodiments, before obtaining the reconstruction accuracy parameter of the model reconstruction module by comparing the ground truth data with the target data, the evaluation unit 32 is further configured to: measure the target object to obtain the ground truth data; or, establish a ground truth model by using the measurement data of the target object, obtain appearance data of the ground truth model, and obtain the ground truth data.

[0099] The scheme can reflect the appearance of the target object intuitively by measuring the target object or establishing a true value model by using the measurement data of the target object to obtain corresponding true value data.

[0100] In some disclosed embodiments, the data acquisition module 31 acquires the test reconstruction result of the model reconstruction module, including: acquiring visual data of the target object, running the model reconstruction module to perform model reconstruction on the target object based on the visual data to obtain the test reconstruction result; or receiving the test reconstruction result sent by the test device in which the model reconstruction module is installed.

[0101] The above scheme provides multiple ways to acquire the test reconstruction result, so that the ways to acquire the test reconstruction result are diversified.

[0102] In some disclosed embodiments, the performance loss data includes at least one of a processing time consumption and a resource occupation in the reconstruction process; and / or the evaluation unit 32 determines the performance characterization parameter of the model reconstruction module by using the reconstruction result, including: taking the performance loss data as the performance loss parameter of the model reconstruction module, or statistically analyzing the performance loss data to obtain the performance loss parameter of the model reconstruction module.

[0103] The above scheme can determine the performance characterization parameter of the model reconstruction module based on the processing time consumption and the resource occupation in the reconstruction process, so as to test the performance of the model reconstruction module. In addition, by directly taking the performance loss data as the performance loss parameter of the model reconstruction module, or statistically analyzing the performance loss data to obtain the performance loss parameter of the model reconstruction module, the performance loss parameter of the model reconstruction module is diversified.

[0104] In some disclosed embodiments, the processing time consumption includes single-frame processing time consumption and plane processing time consumption in the reconstruction process, the single-frame processing time consumption includes at least one of single-frame average processing time consumption and per-frame processing time consumption in the reconstruction process, and the plane processing time consumption includes average reconstruction time consumption of a single plane and reconstruction time consumption of each plane in the reconstruction process; and the resource occupation includes at least one of central processing unit occupation, graphics processing unit occupation, and memory occupation.

[0105] The above scheme acquires multiple aspects of processing time consumption and plane processing time consumption in the reconstruction process, so that the determined performance characterization parameter is more accurate.

[0106] In some disclosed embodiments, after the evaluation unit 32 determines the performance characterization parameter of the model reconstruction module by using the reconstruction result, the evaluation unit 32 is further configured to: for each performance characterization parameter, determine a performance type corresponding to the performance characterization parameter according to a numerical interval in which the performance characterization parameter is located.

[0107] The above scheme can intuitively observe the performance of the model reconstruction module by determining the performance type corresponding to the performance characterization parameter.

[0108] In some disclosed embodiments, the test method of the model reconstruction module is performed before the model reconstruction module is integrated into the application program.

[0109] The above scheme can decouple the model reconstruction module from other modules of the application program by performing the test method of the model reconstruction module before the model reconstruction module is integrated into the application program. Compared with testing the performance of the model reconstruction module in the application program by using the integrated application program, only the model reconstruction module is tested, which can reduce the influence of other modules of the application program on the reconstructed model, thereby accurately determining the performance of the model reconstruction module itself and reducing the positioning analysis of the problems obtained by testing. In addition, performing the test method of the model reconstruction module before the model reconstruction module is integrated into the application program can also reduce the probability of integrating the model reconstruction module with low accuracy and precision into the application program.

[0110] In some disclosed embodiments, the model reconstruction module is integrated into an augmented reality application program.

[0111] The above scheme can accurately evaluate the performance of the model reconstruction module for the augmented reality application program by integrating the model reconstruction module into the augmented reality application program, so that the model reconstruction module can be accurately improved based on the performance, and the accuracy of the model reconstruction of the augmented reality application program can be improved subsequently.

[0112] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 40 includes a memory 41 and a processor 42, and the processor 42 is configured to execute program instructions stored in the memory 41 to implement the steps in the above-mentioned test method of the model reconstruction module. In one specific implementation scenario, the electronic device 40 can include but is not limited to: a medical device, a microcomputer, a desktop computer, a server, in addition, the electronic device 40 can also include a notebook computer, a tablet computer and other mobile devices, which are not limited here.

[0113] Specifically, the processor 42 is configured to control itself and the memory 41 to implement the steps in any of the embodiments of the training method of the saliency detection model described above. The processor 42 can also be referred to as a CPU (Central Processing Unit). The processor 42 can be an integrated circuit chip having a processing capability of signals. The processor 42 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 42 can be implemented by an integrated circuit chip together.

[0114] The above scheme can determine the performance characterization parameter of the model reconstruction module by the corresponding target model and / or performance loss data after the model reconstruction module reconstructs the target object to obtain the target model and / or performance loss data during the reconstruction process based on the visual data of the target object, implement the test of the model reconstruction module, and does not need to perform subjective feeling in the use process of the target model. The method provided in the present application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0115] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of the test system of the model reconstruction module of the present application. As Figure 5 shown, the test system 50 of the model reconstruction module includes a test device 51 and an evaluation device 52. The test device 51 is configured to reconstruct a target object by using the model reconstruction module to obtain a test reconstruction result, and the evaluation device 52 is configured to execute the test method of the model reconstruction module described above.

[0116] In some application scenarios, after the test device 51 reconstructs a target object by using the model reconstruction module to obtain a test reconstruction result, the test device 51 sends the test reconstruction result to the evaluation device 52, so that the evaluation device 52 determines the performance characterization parameter of the model reconstruction module by using the test reconstruction result after receiving the test reconstruction result.

[0117] After the model reconstruction module based on the visual data of the target object reconstructs a model of the target object and / or performance loss data in the reconstruction process, the performance characterization parameters of the model reconstruction module can be determined through the corresponding target model and / or performance loss data, the model reconstruction module is tested, subjective feelings are not required in the use process of the target model, the method provided in the application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0118] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of an embodiment of the computer-readable storage medium of the application. The computer-readable storage medium 60 stores program instructions 61, and the program instructions 61 are used to implement the steps in the method embodiment of the model reconstruction module test method described above when executed by a processor.

[0119] The above scheme, after the model reconstruction module based on the visual data of the target object reconstructs a model of the target object and / or performance loss data in the reconstruction process, the performance characterization parameters of the model reconstruction module can be determined through the corresponding target model and / or performance loss data, the model reconstruction module is tested, subjective feelings are not required in the use process of the target model, the method provided in the application is more objective, and the evaluation result of the model reconstruction module is more accurate.

[0120] In some embodiments, the device provided by the embodiments of the present disclosure has functions or contains modules which can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, they will not be described here.

[0121] Some embodiments of the present disclosure relate to the field of augmented reality. By obtaining image information of a target object in a real environment, and then using various visual-related algorithms to detect or identify the relevant features, states and attributes of the target object, an AR effect combining virtual and real objects that matches specific applications is obtained. For example, the target object can be related to a face, a limb, a gesture, an action, etc. of a human body, or a marker, a sign, etc. of an object, or a sand table, a display area or a display object, etc. of a venue or a place. The visual-related algorithms can include visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of an object, pose or depth detection of an object, etc. The specific applications can not only include interactive scenarios such as tour guide, navigation, explanation, reconstruction, virtual effect superimposed display, etc. related to real scenes or objects, but also include interactive scenarios such as makeup beautification, limb beautification, special effect display, virtual model display, etc. related to a person. Convolutional neural networks can be used to detect or identify the relevant features, states and attributes of the target object.

[0122] The convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0123] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be referred to each other, and for brevity, details are not repeated herein.

[0124] In several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the above-described device implementation is only schematic, for example, the division of the module or unit is only a logical function division, and actual implementation can have another division manner, for example, the unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0125] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

Claims

1. A method of testing a model reconstruction module, the method comprising: The method comprises the following steps: acquiring a test reconstruction result of a model reconstruction module, the test reconstruction result comprising a target model obtained by reconstructing a target object and performance loss data of a reconstruction process; determining a performance characterization parameter of the model reconstruction module by using the test reconstruction result; wherein the model reconstruction module comprises a plane detection sub-module, and the target model obtained by reconstructing the target object is obtained by: obtaining an image processing result by processing a plurality of images taken of the target object, obtaining a plurality of planes according to the image processing result, and constructing the target model of the target object based on the plurality of planes; the performance loss data comprises single-frame processing time consumption of the reconstruction process, and the single-frame processing time consumption comprises at least one of single-frame average processing time consumption and each-frame processing time consumption, the each-frame processing time consumption being time consumed for processing each of the images, and the single-frame average processing time consumption being an average value of the each-frame processing time consumption; the determining of the performance characterization parameter of the model reconstruction module by using the test reconstruction result comprises: acquiring target data of the target model; comparing ground truth data with the target data to obtain a reconstruction accuracy parameter of the model reconstruction module; wherein the ground truth data comprises a true value pose of a surface plane of the target object, the target data comprises a target pose of a corresponding target plane of the target model, and the reconstruction accuracy parameter comprises a second reconstruction accuracy parameter, the second reconstruction accuracy parameter representing a difference between the true value pose of the surface plane and the target pose of the corresponding target plane of the target model.

2. The method of claim 1, wherein, The ground truth data comprises true value geometric data of the target object, the target data comprises target geometric data of the target model, the reconstruction accuracy parameter comprises a first reconstruction accuracy parameter, and the first reconstruction accuracy parameter represents a difference between the true value geometric data and the target geometric data.

3. The method of claim 2, wherein, The model reconstruction module comprises a dense reconstruction sub-module, and the dense reconstruction sub-module is used to perform a step of model reconstruction of the target object; the comparing of the ground truth data with the target data to obtain the reconstruction accuracy parameter of the model reconstruction module comprises: obtaining the first reconstruction accuracy parameter based on the difference between the true value geometric data and the target geometric data; and / or the comparing of the ground truth data with the target data to obtain the reconstruction accuracy parameter of the model reconstruction module comprises: obtaining the second reconstruction accuracy parameter based on the difference between the true value pose of the surface plane and the target pose of the corresponding target plane of the target model.

4. The method according to any one of claims 1 to 3, characterized in that, Before the comparing of the ground truth data with the target data to obtain the reconstruction accuracy parameter of the model reconstruction module, the method further comprises: measuring the target object to obtain the ground truth data; or establishing a ground truth model by using measurement data of the target object, acquiring appearance data of the ground truth model, and obtaining the ground truth data.

5. The method according to any one of claims 1 to 4, characterized in that, The acquiring of the test reconstruction result of the model reconstruction module comprises: Obtain visual data of the target object, run the model reconstruction module to reconstruct the model of the target object based on the visual data, and obtain the test reconstruction result; or... Receive the test reconstruction results sent by the test equipment that has the model reconstruction module installed.

6. The method according to any one of claims 1 to 5, characterized in that, The performance loss data also includes resource usage; The process of determining the performance characterization parameters of the model reconstruction module using the test reconstruction results includes: The performance loss data can be used as the performance loss parameter of the model reconstruction module, or the performance loss data can be statistically analyzed to obtain the performance loss parameter of the model reconstruction module.

7. The method of claim 6, wherein, The performance loss data also includes the plane processing time, which includes the average reconstruction time of a single plane and the reconstruction time of each plane during the reconstruction process; The resource usage includes at least one of the following: CPU usage, GPU usage, and memory usage.

8. The method according to any one of claims 1 to 7, characterized in that, After determining the performance characterization parameters of the model reconstruction module using the test reconstruction results, the method further includes: For each performance characterization parameter, the performance type corresponding to the performance characterization parameter is determined according to the numerical range in which the performance characterization parameter is located.

9. The method according to any one of claims 1 to 8, characterized in that, The testing method for the model reconstruction module is performed before the model reconstruction module is integrated into the application; And / or, the model reconstruction module is used for integration into augmented reality applications.

10. A test apparatus for a model reconstruction module, characterized in that include: The data acquisition unit is used to acquire the test reconstruction results of the model reconstruction module. The test reconstruction results include: the target model obtained by reconstructing the target object and the performance loss data of the reconstruction process. The performance loss data includes the single-frame processing time of the reconstruction process. The single-frame processing time includes at least one of the single-frame average processing time and the processing time per frame. The processing time per frame is the time consumed in processing each frame of the image. The single-frame average processing time is the average of the processing times per frame. An evaluation unit is used to determine the performance characterization parameters of the model reconstruction module using the test reconstruction results. Determining the performance characterization parameters of the model reconstruction module using the test reconstruction results includes: acquiring target data of the target model; comparing the target data with ground truth data to obtain the reconstruction accuracy parameters of the model reconstruction module; wherein the ground truth data includes the ground truth pose of the surface plane of the target object, the target data includes the target pose of the corresponding target plane on the target model, and the reconstruction accuracy parameters include a second reconstruction accuracy parameter, the second reconstruction accuracy parameter characterizing the difference between the ground truth pose of the surface plane and the target pose of the corresponding target plane on the target model. The model reconstruction module includes a plane detection submodule. The plane detection submodule is used to reconstruct the target object to obtain the target model, which includes: processing several images of the target object to obtain image processing results, obtaining several planes based on the image processing results, and constructing a target model about the target object based on the several planes.

11. An electronic device, comprising: A computer program product comprising a memory and a processor for executing program instructions stored in the memory to implement the method of any one of claims 1 to 9.

12. A test system for a model reconstruction module, characterized in that A testing device and an evaluation device are comprised, wherein the testing device is configured to perform model reconstruction on a target object using the model reconstruction module to obtain a testing reconstruction result, and the evaluation device is configured to implement the method of any one of claims 1 to 9.

13. A computer-readable storage medium having stored thereon program instructions, wherein the program instructions are executable by a computer for causing the computer to carry out the method according to any one of claims 1 to 12. The program instructions, when executed by a processor, implement the method of any one of claims 1 to 9.

Images