Virtual reality and augmented reality fusion method based on indoor 3D reconstruction

By extracting and processing multi-view image features in indoor three-dimensional reconstruction and combining the reconstruction processing model trained by digital sample, the problem of rigid fusion strategy in the existing technology is solved, achieving higher reconstruction accuracy and user experience improvement.

CN118485796BActive Publication Date: 2024-11-15BEIJING DAXI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410948303.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-15
Estimated Expiration
2044-07-16

Smart Images

  • Figure CN118485796B_ABST
    Figure CN118485796B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for merging virtual reality and augmented reality based on indoor three-dimensional reconstruction, and relates to the technical field of computer graphics and virtual reality fusion. The method comprises: reading a three-dimensional reconstruction task and task benchmark information containing a multi-view image of a target, making a fusion rigid requirement decision in combination with the task to determine fusion constraint information, constructing a reconstruction processing model based on digital sample training convergence and containing an affine transformation area and a registration fusion area, combining the model, feature descriptor and fusion constraint information to perform target three-dimensional reconstruction and determine the reconstruction target, randomly determine task verification points, verify and compensate the reconstruction target and visualize it on a terminal reality interface. The method solves the technical problems in the prior art that the indoor three-dimensional reconstruction process of virtual reality and augmented reality fusion cannot accurately adapt to different needs, effectively extract and process features, and is difficult to ensure the accuracy and reliability of the reconstruction results, and achieves the technical effect of improving the accuracy and reliability of the reconstruction results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of fusion of computer graphics and virtual reality, and in particular to a method for fusion of virtual reality and augmented reality based on indoor three-dimensional reconstruction. Background Art

[0002] In the application scenarios of indoor 3D reconstruction and the integration of virtual reality and augmented reality, the development needs of related technologies are becoming increasingly urgent, and the challenges they face are becoming more and more significant. Accurately extracting and processing the features of indoor multi-view images to achieve accurate reconstruction, and flexibly determining the fusion strategy according to different task requirements to provide a more realistic experience have become key issues that need to be urgently solved in this field. Traditional indoor 3D reconstruction and virtual reality and augmented reality fusion technology often has many limitations. For example, it usually relies only on simple image analysis methods or limited algorithm models, and has insufficient grasp of the complex characteristics of the indoor environment. It lacks in-depth mining and comprehensive utilization of multi-view image features, making it difficult to accurately construct a 3D model and achieve high-quality integration of virtual and reality. At the same time, the formulation of the fusion strategy is relatively rigid and single, and cannot adapt well to different application scenarios and task requirements, resulting in poor fusion effect and poor user experience.

[0003] Among the current related technologies, there are technical problems such as being unable to accurately adapt to different task requirements, effectively extract and process features, handle dynamic scenes, and ensure the accuracy and reliability of reconstruction results in the process of indoor three-dimensional reconstruction that integrates virtual reality and augmented reality. Summary of the invention

[0004] The present application provides a virtual reality and augmented reality fusion method based on indoor 3D reconstruction, adopts the method of reading 3D reconstruction tasks and task reference information containing multi-view images of the target (the reconstruction tasks include static and dynamic reconstruction), combines the tasks to make fusion rigid requirement decisions to determine fusion constraint information, identifies the task reference information to extract feature descriptors, constructs a reconstruction processing model based on digitized sample training convergence and containing affine transformation areas and registration fusion areas, combines the model, feature descriptors and fusion constraint information to perform 3D reconstruction of the target and determine the reconstruction target, randomly determines task verification points, verifies and compensates the reconstruction target and visualizes it on the terminal reality interface, realizes the technical effect of flexibly adjusting the fusion strategy according to different task requirements, accurately extracting and processing image features, effectively coping with dynamic scenes, and significantly improving the accuracy and reliability of the reconstruction results.

[0005] This application provides a method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction, including:

[0006] Read the 3D reconstruction task and task benchmark information, wherein the task benchmark information includes a target multi-view image, and the 3D reconstruction task is a static reconstruction and a dynamic reconstruction; in combination with the 3D reconstruction task, make a fusion rigidity requirement decision and determine the fusion constraint information; identify the task benchmark information, perform image feature extraction and multi-view fusion to extract feature descriptors, wherein the feature descriptors represent the description scale space of feature points; construct a reconstruction processing model, wherein the reconstruction processing model is based on digitized sample training to convergence acquisition, and includes an affine transformation area and a registration fusion area; in combination with the reconstruction processing model, perform target 3D reconstruction based on the feature descriptors and the fusion constraint information, and determine the 3D reconstruction target; randomly determine the task verification point of the 3D reconstruction task, verify and compensate the 3D reconstruction target, and visualize it on the terminal reality interface.

[0007] The virtual reality and augmented reality fusion method based on indoor 3D reconstruction proposed in this application first reads the 3D reconstruction task and task benchmark information containing the target multi-view image (the reconstruction task includes static and dynamic reconstruction), makes a fusion rigid requirement decision in combination with the task to determine the fusion constraint information, identifies the task benchmark information to extract the feature descriptor, builds a reconstruction processing model based on the convergence of digitized sample training and containing affine transformation area and registration fusion area, combines the model, feature descriptor and fusion constraint information to perform target 3D reconstruction and determine the reconstruction target, randomly determines the task verification point, verifies and compensates the reconstruction target and visualizes it on the terminal reality interface, thus achieving the technical effect of flexibly adjusting the fusion strategy according to different task requirements, accurately extracting and processing image features, effectively coping with dynamic scenes, and significantly improving the accuracy and reliability of the reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0009] Figure 1 A flowchart of a method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction provided in an embodiment of the present application;

[0010] Figure 2 A flowchart of a method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0012] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0013] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0014] The embodiment of the present application provides a method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction, such as Figure 1 As shown, the method includes:

[0015] Step S100, read the three-dimensional reconstruction task and task benchmark information, wherein the task benchmark information includes the target multi-view image, and the three-dimensional reconstruction task is static reconstruction and dynamic reconstruction. Specifically, the three-dimensional reconstruction task and task benchmark information refer to the task type and related basic data specified at the beginning of the three-dimensional reconstruction work. Determine the target object to be reconstructed, prepare the hardware and software equipment required for the three-dimensional reconstruction, set the reconstruction task parameters, such as reconstruction accuracy, reconstruction range, output format, etc., start the reading process, obtain the static reconstruction or dynamic reconstruction task type record data, the static reconstruction task type record data indicates that the three-dimensional construction is performed for a relatively fixed object, and its characteristic is that the object has high stability in the time dimension. Continue to perform the reading operation, obtain the task benchmark information record data containing the target multi-view image, and the multi-view image task benchmark information record data indicates the image set obtained by shooting the target from multiple different angles and positions, and its characteristic is that it provides rich spatial perspective information. Record all relevant information involved in the entire reading process.

[0016] Step S200, in combination with the three-dimensional reconstruction task, a fusion rigid requirement decision is made to determine the fusion constraint information. Specifically, the fusion rigid requirement decision and fusion constraint information refer to the rules and constraints formulated in the three-dimensional reconstruction process to achieve accurate and effective fusion. The type of three-dimensional reconstruction task to be processed is clarified, the computing resources and algorithm tools required for the fusion decision are prepared, the fusion decision parameters are set, such as reconstruction accuracy standards, data compatibility requirements, algorithm selection range, etc., the fusion decision process is executed, and the initial fusion strategy record data is obtained. The initial fusion strategy record data represents the preliminary determination of the basic fusion direction and method at the beginning of the fusion, and its characteristic is to lay the foundation for fusion. Continue to execute the fusion decision, obtain the refined fusion constraint information record data, and the refined fusion constraint information record data represents the clarification of specific fusion details and constraints at the in-depth stage of the fusion process, and its characteristic is to accurately standardize the fusion process. Record all relevant information in the entire fusion decision process.

[0017] In a possible implementation, in combination with the three-dimensional reconstruction task, a fusion rigidity requirement decision is made to determine the fusion constraint information. Step S200 further includes step S210, determining the fusion type, wherein the fusion type includes a first-class fusion and a second-class fusion. The first-class fusion is the fusion of virtuality into reality with reality as the main body, and the second-class fusion is the fusion of reality into virtuality. Specifically, it is necessary to clarify the fusion type, and there are two categories of fusion: the first-class fusion and the second-class fusion. The first-class fusion is to fuse virtuality into reality with reality as the main body, such as adding virtual objects or information to the real scene; the second-class fusion is to fuse reality into virtuality, such as integrating real elements into the virtual environment, assuming that the weight of the real part is A1, the weight of the virtual part is B1, and A1+B1=1. For example, A1=0.7, B1=0.3, indicating that reality accounts for 70% and virtual accounts for 30%. The second-class fusion is the fusion of reality into virtuality, assuming that the weight of the real part is A2, the weight of the virtual part is B2, and A2+B2=1. For example, A2=0.3, B2=0.7, which means that reality accounts for 30% and virtuality accounts for 70%.

[0018] Step S220, identify the three-dimensional reconstruction task, and determine the target fusion type based on the task requirements and the fusion ratio. Specifically, identify the three-dimensional reconstruction task, and analyze the specific requirements of the task in depth, such as whether to emphasize the display of the real scene or focus more on the construction of the virtual environment. At the same time, consider the fusion ratio, that is, the proportional relationship between the real and virtual elements in the fusion, and use the task requirements and the fusion ratio as a benchmark to determine the final suitable target fusion type. For example, if the task is to display virtual promotional information in a real shopping mall, the fusion ratio may be set to 60% reality and 40% virtual. By comparing different fusion types with the set ratio, if one type of fusion is closer to this ratio, the target fusion type is determined to be a type of fusion.

[0019] Step S230, determine the fusion inter-embedding degree based on the task requirements, and determine the fusion constraint information in combination with the target fusion type. Specifically, the fusion inter-embedding degree is determined according to the specific requirements of the task. The fusion inter-embedding degree refers to the degree of mutual embedding and combination between real elements and virtual elements. If the task requires a highly realistic effect, the fusion inter-embedding degree may be higher; on the contrary, if more emphasis is placed on virtual creative expression, the fusion inter-embedding degree may be relatively low. Combined with the determined target fusion type, the fusion inter-embedding degree and other factors are comprehensively considered to determine the fusion constraint information. The constraint information includes but is not limited to the boundary conditions of the fusion, the rules of data processing, the standards for effect evaluation, etc., to provide clear guidance and restrictions for subsequent specific fusion operations to ensure that the fusion results can meet the task requirements and expected effects. Assuming that the task requires a highly realistic fusion effect, the value range of the fusion inter-embedding degree is set to 0 to 1, 0 represents complete separation, and 1 represents complete fusion. If it is considered after evaluation that a fusion level of 80% is required, that is, the fusion inter-embedding degree is 0.8, the fusion constraint information is determined based on the determined target fusion type and fusion inter-embedding degree. If it is a type fusion and the fusion inter-embedding degree is 0.8, then the constraint information may include that the position accuracy of the virtual elements in the real scene must reach ±5 cm, the color matching error must not exceed 5%, etc.

[0020] Step S300, identify the task benchmark information, perform image feature extraction and multi-view fusion to extract feature descriptors, wherein the feature descriptors represent the description scale space of feature points. Specifically, the acquired task benchmark information is comprehensively identified and analyzed. The task benchmark information contains multi-view images of the target. The images come from different angles, distances and lighting conditions. Image feature extraction is performed, and a variety of image processing algorithms and techniques are used, such as edge detection, corner detection, texture analysis, etc., to extract representative and distinguishing features from each image. In the stage of multi-view fusion to extract feature descriptors, the system will integrate and comprehensively analyze the image features obtained from multiple viewpoints. Images from different viewpoints have different and complementary information. A more comprehensive and accurate feature description can be obtained through fusion. Feature descriptors are used to represent The description scale space of feature points describes the characteristics of feature points at different scales, such as capturing detail information at a smaller scale and obtaining overall structural information at a larger scale. During multi-view fusion, these feature points in the images of the building taken from different angles will be integrated to form a more complete and accurate feature description. The feature descriptor can describe and quantify the importance, position, shape, etc. of these feature points in detail according to different observation scales. The process of extracting feature descriptors through image feature extraction and multi-view fusion provides key basic data and information for subsequent 3D reconstruction, which helps to build 3D models more accurately.

[0021] In a possible implementation, the task benchmark information is identified, image feature extraction and multi-view fusion are performed to extract feature descriptors, wherein the feature descriptors represent the description scale space of feature points, such as Figure 2 As shown, step S300 further includes step S310, identifying the 3D reconstruction task and determining the 3D reconstruction requirements. Specifically, the 3D reconstruction task is identified, and the requirements and expected goals of the task are analyzed, such as the type of object to be reconstructed, accuracy requirements, application scenarios, etc.

[0022] Step S320, based on the three-dimensional reconstruction requirements, with non-eigenvalue suppression as a constraint condition, traverse the target multi-view images to extract image features and determine the feature point distribution set. Specifically, based on the determined three-dimensional reconstruction requirements, with non-eigenvalue suppression as a constraint condition, traverse the target multi-view images to extract image features, determine the resolution of each image, and calculate the grayscale change rate of each pixel point and its neighboring pixel points through an edge detection algorithm. If the change rate is less than a set threshold, it is considered that the point is not a feature point; otherwise, it may be a feature point. Such calculations are performed on all images to ultimately determine the feature point distribution set.

[0023] Step S330, based on the feature point distribution set, screen valid feature points and generate feature descriptors, wherein the feature descriptors are represented as feature point vector tuples. Specifically, based on the obtained feature point distribution set, further screen valid feature points, set a feature point repeatability threshold, that is, if a feature point is detected in all viewing angle images exceeding the feature point repeatability threshold, it is considered to be a valid feature point, screen the initially determined feature points to obtain valid feature points, generate feature descriptors for the valid feature points, and represent them as feature point vector tuples, assuming that each feature point is described by a 9-tuple containing position, direction, color and texture features, and generate a corresponding 9-tuple feature descriptor for each valid feature point.

[0024] In a possible implementation, based on the feature point distribution set, valid feature points are screened and feature descriptors are generated, wherein the feature descriptors are represented as a multi-tuple of feature point vectors, and step S330 further includes step S331, traversing the feature point distribution set, identifying repeatedly distributed feature points based on the description scale space as a reference, and screening based on the maximum response value to determine valid feature points. Specifically, based on the description scale space as a reference, these feature points are compared one by one, and the description scale space is divided into three scales: small, medium, and large. During the comparison process, repeatedly distributed feature points are identified, and for these repeatedly distributed feature points, their response values ​​are calculated. The response values ​​are calculated based on factors such as the grayscale change degree of pixels around the feature points. For repeatedly distributed feature points, the ones with the largest response value are selected based on the corresponding response value size and determined as valid feature points.

[0025] Step S332, based on the effective feature points, determine the response descriptor of the description scale space mapped by the feature points, perform distribution integration to generate the feature descriptor, wherein each response descriptor represents the description scale space covered by the mapped feature points. Specifically, all effective feature points are determined from the repeatedly distributed feature points through screening, and the response descriptor of each effective feature point mapped to the description scale space is determined based on the determined effective feature points. For each effective feature point, its response value in different description scale spaces is calculated to form a response descriptor, and the response value of each feature point in the small-scale space, the medium-scale space, and the large-scale space is determined. The response descriptor is distributed and integrated, and the weights of the small, medium, and large-scale spaces are set according to the needs. The integrated response descriptor is calculated by weighted average. By performing such integration on the response descriptors of all effective feature points, a complete feature descriptor is finally generated, and the generation from the feature point distribution set to the feature descriptor is completed, providing effective data support for subsequent work.

[0026] In a possible implementation, the feature point distribution set is traversed, and the description scale space is used as a reference to identify repeated distribution feature points and screen based on the maximum response value to determine valid feature points. Step S331 further includes step S3311, if the repeated distribution feature points have a spatially dislocated distribution, overlapping localization processing is performed, wherein the overlapping localization method includes response value localization and reorganization localization. Specifically, when it is found that the repeated distribution feature points have a spatially dislocated distribution, overlapping localization processing will be performed, and the processing methods include response value localization and reorganization localization.

[0027] Step S3312, wherein the response value localization and the reorganization localization include: reading the response value of the repeated distribution feature point, localizing the overlapping area with the high response value, and determining the first processing result. Specifically, for the response value localization, the response value of the existing repeated distribution feature point is read, and the magnitude of the response value reflects the prominence and importance of the feature point. The feature point with the highest response value is found, indicating that it is more prominent and important in the current description scale space, and the overlapping area is localized according to the high response value, that is, it is determined that the overlapping area belongs to the feature point, and the first processing result is obtained.

[0028] Step S3314, extract and read the description scale space of the repeated distribution feature points, decompose based on the dislocation boundary, reconstruct the decomposed space features, and determine the second processing result. Specifically, for the reorganization localization, first, the description scale space of these repeated distribution feature points must be fully extracted and read, assuming that the respective description scale spaces are decomposed based on the dislocation boundary, and the dislocation boundary divides their description scale space into two parts, which are named separately, and the decomposed space features are reconstructed for the decomposed space. For the divided part of the description scale space, the feature information originally belonging to the feature points in this area will be collected, recombined and sorted to form a new feature representation, and for the other part, the same operation is performed, and the feature points originally dislocated in the decomposed space are reconstructed and combined, and the second processing result is determined, so as to achieve a more accurate processing of the spatial dislocation problem of the repeated distribution feature points, and provide more accurate, clear and useful information for subsequent related operations and analysis, thereby improving the processing efficiency and accuracy of the entire system.

[0029] In a possible implementation, based on the effective feature points, a response descriptor of the description scale space of the feature point mapping is determined, and the feature descriptor is generated by distribution integration, wherein each response descriptor represents the description scale space covered by the mapped feature points, and step S332 further includes step S3321, if the three-dimensional reconstruction task is a dynamic reconstruction, the dynamic evolution characteristics based on the effective feature points are determined. Specifically, if the three-dimensional reconstruction task is determined to be a dynamic reconstruction, the dynamic evolution characteristics based on the effective feature points are determined. In a series of continuous images, the position changes of the effective feature points can be represented by coordinates, and the coordinate positions of the effective feature points at different times are determined. By calculating the distance between the two coordinates and the time interval, the speed of the feature point is obtained by dividing the distance by the time, and the dynamic evolution characteristics such as the speed and direction of its movement are analyzed.

[0030] Step S3322, based on the preset trend change degree and in combination with the dynamic evolution characteristics, the three-dimensional reconstruction target is dynamically converted to a frame rate, and multiple frame rate targets are determined, and the preset trend change degree is determined based on the dynamic flow field degree. Specifically, based on the preset trend change degree and in combination with the above-mentioned dynamic evolution characteristics, the three-dimensional reconstruction target is dynamically converted to a frame rate, and the preset trend change degree is determined based on the dynamic flow field degree, and the dynamic flow field degree is determined by calculating the average speed and speed change rate of multiple feature points. The preset trend change degree can be set as a functional relationship with the average speed and speed change rate. According to the preset trend change degree and the dynamic evolution characteristics, the frame rate of the three-dimensional reconstruction target is converted. If the preset trend change degree is large, it means that the object moves faster or changes dramatically, and the frame rate is increased. On the contrary, if the preset trend change degree is small, the frame rate is reduced. Through conversion, multiple frame rate targets are determined.

[0031] Step S3323, positively serialize the multiple frame rate targets to determine the dynamic reconstruction target. Specifically, the multiple frame rate targets are positively serialized in chronological order, and the targets with different frame rates are arranged in chronological order to form an orderly sequence, so as to finally determine the dynamic reconstruction target, and realize the analysis of effective feature points to the determination of dynamic reconstruction targets, which provides an effective solution for accurately presenting dynamic scenes.

[0032] Step S400, constructing a reconstruction processing model, wherein the reconstruction processing model is acquired based on digitized sample training until convergence, and includes an affine transformation area and a registration fusion area. Specifically, a reconstruction processing model is constructed, and a large number of digital samples are collected. The samples should cover various possible situations and scenarios to ensure that the model has wide applicability and accuracy. The model is trained using digital samples. During the training process, the model will continuously adjust its internal parameters to try to accurately predict or process the input sample data. The affine transformation area in the model is mainly responsible for geometric transformation of the input data, such as translation, rotation, scaling and other operations, which helps to adjust data from different sources or postures to a relatively unified state to prepare for subsequent processing. The registration and fusion area is responsible for accurately matching and fusing different data after affine transformation. By comparing the characteristics and structure of the data, the best fusion method is found to make the final reconstruction result more accurate and complete. The training will continue until the performance indicators of the model reach a stable optimal state, that is, convergence. The model can effectively process and reconstruct the newly input data, output results that meet the requirements, make full use of the information in the digital samples, and combine the functions of the affine transformation area and the registration and fusion area to achieve high-quality three-dimensional reconstruction processing.

[0033] Step S500 , combining the reconstruction processing model, based on the feature descriptor and the fusion constraint information, performing three-dimensional reconstruction of the target, and determining the three-dimensional reconstruction target. Specifically, the reconstruction processing model that has been constructed is obtained. The model has been trained based on digital samples and has converged, and has the ability to process and reconstruct three-dimensional targets. The feature descriptors and fusion constraint information extracted and determined before are obtained. The feature descriptors provide detailed feature information about the target object, such as position, direction, scale, etc. The fusion constraint information stipulates various restrictions and rules in the reconstruction process, such as accuracy requirements, data weight distribution, etc. The feature descriptors and fusion constraint information are input into the reconstruction processing model. The affine transformation area in the model will perform geometric transformation on the data according to the input feature descriptors, so that data from different perspectives and sources can achieve a relatively unified state in space. The registration fusion area will accurately match and fuse the affine transformed data based on the fusion constraint information, compare the features between different data, integrate them according to the constraint rules, and gradually construct the preliminary form of the three-dimensional model. During the reconstruction process, the model will continuously adjust and optimize according to the input information to ensure that the reconstruction result meets the requirements of the fusion constraint information, and finally determine the three-dimensional reconstruction target, that is, a complete, accurate and expected three-dimensional model. The ability of the reconstruction processing model and the guidance of the feature descriptors and fusion constraint information are utilized to achieve accurate target three-dimensional reconstruction.

[0034] In a possible implementation, in combination with the reconstruction processing model, based on the feature descriptor and the fusion constraint information, target 3D reconstruction is performed, and the 3D reconstruction target is determined. Step S500 further includes step S510, traversing the feature descriptor, decomposing the task sub-targets and performing 3D reconstruction to determine the 3D component. Specifically, the acquired feature descriptors are traversed. In this process, the entire 3D reconstruction task is carefully decomposed and divided into multiple specific sub-targets. If a complex mechanical part is to be reconstructed, the sub-targets may include various components of the part, such as the main structure, connecting parts, holes, etc. For each decomposed sub-target, the corresponding algorithm and technology are used to perform independent 3D reconstruction work. By interpreting and processing the position, shape, texture and other information contained in the feature descriptor, a preliminary 3D component is constructed. For example, for the main structure of the part, its approximate geometric shape and size are determined according to the feature descriptor, thereby generating the corresponding 3D component.

[0035] Step S520, based on the 3D reconstruction task, determine personalized reconstruction features. Specifically, based on the initially set 3D reconstruction task, in-depth analysis and determination of personalized reconstruction features involve specific accuracy requirements, unique appearance design, specific material properties, etc. For example, if the reconstruction task is to show the performance of mechanical parts in a specific working environment, then the personalized reconstruction features may include enhanced representation of the stress-bearing parts of the parts, or simulation of specific surface treatment effects.

[0036] Step S530, based on the personalized reconstruction feature, the three-dimensional component is subjected to affine transformation and registration analysis, and three-dimensional reconstruction is performed to determine the three-dimensional reconstruction target. Specifically, according to the determined personalized reconstruction feature, the previously constructed three-dimensional component is subjected to affine transformation and registration analysis. Affine transformation is used to adjust the position, direction, size and shape of the three-dimensional component to meet the personalized requirements. Registration analysis is to accurately calculate and adjust the relative position and relationship between multiple three-dimensional components to ensure that they can be accurately combined together. For example, if the personalized feature requires a certain tilt angle of a part of the part, it is achieved through affine transformation; if the connection between two components is required to be tight and seamless, it is achieved through registration analysis. The three-dimensional reconstruction operation is performed again, and the three-dimensional components after affine transformation and registration analysis are integrated and optimized. Combined with the relationship between personalized features and components, the final three-dimensional reconstruction target that meets the requirements is generated. Continuous adjustment and improvement ensure that the reconstructed three-dimensional model fully demonstrates personalized characteristics and requirements while meeting the basic structure. From the traversal decomposition task of the feature descriptor to the determination of personalized features and corresponding processing and reconstruction, the three-dimensional reconstruction target that meets the expectations is finally successfully determined.

[0037] Step S600, randomly determine the task checkpoints of the three-dimensional reconstruction task, check and compensate the three-dimensional reconstruction target, and visualize it on the terminal reality interface. Specifically, some key positions or stages are randomly selected as task checkpoints in the three-dimensional reconstruction task. There is no fixed rule for the selection of checkpoints, but they are randomly distributed in the entire reconstruction process to ensure that the reconstruction target can be fully checked. For the selected checkpoints, the current three-dimensional reconstruction target is checked, including comparing the reconstruction result with the expected standard or original data, and checking whether there are deviations or errors in shape, size, position, texture, etc. If problems or deviations are found during the verification process, compensation operations need to be performed. Compensation can be adjustment of model parameters, supplementation or correction of missing or erroneous data, or recalculation of certain parts to improve the reconstruction results. After completing the verification and compensation, the final three-dimensional reconstruction target is visualized on the terminal display interface. Through the graphics rendering technology, the three-dimensional model is displayed in an intuitive way, so that the user can clearly see the reconstruction effect. Through the random determination of checkpoints, verification and compensation, and visualization of the solution process, the quality and accuracy of the three-dimensional reconstruction can be continuously improved, and the final results can be displayed to the user in an intuitive way.

[0038] In a possible implementation, the task checkpoints of the three-dimensional reconstruction task are randomly determined, the three-dimensional reconstruction target is checked and compensated, and visualized on the terminal reality interface. Step S600 further includes step S610, based on the three-dimensional reconstruction task, a checkpoint set is determined. Specifically, according to the given three-dimensional reconstruction building task, an analysis is performed, considering the overall structure, appearance style, room layout, door and window positions, and functional zoning of the building, and according to the characteristics and requirements of the task, a series of key positions, stages or data points that may be used for verification are determined to form a checkpoint set. For example, the four corner points of the building, the staircases of each floor, the center point of the main room, the key support points of the roof, etc. can be used as checkpoints, and the checkpoints are preliminarily determined to form a checkpoint set.

[0039] Step S620, based on a preset number, traverse the checkpoint set for random matching to determine the task checkpoint. Specifically, a preset number is set, the checkpoint set is traversed for random matching, and a random number generator is used to implement random selection. Assuming that the range generated by the random number generator is 1 to 200, a number is randomly generated for the first time, and this checkpoint is selected as the first task checkpoint. A number is randomly generated for the second time, and the checkpoint is selected as the second task checkpoint, and so on, until all different checkpoints are selected as the final task checkpoint. Through the random selection process, the selected checkpoints are guaranteed to be random and representative to a certain extent, so as to verify and evaluate the three-dimensional reconstruction building task more comprehensively and effectively.

[0040] The embodiment of the present application adopts the method of reading the 3D reconstruction task and task reference information including multi-view images of the target (the reconstruction task includes static and dynamic reconstruction), combining the task to make a fusion rigid requirement decision to determine the fusion constraint information, identifying the task reference information to extract feature descriptors, constructing a reconstruction processing model based on the convergence of digitized sample training and containing affine transformation areas and registration fusion areas, combining the model, feature descriptors and fusion constraint information to perform 3D reconstruction of the target and determine the reconstruction target, randomly determining the task verification points, verifying and compensating the reconstruction target and visualizing it on the terminal reality interface, achieving the technical effect of flexibly adjusting the fusion strategy according to different task requirements, accurately extracting and processing image features, effectively coping with dynamic scenes, and significantly improving the accuracy and reliability of the reconstruction results.

[0041] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction, characterized in that: The method comprises: Reading a three-dimensional reconstruction task and task benchmark information, wherein the task benchmark information includes a target multi-view image, and the three-dimensional reconstruction task is static reconstruction and dynamic reconstruction; In combination with the three-dimensional reconstruction task, a fusion rigidity requirement decision is made and fusion constraint information is determined. The fusion rigidity requirement decision and fusion constraint information refer to rules and restrictions formulated in the three-dimensional reconstruction process to achieve accurate and effective fusion; Identify the task benchmark information, perform image feature extraction and multi-view fusion to extract feature descriptors, wherein the feature descriptors represent the description scale space of feature points; Constructing a reconstruction processing model, wherein the reconstruction processing model is acquired based on digitized sample training until convergence, and comprises an affine transformation area and a registration fusion area; In combination with the reconstruction processing model, based on the feature descriptor and the fusion constraint information, a target three-dimensional reconstruction is performed to determine a three-dimensional reconstruction target; Randomly determine a task check point of the three-dimensional reconstruction task, check and compensate the three-dimensional reconstruction target, and visualize it on a terminal reality interface; The determining of fusion constraint information includes: Determine a fusion type, wherein the fusion type includes a first-class fusion and a second-class fusion, wherein the first-class fusion is the fusion of the virtual with the reality with the reality as the main body, and the second-class fusion is the fusion of the reality with the virtual; Identify the three-dimensional reconstruction task, and determine the target fusion type based on the task requirements and the fusion ratio; Determine the fusion inter-embedding degree based on the task requirements, and determine the fusion constraint information in combination with the target fusion type, wherein the fusion inter-embedding degree refers to the degree of mutual embedding and combination between real elements and virtual elements.

2. The method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction according to claim 1, characterized in that: The extracting feature descriptor comprises: Identifying the three-dimensional reconstruction task and determining the three-dimensional reconstruction requirement; Based on the three-dimensional reconstruction requirements and with non-eigenvalue suppression as a constraint condition, traverse the target multi-view images to extract image features and determine a feature point distribution set; Based on the feature point distribution set, valid feature points are screened and feature descriptors are generated, wherein the feature descriptors are represented as a multi-tuple of feature point vectors.

3. The method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction according to claim 2, characterized in that: The screening of effective feature points and generating feature descriptors includes: Traversing the feature point distribution set, taking the description scale space as a reference, identifying repeatedly distributed feature points and screening them based on the maximum response value to determine valid feature points; Based on the effective feature points, a response descriptor of the description scale space mapped by the feature points is determined, and distribution integration is performed to generate the feature descriptors, wherein each response descriptor represents the description scale space covered by the mapped feature points.

4. The method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction according to claim 3, characterized in that: After identifying repeatedly distributed feature points, including: If the repeated distribution feature points are spatially dislocated, overlapping localization is performed, wherein the overlapping localization method includes response value localization and recombinant localization; Among them, response value localization and recombinant localization, including Read the response values ​​of the repeatedly distributed feature points, and use the high response values ​​to localize the overlapping areas, and determine the first processing result; The description scale space of the repeatedly distributed feature points is extracted and read, and decomposed based on the dislocation boundary, and the decomposed space feature is reconstructed to determine the second processing result.

5. The method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction according to claim 1, characterized in that: The performing of target three-dimensional reconstruction and determining the three-dimensional reconstruction target comprises: Traversing the feature descriptors, decomposing the task sub-goals and performing three-dimensional reconstruction to determine the three-dimensional components; Based on the three-dimensional reconstruction task, determining personalized reconstruction features; Based on the personalized reconstruction features, affine transformation and registration analysis are performed on the three-dimensional component, and three-dimensional reconstruction is performed to determine the three-dimensional reconstruction target.

6. The method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction according to claim 3, characterized in that: The method further comprises: If the three-dimensional reconstruction task is dynamic reconstruction, determining dynamic evolution features based on the effective feature points; Based on a preset trend change degree and in combination with the dynamic evolution characteristics, the three-dimensional reconstruction target is dynamically converted to a frame rate to determine a multi-frame rate target, wherein the preset trend change degree is determined based on the dynamic flow field degree; The multiple frame rate targets are being sequenced and a dynamic reconstruction target is determined.

7. The method for integrating virtual reality and augmented reality based on indoor three-dimensional reconstruction according to claim 1, characterized in that: Randomly determining a task check point of the three-dimensional reconstruction task, including: Based on the three-dimensional reconstruction task, determining a check point set; Based on a preset number, the checkpoint set is traversed to perform random matching to determine the task checkpoint.