Three-dimensional model construction method, device and electronic equipment for a scene

Abstract

The application discloses a three-dimensional model construction method and device of a scene and electronic equipment. The method comprises the following steps: video data of a target scene is collected by using video collection devices arranged at multiple angles, and multiple video data are obtained; multiple feature data in the multiple video data are acquired, the multiple video data are fused based on the correlation between the multiple feature data, and a fusion result is obtained; depth information of the target scene is extracted from the fusion result, and an initial three-dimensional model of the target scene is constructed based on the depth information, wherein the depth information is used for representing the distance between the target scene and the video collection device; an evaluation result of the initial three-dimensional model is determined; and the initial three-dimensional model satisfying the evaluation result threshold is determined as a target three-dimensional model of the target scene. The application solves the technical problem of low three-dimensional model construction accuracy of a scene.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and more specifically, to a method, apparatus, and electronic device for constructing a three-dimensional model of a scene. Background Technology

[0002] Currently, three-dimensional (3D) reconstruction technology has significant application value and importance in many fields such as intelligent monitoring, virtual reality, three-dimensional measurement, robot navigation, disaster assessment, and urban planning.

[0003] In related technologies, 3D reconstruction based on multi-view video data primarily relies on deep learning. However, the aforementioned 3D model construction depends on manually designed feature points, making the process time-consuming, and the quality of the 3D model depends on manual operation, making it difficult to meet the needs of real-time applications. Furthermore, the 3D model construction process relies on manually designed parameters, which are time-consuming to adjust, and the model quality also depends on manual operation, making it difficult to meet the needs of real-time applications. The 3D model construction process is highly dependent on the performance of deep learning networks, and improving the performance of deep learning networks depends on the accumulation of a large number of data labels. However, current data labels are time-consuming and labor-intensive, and their quality is low, resulting in a time-consuming and ineffective 3D reconstruction process. Therefore, the technical problem of low accuracy in 3D model construction for specific scenarios still exists.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, and electronic device for constructing a 3D model of a scene, in order to at least solve the technical problem of low accuracy in constructing a 3D model of a scene.

[0006] According to one aspect of the present invention, a method for constructing a three-dimensional model of a scene is provided, comprising: acquiring video data of a target scene using video acquisition devices deployed at multiple viewpoints to obtain multiple video data; acquiring multiple feature data from the multiple video data, and fusing the multiple video data based on the correlation between the multiple feature data to obtain a fusion result; extracting depth information of the target scene from the fusion result, and constructing an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to represent the distance between the target scene and the video acquisition devices; determining the evaluation result of the initial three-dimensional model; and determining the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene.

[0007] Optionally, multiple feature data from multiple video data can be obtained, including: using an attention mechanism to identify multiple feature data from multiple video data respectively.

[0008] Optionally, an attention mechanism is used to identify multiple feature data in multiple video data, including: using a feature extraction model to extract features from video frames in multiple video data to obtain multiple feature maps; and using an attention mechanism to identify multiple feature data from the multiple feature maps.

[0009] Optionally, the feature extraction model includes a Gaussian model and a convolutional model, and the feature map includes a first feature map and a second feature map. The Gaussian model corresponds to the first weight and the first feature map, and the convolutional model corresponds to the second weight and the second feature map. Specifically, the feature extraction model is used to extract features from video frames in multiple video data to obtain multiple feature maps, including: inputting the video frame into the Gaussian model, using the Gaussian model to extract features from the video frame to obtain the first feature map, inputting the video frame into the convolutional model, using the convolutional model to extract features from the video frame to obtain the second feature map; assigning a first weight to the first feature map and assigning a second weight to the second feature map, and then weighted summing to obtain the feature map.

[0010] Optionally, the method further includes: determining the feature similarity between multiple feature data based on multiple feature data from different perspectives; determining the positional relationship data between multiple feature data based on the coordinate positions corresponding to the multiple feature data from different perspectives; determining the matching weight of multiple feature data based on the feature similarity and positional relationship data, wherein the matching weight is used to represent the degree of association between multiple feature data; in response to the matching weight exceeding the matching weight threshold, selecting feature data pairs from the multiple feature data, wherein the feature data pair includes two feature data; and constructing the association relationship corresponding to the feature data pair.

[0011] Optionally, constructing the association relationship between feature data pairs includes: obtaining the feature vectors corresponding to the two feature data in the feature data pair, and determining the inner product similarity and the distance similarity between the two feature vectors; obtaining the coordinate positions of the two feature data in the feature data pair from their respective perspectives, and determining the positional relationship weight between the two coordinate positions; and constructing the association relationship based on the inner product similarity, distance similarity, and positional relationship weight.

[0012] Optionally, based on the correlation between multiple feature data, multiple video data are fused to obtain a fusion result, including: determining the fusion weights corresponding to each of the multiple video data based on the correlation; and fusing the multiple video data based on the fusion weights to obtain a fusion result.

[0013] Optionally, based on the correlation, the fusion weights corresponding to multiple video data are determined, including: obtaining the time information of the acquired video data, and setting a spatiotemporal consistency factor based on the time information, wherein the spatiotemporal consistency factor includes the time information and the coordinate position of the feature data; and determining the fusion weights based on the spatiotemporal consistency factor and the correlation.

[0014] Optionally, based on depth information, an initial 3D model of the target scene is constructed, including: converting the depth information into 3D coordinates, and constructing the initial 3D model based on the 3D coordinates.

[0015] Optionally, determining an initial 3D model whose evaluation result meets the evaluation result threshold as the target 3D model of the target scene includes: determining the initial 3D model as the target 3D model in response to the evaluation result exceeding the evaluation result threshold; adjusting the initial 3D model in response to the evaluation result not exceeding the evaluation result threshold, and determining the adjusted initial 3D model as the target 3D model, wherein the evaluation result of the adjusted initial 3D model exceeds the evaluation result threshold.

[0016] According to another aspect of the present invention, a three-dimensional model construction apparatus for a scene is also provided, comprising: an acquisition unit, configured to acquire video data of a target scene using video acquisition devices deployed at multiple viewpoints, to obtain multiple video data; a fusion unit, configured to acquire multiple feature data from the multiple video data, and fuse the multiple video data based on the correlation between the multiple feature data, to obtain a fusion result; a construction unit, configured to extract depth information of the target scene from the fusion result, and construct an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to represent the distance between the target scene and the video acquisition devices; a first determination unit, configured to determine the evaluation result of the initial three-dimensional model; and a second determination unit, configured to determine the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene.

[0017] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a plurality of instructions adapted for a processor to load and execute any one of the above-described methods for constructing a three-dimensional model of a scene.

[0018] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform a three-dimensional model construction method for any of the above-described scenarios.

[0019] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements a method for constructing a three-dimensional model of any of the above-described scenarios.

[0020] In this embodiment of the invention, if it is necessary to construct a 3D model corresponding to a target scene, video acquisition devices deployed at multiple perspectives of the target scene can be used to acquire video data of the target scene, obtaining multiple video data corresponding to multiple perspectives. The video data can be analyzed to obtain corresponding feature data and the correlation between feature data. The video data is then fused using these correlations to obtain a fusion result. Depth information between the target scene and the video acquisition devices is extracted from the fusion result, and an initial 3D model is constructed using this depth information. The constructed initial 3D model is then evaluated to obtain an evaluation result. The initial 3D model whose evaluation result meets the evaluation result threshold is determined as the target 3D model of the target scene. In this embodiment, the multi-view video data fusion technology can dynamically identify the correlation between feature datasets of multiple perspectives. Therefore, under specific conditions (e.g., a perspective is occluded or there is insufficient lighting), the proportion of video data from that perspective in the fusion result can be automatically reduced, achieving the goal of improving the reliability and stability of the fusion result. This, in turn, improves the accuracy of the 3D model construction of the scene, solving the technical problem of low accuracy in 3D model construction of the scene. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1 This is a flowchart of a method for constructing a three-dimensional model of a scene according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of a three-dimensional scene reconstruction and ranging system based on multi-video fusion according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of a three-dimensional model construction device for a scene according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of an electronic device for a method of constructing a three-dimensional model of a scene according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1

[0029] According to an embodiment of the present invention, a three-dimensional model construction embodiment of a scene is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] Figure 1 This is a flowchart of a method for constructing a 3D model of a scene according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0031] Step S102: Video data of the target scene is collected using video acquisition devices deployed at multiple viewpoints to obtain multiple video data.

[0032] In the technical solution provided by step S102 of the present invention, the target scene can be the scene to be constructed as a three-dimensional model (three-dimensional scene reconstruction), or it can be called a three-dimensional scene. It can be determined according to the field and function of the application. For example, for autonomous driving and robot navigation, the target scene can be a road or indoor space; for virtual reality and augmented reality, the target scene can be a historical site, museum, natural landscape, etc.; for architecture and real estate, the target scene can be an architectural model; for cultural relic protection and restoration, the target scene can be cultural relics, architectural ruins, etc.

[0033] It should be noted that the above target scenarios are merely illustrative examples and are not subject to specific limitations. Any three-dimensional scene that can be constructed using the three-dimensional model construction method in the embodiments of the present invention is within the protection scope of the embodiments of the present invention.

[0034] Optionally, video acquisition devices can be deployed around the target scene from corresponding perspectives to collect video data of the target scene. These devices can be cameras, such as high-definition cameras, infrared cameras, 3D cameras, etc. This is merely an example and does not impose specific limitations on the video acquisition devices; the appropriate device can be selected based on the adaptability of the target scene to be reconstructed in 3D. The video data can be the video stream captured by the video acquisition device, also known as multi-view video, such as high-definition video of the target scene.

[0035] In this embodiment, if 3D reconstruction of the target scene is required, video acquisition devices can be deployed from multiple perspectives that can capture a panoramic view of the target scene. The video acquisition devices can be controlled to acquire video data of the target scene from the corresponding perspectives.

[0036] Optionally, appropriate video acquisition equipment should be selected based on the characteristics of the target scene and reconstruction requirements. The installation location and angle of the video acquisition equipment should be determined to ensure coverage of all important areas of the target scene and minimize blind spots and obstructions. Viewpoint planning can consider the geometric characteristics, lighting conditions, and dynamic elements of the target scene.

[0037] Alternatively, when using video capture devices from multiple perspectives to capture the target scene, the temporal consistency of the video data captured by each device can be considered. Therefore, to ensure temporal consistency of video data across multiple perspectives, hardware synchronization signals or software algorithms can be used to ensure that video capture devices targeting the same target scene can start and stop recording simultaneously, reducing errors caused by time differences.

[0038] Optionally, the resolution, frame rate, exposure, white balance, and other parameters of the image acquisition device can be configured according to lighting conditions and environmental changes to adapt to the diversity of target scenes and ensure the quality of video data.

[0039] Optionally, the video acquisition device captures a video stream of the target scene in real time, ensuring the detail and clarity of the reconstructed results by capturing high-definition video. The acquired video data needs to be transmitted to the processing unit in real time, which may be done via wired or wireless network. The data transmission process should ensure the integrity of the video stream, avoiding delays and packet loss to maintain data continuity and synchronization.

[0040] In this embodiment of the invention, acquiring video data from multiple perspectives ensures comprehensive capture of various details in the target scene, as well as complementarity between different details in the video data from multiple perspectives. Video data from different perspectives can provide multi-angle details of the target scene, which is crucial for the accuracy of 3D reconstruction. Real-time acquired video data forms a continuous video stream, facilitating the capture of dynamic changes in the target scene over time, which is particularly important for handling dynamic scenes such as moving objects and changes in lighting.

[0041] Step S104: Obtain multiple feature data from multiple video data sets, and fuse the multiple video data sets based on the correlation between the multiple feature data sets to obtain the fusion result.

[0042] In the technical solution provided in step S104 of this embodiment of the invention, feature data can be used to represent key local feature regions in video data, also known as key feature points (feature points / key points). These can be regions with complex textures or patterns in the target scene, such as carpet patterns or wall decorations; physical edges and contours in the target scene, such as vehicle outlines or tree branches; or prominent landmarks in the target scene, such as billboards or road signs. It should be noted that the above feature data are merely illustrative examples and are not specifically limited; they can be determined based on the actual target scene to be reconstructed in 3D. The correlation can be the correspondence between key feature points, representing the association between multiple feature data in terms of time, space, texture, or structure. For example, spatial correlation can represent the positional relationship of the same feature point in 3D space from different perspectives, while temporal correlation can represent the positional change of the same feature point at different time points (continuous video frames). Texture correlation can represent the color or texture similarity of local regions based on feature points. Structural correlation can represent the relative position and shape of feature points in the scene. This is merely an example and does not impose any specific limitations. The fusion result can be used to represent a video obtained by fusing video data from multiple perspectives. For example, it can be used to fuse multi-view videos to generate a high-quality fused video stream.

[0043] In this embodiment, after acquiring video data of the target scene using video acquisition devices deployed at multiple viewpoints, multiple feature data can be obtained from the multiple video data. The correlation between these multiple feature data is then analyzed, and the multiple video data corresponding to the multiple viewpoints are fused based on these correlations to obtain a fusion result.

[0044] Optionally, key feature points can be extracted from video data from various perspectives. Feature matching can be performed to establish the correspondence between feature data from different perspectives.

[0045] Optionally, during feature matching, the similarity between feature data from different perspectives can be used to determine whether a correlation exists between the feature data. For example, if the similarity is high, a correlation exists. Using the above method, the correlation between key feature points can be established.

[0046] Optionally, video data from multiple perspectives can be fused based on the established relationships between key feature points. For example, weights can be assigned to video data from different perspectives according to the relationships, thus enabling video data fusion through a weighted method. Feature points with larger weights have a greater impact on the fusion result. It should be noted that the above process and method for fusing video data from different perspectives are merely illustrative examples and are not intended to impose specific limitations.

[0047] Alternatively, in order to maintain the spatiotemporal consistency between video frames in the video data, the correlation between consecutive video frames can also be considered to ensure the temporal continuity and spatial stability of the fusion result.

[0048] In this embodiment of the invention, the above method generates a fusion result that integrates feature data from multiple perspectives. This fusion result not only includes a comprehensive view of the target scene but also effectively reduces errors and information gaps that may arise from a single perspective by dynamically identifying and matching key feature points. The fusion result will then be used for subsequent 3D reconstruction, providing more accurate and complete visual information. This method is the core of real-time 3D scene understanding, enabling accurate identification and tracking of key feature points in a dynamic environment, which is crucial for many advanced vision tasks (such as autonomous driving, robot navigation, and augmented reality).

[0049] Step S106: Extract depth information of the target scene from the fusion result, and construct an initial 3D model of the target scene based on the depth information.

[0050] In the technical solution provided by step S106 of the present invention, depth information can be used to represent the distance between the target scene and the video acquisition device, and can be the distance between the target scene and the video acquisition device, or it can be called a depth value. The initial three-dimensional model can be a three-dimensional scene model, or it can be called a reconstruction result (three-dimensional reconstruction result).

[0051] In this embodiment, based on the correlation between feature data, multiple video data from multiple perspectives are fused to obtain the fusion result. Then, the depth information of the target scene can be extracted from the fusion result, and an initial three-dimensional model of the target scene can be constructed based on the depth information.

[0052] Optionally, depth information can be extracted from the fusion results of multi-view video streams, and an initial 3D model of the target scene can be constructed based on the depth information.

[0053] Optionally, depth information extraction can be based on depth estimation methods. In the fused video stream (i.e., the fused result), a depth estimation network (e.g., a deep learning model) is used to predict a depth map from each video frame of the fused result. The depth map is a two-dimensional image where the value of each pixel represents the depth of a feature point in the corresponding target scene, i.e., the distance between that feature point and the video capture device.

[0054] Optionally, after extracting depth information, multi-view geometry principles can be applied to construct a 3D model. By utilizing depth maps from different perspectives and parameters from the video acquisition device, the 2D pixel coordinates in the depth map can be converted into 3D spatial coordinates, thereby constructing an initial 3D model.

[0055] In this embodiment of the invention, depth information can be extracted from the fused video stream to construct an initial 3D model of the target scene. This provides a preliminary framework for the 3D structure of the target scene, enabling subsequent model optimization using evaluation results. Although the initial 3D model is preliminary, it already contains the spatial layout and depth information of the target scene, serving as the foundation for advanced functions such as 3D scene understanding, object localization, and scene ranging.

[0056] Step S108: Determine the evaluation results of the initial 3D model.

[0057] In the technical solution provided by step S108 of the present invention, the evaluation result can be used to indicate whether the initial three-dimensional model meets the requirements (preset standard / evaluation standard), or to indicate the satisfaction of relevant personnel with the initial three-dimensional model, and to determine whether the initial three-dimensional model needs to be adjusted.

[0058] In this embodiment, after constructing an initial 3D model of the target scene based on depth information, the evaluation result of the initial 3D model can be determined.

[0059] Optionally, in this embodiment of the invention, the above process is a stage of evaluating the initial three-dimensional model and optimizing the initial three-dimensional model using the evaluation results. The purpose is to ensure that the reconstructed three-dimensional model meets the preset performance standards and application requirements, that is, to ensure that the final three-dimensional model meets the requirements.

[0060] Optionally, before evaluating the initial 3D model, an evaluation standard can be established. This standard can be set based on application requirements and may include, but is not limited to, reconstruction accuracy, model detail, computational efficiency, and the integrity and consistency of the 3D model. For example, in autonomous driving applications, the accuracy of the 3D model (especially the precise location of roads and obstacles) is crucial; while in virtual reality or augmented reality applications, the detail and texture quality of the 3D model may be the key focus.

[0061] It should be noted that the above evaluation criteria are merely illustrative examples and are not intended to impose any specific limitations. As long as the content and process of developing evaluation criteria can be tailored to the accuracy requirements of the 3D model for different target scenarios, they are all within the scope of protection of this invention.

[0062] Optionally, the quality of the constructed initial 3D model can be evaluated based on the evaluation criteria established by the above method. The evaluation can involve multiple aspects, including the geometrical matching degree between the 3D model and the actual target scene, the smoothness and integrity of the model surface, the level of detail of the 3D model, and the accuracy of texture mapping. The evaluation process can be automated, using pre-set algorithms and indicators; or it can be semi-automatic, combining visual inspection and correction suggestions from relevant personnel.

[0063] It should be noted that the above-described process and method for evaluating the initial 3D model using evaluation criteria are merely illustrative examples and are not subject to specific limitations. Any method that can evaluate the initial 3D model and ensure accurate 3D reconstruction of the target scene is within the protection scope of this invention.

[0064] Step S110: The initial 3D model whose evaluation result meets the evaluation result threshold is determined as the target 3D model of the target scene.

[0065] In the technical solution provided by step S110 of the present invention, the evaluation result threshold can be a satisfaction threshold, which can be determined according to the needs of the target scene to be reconstructed in the actual situation, or it can be preset, for example, it can be preset to 98%. It should be noted that the setting method and value of the above evaluation result threshold are only illustrative examples and are not specifically limited here. Any process and method that can be used to measure whether the initial three-dimensional model meets the requirements is within the protection scope of the present invention.

[0066] In this embodiment, after determining the evaluation result of the initial 3D model, it can be determined whether the evaluation result meets the evaluation result threshold. If the evaluation result meets the evaluation result threshold, it indicates that the relevant personnel are relatively satisfied with the initial 3D model, and the corresponding initial 3D model can be determined as the final target 3D model. Conversely, if the evaluation result does not meet the threshold, it indicates that the relevant personnel are relatively dissatisfied with the initial 3D model, and it is necessary to adjust the initial 3D model or redetermine a target 3D model that satisfies the relevant personnel.

[0067] Optionally, based on the evaluation results, it can be determined whether the initial 3D model needs adjustment. If the evaluation result of the initial 3D model meets the evaluation result threshold, that is, if the initial 3D model meets the evaluation criteria (preset criteria), it can directly proceed to subsequent applications or undergo minor optimization. If the evaluation result of the initial 3D model does not meet the evaluation result threshold, that is, if the initial 3D model does not meet the preset criteria, the specific reasons can be analyzed (such as large depth estimation error, inaccurate feature point matching, fusion problems, etc.), and corresponding adjustment strategies can be set to adjust the initial 3D model.

[0068] In this embodiment of the invention, determining the evaluation results and, based on those results, determining the target 3D model are crucial steps in the 3D scene reconstruction process. This not only ensures the quality of the final target 3D model but also provides guidance for subsequent optimization and improvement of the 3D reconstruction. The optimized target 3D model will be more accurate and reliable, thus providing a solid foundation for using the final target 3D model for ranging, navigation, object recognition, and other applications.

[0069] In steps S102 to S110 of this embodiment of the invention, if it is necessary to construct a 3D model corresponding to the target scene, video acquisition devices deployed at multiple perspectives of the target scene can be used to acquire video data of the target scene, obtaining multiple video data corresponding to multiple perspectives. The video data can be analyzed to obtain corresponding feature data and the correlation between the feature data. The video data is then fused using these correlations to obtain a fusion result. Depth information between the target scene and the video acquisition devices is extracted from the fusion result, and an initial 3D model is constructed using this depth information. The constructed initial 3D model is then evaluated to obtain an evaluation result. The initial 3D model whose evaluation result meets the evaluation result threshold is determined as the target 3D model of the target scene. In this embodiment, a multi-view video data fusion technology is adopted, which can dynamically identify the correlation between feature datasets of multiple views. Under specific conditions (such as when a view is occluded or there is insufficient lighting), the proportion of video data from that view in the fusion result can be automatically reduced, thereby improving the reliability and stability of the fusion result. This achieves the technical effect of improving the accuracy of the 3D model construction of the scene and solves the technical problem of low accuracy in the 3D model construction of the scene.

[0070] The embodiments of the present invention will now be described in detail with reference to the steps described above.

[0071] As an optional embodiment, step S104, acquiring multiple feature data from multiple video data, includes: using an attention mechanism to identify multiple feature data from multiple video data respectively.

[0072] In this embodiment, during the process of acquiring multiple feature data from multiple video data, an attention mechanism can be used to identify multiple video data and obtain multiple feature data from each video data.

[0073] Optionally, in order to process multi-view video data more efficiently and intelligently, and to ensure that the feature data extracted from each viewpoint is a representative and informative local area in the target scene, attention mechanisms can be used to identify and extract more critical feature data from multiple video data.

[0074] Alternatively, the attention mechanism is a computational model that mimics human visual attention selection. It can focus on certain parts of the input data (i.e., local regions with representativeness and information content) while ignoring other unimportant or redundant parts. In multi-view video fusion scenarios, the attention mechanism can dynamically identify and centrally process feature points with distinctiveness and information value in each video data, thereby improving the accuracy of feature extraction and the efficiency of subsequent fusion processes.

[0075] In this embodiment of the invention, the above method enables more intelligent extraction of feature data from multi-view video streams, which will then be used for video fusion and 3D reconstruction. The introduction of the attention mechanism not only improves the efficiency of feature extraction but also enhances the system adaptability and robustness of the 3D model construction method for handling scenes, making the system more accurate and stable when dealing with complex and dynamic scenes. This processing method is crucial for achieving high-quality video fusion and 3D scene reconstruction, especially in real-time applications such as autonomous driving, robot navigation, and augmented reality, providing a more stable and reliable foundation of visual information.

[0076] As an optional embodiment, an attention mechanism is used to identify multiple feature data in multiple video data, including: using a feature extraction model to extract features from video frames in multiple video data to obtain multiple feature maps; and using an attention mechanism to identify multiple feature data from the multiple feature maps.

[0077] In this embodiment, during the process of identifying multiple feature data from multiple video data using an attention mechanism, a feature extraction model can be used to extract features from video frames in the multiple video data, resulting in multiple feature maps. The attention mechanism can then be used to identify the corresponding feature data from these feature maps. Here, a video frame can be any image frame in the video data.

[0078] Optionally, the process of using attention mechanisms and feature extraction models to process video frames from multiple video data to identify and extract key feature data can be divided into two steps: feature extraction and feature recognition.

[0079] Optionally, video frames can be extracted from each video data set. A video frame can be any image frame within the video data. For real-time performance and efficiency, video data typically undergoes preprocessing, such as decoding, noise reduction, and color correction. A feature extraction model, such as a deep learning-based convolutional neural network model, is selected. These models, once trained, can automatically identify and extract representative features from images, such as object edges, corners, and textures.

[0080] Optionally, for each video frame, a feature extraction model is applied. This model identifies local features within the video frame and generates one or more feature maps. A feature map can be understood as an abstract representation of the original image, where each pixel or unit corresponds to the importance or descriptor of a feature, thus preserving spatial location and feature information.

[0081] Optionally, the attention mechanism is a computational method that focuses more on the important parts of the input data. In this embodiment, the attention mechanism is used to identify and focus the most important feature data from the feature map, namely those feature points that are valuable for subsequent processing (such as video fusion and 3D reconstruction). The attention mechanism dynamically selects salient feature points by analyzing each feature point in the feature map, evaluating its importance and relevance in the current scene. These feature points may be object boundaries, areas with complex textures, key visual landmarks, etc., which can be stably identified in video frames from different viewpoints. The attention mechanism not only identifies key feature points but also assigns a weight to each feature point, which reflects the importance of the feature point. Feature points with higher weights will receive greater attention and play a greater role in subsequent video fusion and 3D reconstruction processes. Based on the results of the attention mechanism's identification and weight assignment, the highlighted feature data is extracted from the feature map. This data includes feature descriptors, feature point location information, size, orientation, etc., and forms the basis for subsequent processing.

[0082] As an optional embodiment, the feature extraction model includes a Gaussian model and a convolutional model, and the feature map includes a first feature map and a second feature map. The Gaussian model corresponds to the first weight and the first feature map, and the convolutional model corresponds to the second weight and the second feature map. The feature extraction model is used to extract features from video frames in multiple video data to obtain multiple feature maps, including: inputting the video frames into the Gaussian model and using the Gaussian model to extract features from the video frames to obtain a first feature map; inputting the video frames into the convolutional model and using the convolutional model to extract features from the video frames to obtain a second feature map; assigning a first weight to the first feature map and a second weight to the second feature map, and then weighted summing to obtain the second feature map.

[0083] In this embodiment, during the feature extraction process of video frames from multiple video data using a feature extraction model, the video frames can be input into a Gaussian model within the feature extraction model to extract features and obtain a corresponding first feature map. Alternatively, the video frames can be input into a convolutional model within the feature extraction model to extract features and obtain a second feature map. First and second weights can be assigned to the first and second feature maps respectively, and the final feature map is obtained through weighted summation. The feature extraction model can include a Gaussian model and a convolutional model. The feature map can include a first feature map and a second feature map. The first feature map can be the feature map extracted using the Gaussian model. The second feature map can be the feature map extracted using the convolutional model. The first weight can be the weight assigned to the first feature map. The second weight can be the weight assigned to the second feature map.

[0084] Optionally, a feature extraction model is used to extract features from video frames from each camera's perspective. These features should contain enough information to distinguish different objects and regions in the scene. An attention mechanism is applied to the extracted feature map to dynamically identify key feature points between each camera's perspective. These key feature points are usually prominent objects, edges, and corners in the scene, which can be identified from multiple perspectives.

[0085] Optionally, feature maps are obtained by extracting features from video frames (video frame images) at various viewpoints using the feature extraction model shown below:

[0086]

[0087] Where F can be used to represent the feature extraction model; I ij This represents the j-th frame image from the perspective of the i-th camera; It can be used to represent a Gaussian model; M and N are used to represent the dimensions of a video frame; (μ x ,μ y ) can be used to represent the center coordinates of a video frame; σ x and σ y It can be used to represent the parameters controlling the Gaussian distribution corresponding to the Gaussian model; ω2·Convolution(I ij ω1 can be used to represent a convolutional model, used to perform convolution operations on video frames to extract features; ω2 can be used to represent the first weight; ω1 can be used to represent the second weight; F ij It can be used to represent the feature map obtained by the final weighted summation.

[0088] Alternatively, after determining the feature map, the feature map can be input into an attention mechanism to identify feature data from the feature map.

[0089] Alternatively, feature data can be determined through an attention mechanism as shown below:

[0090]

[0091] Among them, K ij A can be used to represent the set of key feature points (feature data) of the j-th frame image from the i-th camera viewpoint after the attention mechanism; A can be used to represent the attention mechanism; L can be used to represent the number of feature data in the feature map.

[0092] As an optional embodiment, the method further includes: determining the feature similarity between multiple feature data based on multiple feature data from different perspectives; determining the positional relationship data between multiple feature data based on the coordinate positions corresponding to the multiple feature data from different perspectives; determining the matching weight of multiple feature data based on the feature similarity and positional relationship data, wherein the matching weight is used to represent the degree of association between the multiple feature data; in response to the matching weight exceeding the matching weight threshold, selecting feature data pairs from the multiple feature data, wherein the feature data pair includes two feature data; and constructing the association relationship corresponding to the feature data pair.

[0093] In this embodiment, the feature similarity between multiple feature data can be determined; alternatively, the positional relationship data between feature data can be determined based on multiple coordinate positions corresponding to different feature numbers. Based on the feature similarity and positional relationship data, the matching weights of multiple feature data can be determined, and it can be judged whether the matching weights exceed a matching weight threshold. If the matching weights exceed the matching weight threshold, feature data pairs can be selected from the multiple feature data, and the association relationship between the two feature data in the feature data pair can be constructed. The matching weights can be used to represent the degree of association between multiple feature data. A feature data pair can include two feature data that are related. Positional relationship data can also be called positional relationship weights.

[0094] Optionally, in addition to extracting features from multi-view video data, the relationships between features can be further analyzed to determine which feature points are associated, thereby constructing the association between two feature data in a feature data pair.

[0095] Optionally, for multiple feature data extracted from different perspectives, each feature data contains a descriptor that quantifies the visual appearance of the feature points, such as color, texture, and shape. The feature similarity between the descriptors of the feature data from different perspectives is calculated. For example, similarity measures that can be used include, but are not limited to, Euclidean distance, cosine similarity, and Mahalanobis distance. If the feature descriptors exist in vector form, the dot product or cosine similarity between the vectors corresponding to two feature data can be calculated.

[0096] Optionally, during feature extraction, each feature data also includes its coordinate position in the original video frame. This positional information is crucial for understanding the relationship between feature points in three-dimensional space. By analyzing the coordinate positions of feature points from different viewpoints, the relative positional relationships between them are calculated, including information such as direction and distance. The calculation of positional relationships can be achieved through simple geometric operations or more complex models, with the aim of evaluating whether feature points are in the same or similar positions from different viewpoints. Feature similarity and positional relationship data are combined, and a matching weight is calculated between feature data using a weighted formula. The matching weight reflects the strength of the association between feature points and is an important basis for subsequent feature matching and video fusion processes. To ensure that only high-confidence matches are selected, the system sets a matching weight threshold. Only when the matching weight exceeds this threshold are two feature data considered a match. By comparing the matching weights between all feature data, feature data pairs that exceed the threshold are filtered out. Feature points in these feature data pairs are considered to have a consistent or similar correspondence from different viewpoints.

[0097] Optionally, feature matching algorithms can be used to match key points from different perspectives.

[0098] For example, feature similarity can be determined using the following formula:

[0099]

[0100] Where, k ip ,k jq It can be used to represent two feature data in a feature data pair; S(k) ip ,k jq α can be used to represent the feature similarity between two feature data in a feature data pair; α and β can be adjustable parameters, with values ​​ranging from 0 to 1; and They can represent the vectors corresponding to the two feature data respectively.

[0101] For another example, let P(k) ip )=(x ip ,y ip ) and P(k jq )=(x jq ,y jq Given the coordinates of keypoints in their respective viewpoint images, the positional relationship weights can be determined using the following formula:

[0102]

[0103] Among them, W pos (k ip ,k jq) can be used to represent the positional relationship weight between two feature data; γ can be used to represent the adjustment parameter of the positional relationship; x ip and x jq It can be used to represent the x-coordinate of two feature data; y ip and y jq It can be used to represent the ordinate of two feature data.

[0104] As an alternative example, the matching weight can be determined using the following formula:

[0105] M(k ip ,k jq )=λ×S(k ip ,k jq )+(1-λ)×W pos (k ip ,k jq )

[0106] Where M(k) ip ,k jq ) can be used to represent the matching weight between two feature data; λ can be used to represent the parameter that balances the weights of feature similarity and positional relationship. If M(k ip ,k jq If the threshold (matching weight threshold) is exceeded, it indicates that the two feature data (k) in the feature data pair are... ip and k jq ) is the key point of matching.

[0107] As an optional embodiment, constructing the association relationship corresponding to the feature data pair includes: obtaining the feature vectors corresponding to the two feature data in the feature data pair respectively, and determining the inner product similarity and the distance similarity between the two feature vectors; obtaining the coordinate positions of the two feature data in the feature data pair from their respective perspectives, and determining the positional relationship weight between the two coordinate positions; and constructing the association relationship based on the inner product similarity, distance similarity, and positional relationship weight.

[0108] In this embodiment, during the process of constructing the association relationship corresponding to feature data, the feature vectors corresponding to the two feature data in the feature data pair can be determined, and the inner product similarity and distance similarity between the two feature vectors can be determined. The coordinate positions of the two feature data in the feature data pair from their respective perspectives can be obtained, and the corresponding positional relationship weights can be determined. Based on the inner product similarity, distance similarity, and positional relationship weights, the association relationship is constructed.

[0109] Optionally, the correlation between feature data is constructed by analyzing and synthesizing the inner product similarity, distance similarity, and positional relationship weights of feature vectors. This process ensures the accuracy of feature point matching from different perspectives, thus providing a solid foundation for subsequent 3D scene reconstruction and ranging.

[0110] Optionally, a feature descriptor is extracted from each feature data point; this is a vector that quantifies the visual attributes of the feature point. The feature descriptor is typically output by the feature extraction model and contains information such as the feature point's color, texture, and shape. Each feature data point corresponds to a feature vector, representing the attribute of that feature point. The vector's dimension and element values ​​depend on the design and training of the feature extraction model.

[0111] Optionally, for two feature data points in a feature data pair, the system calculates the inner product similarity between their feature vectors. The inner product similarity is a measure based on the consistency of vector orientation and reflects the similarity of the two feature points in visual attributes.

[0112] For example, the inner product similarity S can be calculated using the following formula. f :

[0113]

[0114] Among them, S f (k1,k2) can be used to represent the inner product similarity between k1 and k2; k1 and k2 can each represent two keypoints. It can be used to represent the eigenvector of k1; It can be used to represent the eigenvector of k2.

[0115] Alternatively, the Mahalanobis distance similarity S of the feature vectors can also be calculated using the following formula. m :

[0116]

[0117] Among them, S m (k1,k2) can be used to represent the Mahalanobis distance similarity between k1 and k2; ∑ can be used to represent the covariance matrix of the eigenvectors.

[0118] Optionally, let (x1, y1) and (x2, y2) be the coordinates of k1 and k2 in their respective viewpoint images, and the positional relationship weight W can be determined using the following formula. p :

[0119] W p (k1,k2)=exp(-θ((x1-x2) 2 +(y1-y2) 2 ))

[0120] Among them, W p (k1,k2) can be used to represent the positional relationship weight between k1 and k2; θ can be used to represent the adjustment parameter.

[0121] Alternatively, by combining feature similarity and positional relationships, the edge weights w can be constructed as shown below:

[0122]

[0123] Here, w(k1,k2) can be used to represent the edge weight between k1 and k2; ω1 and ω2 are used to represent the weight allocation parameters, and ω1+ω2=1.

[0124] Optionally, key points from different viewpoints are matched to establish correspondences. Each camera viewpoint is considered a node in a graph, and the matched keypoint pairs are considered edges between nodes. Weights are assigned to each edge based on the matching degree and positional relationship of the key points, thus constructing a graph of relationships between viewpoints. Assigning weights to each edge to construct the graph of relationships between viewpoints: Let the matched keypoint pairs extracted from two different camera viewpoints be (k1, k2), where k1 comes from viewpoint V1 and k2 comes from viewpoint V2.

[0125] As an optional embodiment, step S106, based on the correlation between multiple feature data, fuses multiple video data to obtain a fusion result, including: determining the fusion weights corresponding to each of the multiple video data based on the correlation; and fusing the multiple video data based on the fusion weights to obtain a fusion result.

[0126] In this embodiment, multiple video data are fused based on the correlation between multiple feature data. The fusion weights for each video data point can be determined based on these correlations. The fusion weights can then be used to fuse the multiple video data to obtain the corresponding fusion result.

[0127] Optionally, the calculated fusion weights are used to perform pixel-level fusion of video frames from various camera perspectives. During the fusion process, it is important to maintain the spatiotemporal consistency between video frames and avoid flickering, blurring, or misalignment. This requires the application of some post-processing techniques. After the above steps, a high-quality fused video stream is generated. This video stream should contain complementary information from multiple perspectives while reducing the impact of adverse factors such as lighting changes, occlusion, and camera movement. Real-time performance must be ensured throughout the fusion process. At the same time, based on the actual application results, the steps of feature extraction, key point recognition, weight calculation, and fusion strategy are continuously optimized to improve the overall performance and robustness of the system.

[0128] Optionally, pixel-level fusion is performed on video frames from each camera's perspective. During the fusion process, care is taken to maintain the spatiotemporal consistency between video frames: it can be assumed that there are n camera perspectives, and the pixel value of the video frame from the i-th camera perspective at coordinates (x, y) is P. i (x,y), the calculated fusion weight for this viewpoint is w. i (satisfies ∑) i =1 n w i =1).

[0129] Optionally, in the basic fusion calculation, a weighted sum can be performed to obtain the initial fused pixel value F(x,y):

[0130]

[0131] As an optional embodiment, the fusion weights corresponding to multiple video data are determined based on the correlation relationship, including: obtaining the time information of the acquired video data, and setting a spatiotemporal consistency factor based on the time information, wherein the spatiotemporal consistency factor includes the time information and the coordinate position of the feature data; and determining the fusion weights based on the spatiotemporal consistency factor and the correlation relationship.

[0132] In this embodiment, during the process of determining the fusion weights corresponding to multiple video data based on the correlation, the temporal information of the video data can be collected, and a spatiotemporal consistency factor can be set based on the temporal information. The fusion weights are determined using the spatiotemporal consistency factor and the correlation, wherein the spatiotemporal consistency factor may include temporal information and the coordinate positions of the feature data.

[0133] Optionally, in the embodiments of the invention, considering spatiotemporal consistency, let t represent time (time information). For pixels in the same position in two consecutive frames, a spatiotemporal consistency factor λ(x,y,t) is defined. Assuming the fused pixel value of the previous frame is F(x,y,t-1), the adjusted fused pixel value F(x,y,t) can be obtained as follows:

[0134] F(x,y,t)=λ(x,y,t)F(x,y,t)+(1-λ(x,y,t))F(x,y,t-1).

[0135] Optionally, in a multi-video fusion-based 3D scene reconstruction and ranging system, considering the spatiotemporal consistency factor when determining the fusion weights of video data is an important technical optimization point. This plays a crucial role in improving the coherence of the fused video and the accuracy of 3D reconstruction.

[0136] Optionally, considering the importance of timing information, in real-time video stream fusion, video data from different perspectives often exhibit slight temporal differences. These differences may be caused by variations in hardware latency, network transmission time, or processing speed. Taking timing information into account helps the system maintain temporal correspondence between frames when fusing multi-view videos, avoiding scene misalignment or flickering caused by timing inconsistencies.

[0137] Optionally, regarding the setting of the spatiotemporal consistency factor, this factor not only includes temporal information but also considers the coordinate positions of the feature data in their respective viewpoints. This means that when determining the fusion weights, the system not only focuses on the visual similarity of feature points but also ensures their continuity and consistency in spatial location. For example, in consecutive video frames, even if the visual appearance of feature points changes slightly, as long as their positions are relatively stable, the system will still assign them a high fusion weight. This helps maintain the coherence of the video stream and the stability of the 3D reconstruction.

[0138] Optionally, regarding the dynamic adjustment of fusion weights, the system can dynamically adjust the fusion weights of video data by utilizing the spatiotemporal consistency factor and the correlation between feature points. For feature points that are consistent in time and stable in location, the video data from their respective viewpoints will be assigned higher fusion weights. This indicates that the system relies more on data with high spatiotemporal consistency when fusing videos, thereby improving the quality of the fused video and the accuracy of 3D reconstruction.

[0139] Optionally, to enhance robustness and adaptability, by considering spatiotemporal consistency, the system can more accurately identify and match feature points under complex conditions such as changes in lighting, occlusion, and camera movement. This means that even in dynamic and uncertain environments, the system can maintain high robustness and adaptability, generating stable, flicker-free fused videos and accurate, detailed 3D scene models.

[0140] Optionally, for the optimization of video fusion and 3D reconstruction, a fusion weight calculation method based on a spatiotemporal consistency factor can help optimize the video fusion and 3D reconstruction process. It ensures that the system not only focuses on the static similarity of feature points but also fully considers the consistency of temporal series information and spatial location, thereby generating fused videos and 3D models that better reflect changes in real-world scenes.

[0141] In this embodiment, a spatiotemporal consistency factor is set based on the temporal information of the video data and the coordinate position of the feature data. This is an important point in determining the video data fusion weights. The method described above not only improves the visual quality of the fused video but also significantly enhances the accuracy of 3D scene reconstruction, especially when dealing with dynamic and complex environments. By dynamically adjusting the fusion weights, the system can better cope with challenges such as changes in illumination, occlusion, and camera movement, ensuring the stability and robustness of 3D scene reconstruction and ranging functions.

[0142] As an optional embodiment, step S108, constructing an initial three-dimensional model of the target scene based on depth information, includes: converting the depth information into three-dimensional coordinates, and constructing the initial three-dimensional model based on the three-dimensional coordinates.

[0143] In this embodiment, during the process of constructing an initial 3D model of the target scene based on depth information, the depth information can be converted into 3D coordinates, and the initial 3D model can be constructed based on the 3D coordinates.

[0144] Optionally, depth information can be obtained from the fused video stream using a depth estimation network. Depth information is typically presented as a depth map, where the value of each pixel represents the distance between the corresponding scene point and the camera. To convert the information in the depth map into 3D coordinates, the system needs to know the camera's intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as position and orientation). The intrinsic parameters are used to convert image coordinates to normalized device coordinates, while the extrinsic parameters are used to convert the normalized device coordinates to 3D coordinates in the world coordinate system.

[0145] Optionally, by combining depth information d with camera parameters, the two-dimensional coordinates (u,v) of each pixel in the two-dimensional image are converted into coordinates (X,Y,Z) in three-dimensional space, thereby constructing an initial three-dimensional model of the target scene.

[0146] As an optional embodiment, step S110, determining the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene, includes: determining the initial three-dimensional model as the target three-dimensional model in response to the evaluation result exceeding the evaluation result threshold; adjusting the initial three-dimensional model in response to the evaluation result not exceeding the evaluation result threshold, and determining the adjusted initial three-dimensional model as the target three-dimensional model, wherein the evaluation result of the adjusted initial three-dimensional model exceeds the evaluation result threshold.

[0147] In this embodiment, during the process of determining the initial 3D model whose evaluation result meets the evaluation result threshold as the target 3D model, if the evaluation result exceeds the evaluation result threshold, the initial 3D model can be determined as the target 3D model. If the evaluation result does not exceed the evaluation result threshold, the initial 3D model is adjusted, and the adjusted initial 3D model that exceeds the evaluation result threshold can be determined as the target 3D model.

[0148] Optionally, the process of determining the target 3D model of the target scene is based on a comparison between the evaluation results of the initial 3D model and a preset evaluation result threshold. The above method ensures that a target 3D model that meets quality standards can be generated, and also provides a way for iterative optimization and adjustment to address potential shortcomings of the initial 3D model.

[0149] Optionally, an evaluation result of the initial 3D model is calculated. This evaluation may be based on multiple aspects, including but not limited to the model's geometric accuracy, texture quality, face count, computational efficiency, and its fit with the actual scene. The evaluation result is calculated by comparing factors such as the differences between the model and the actual scene, the model's self-consistency, and completeness. An evaluation result threshold is set to determine whether the initial 3D model meets the quality requirements. If the evaluation result exceeds the preset threshold, it indicates that the model's quality has reached the expected standard and can be directly identified as the target 3D model for subsequent 3D scene analysis and ranging functions.

[0150] Through the above process, the system of this invention can ensure that the generated 3D model meets specific application requirements and quality standards. This mechanism is crucial for applications requiring high-precision 3D models, such as autonomous driving, virtual reality, augmented reality, and robot navigation. Simultaneously, by setting evaluation result thresholds and iterative adjustment processes, the system can continuously optimize model construction when dealing with complex and dynamic environments, improving the model's adaptability and reliability.

[0151] In this embodiment of the invention, if it is necessary to construct a 3D model corresponding to a target scene, video acquisition devices deployed at multiple perspectives of the target scene can be used to acquire video data of the target scene, obtaining multiple video data corresponding to multiple perspectives. The video data can be analyzed to obtain corresponding feature data and the correlation between feature data. The video data is then fused using these correlations to obtain a fusion result. Depth information between the target scene and the video acquisition devices is extracted from the fusion result, and an initial 3D model is constructed using this depth information. The constructed initial 3D model is then evaluated to obtain an evaluation result. The initial 3D model whose evaluation result meets the evaluation result threshold is determined as the target 3D model of the target scene. In this embodiment, the multi-view video data fusion technology can dynamically identify the correlation between feature datasets of multiple perspectives. Therefore, under specific conditions (e.g., a perspective is occluded or there is insufficient lighting), the proportion of video data from that perspective in the fusion result can be automatically reduced, achieving the goal of improving the reliability and stability of the fusion result. This, in turn, improves the accuracy of the 3D model construction of the scene, solving the technical problem of low accuracy in 3D model construction of the scene.

[0152] Example 2

[0153] The following describes in detail another optional implementation method.

[0154] Currently, as 3D scene reconstruction and ranging technologies have demonstrated enormous application potential in multiple fields, such as autonomous driving, virtual reality, augmented reality, and robot navigation, these technologies rely on high-precision 3D model construction and real-time ranging capabilities to achieve accurate understanding and interaction with complex environments.

[0155] In related technologies, real-time adaptive multi-view video fusion modules face many challenges when dealing with complex scenes. They cannot dynamically identify key feature points and corresponding relationships between the viewpoints of each camera, construct a relationship graph between viewpoints, and dynamically adjust the fusion weights between viewpoints based on the current scene features. When achieving real-time fusion of multi-view videos, they often ignore spatiotemporal consistency. Therefore, there is still a technical problem of low accuracy in constructing the 3D model of the scene.

[0156] This invention proposes a method for 3D scene reconstruction and ranging based on multi-video fusion. If a 3D model of a target scene needs to be constructed, this method utilizes video acquisition devices deployed at multiple viewpoints within the target scene to collect video data, obtaining multiple video data points corresponding to multiple viewpoints. The video data can be analyzed to extract relevant feature data and determine the relationships between these feature data points. These relationships are then used to fuse the video data, yielding a fusion result. Depth information between the target scene and the video acquisition devices is extracted from the fusion result, and this depth information is used to construct an initial 3D model. The constructed initial 3D model is then evaluated to obtain an evaluation result. The initial 3D model whose evaluation result meets the evaluation result threshold is identified as the target 3D model of the target scene. In this embodiment, the multi-view video data fusion technology can dynamically identify the relationship between feature datasets from multiple viewpoints. Under specific conditions, it can automatically reduce the proportion of video data from a particular viewpoint in the fusion result, thereby improving the reliability and stability of the fusion result and ultimately enhancing the accuracy of the 3D model construction of the scene, solving the technical problem of low accuracy in 3D model construction.

[0157] The method will be further described below.

[0158] In this embodiment, Figure 2 This is a schematic diagram of a 3D scene reconstruction and ranging system based on multi-video fusion according to an embodiment of the present invention, as shown below. Figure 2As shown, the system may include: a multi-view video real-time acquisition and preprocessing module 202, a real-time adaptive multi-view video fusion module 204, a depth estimation and 3D reconstruction module 206, and an optimization and feedback mechanism module 208. The multi-view video real-time acquisition and preprocessing module 202 uses multiple high-definition cameras to simultaneously acquire video streams from different perspectives, performs preliminary processing on the acquired videos to prepare for subsequent fusion and depth estimation. The real-time adaptive multi-view video fusion module 204 uses an attention mechanism to dynamically identify key feature points and their corresponding relationships between camera perspectives. By constructing a relationship graph between perspectives, it dynamically adjusts the fusion weights between perspectives based on current scene characteristics to address challenges such as lighting changes, occlusion, and camera movement, achieving real-time fusion of multi-view videos and generating a high-quality fused video stream. The depth estimation and 3D reconstruction module 206 combines a depth estimation network from deep learning to directly extract a high-precision depth map from the fused video stream. Using the extracted depth map and multi-view geometric principles, it performs 3D point cloud reconstruction, constructs a 3D scene model, and enables real-time ranging in the 3D scene. The optimization and feedback mechanism module 208 is used to design an adaptive optimization algorithm. Based on the difference between the reconstruction result and the preset standard, it dynamically adjusts the attention mechanism, GNN parameters and the training strategy of the depth estimation network to achieve continuous optimization of system performance. It also introduces a user feedback mechanism to allow users to adjust system parameters according to actual needs.

[0159] Optionally, the multi-view video real-time acquisition and preprocessing module selects a certain number and type of high-definition cameras according to the needs of the application scenario, determines the installation position and angle of the cameras to ensure that different perspectives of the target scene can be covered and blind spots are minimized. The resolution, frame rate and exposure parameters of the cameras are configured to adapt to different lighting conditions and environmental changes. Hardware synchronization signals or software synchronization algorithms are used so that all cameras can start and stop video recording simultaneously to reduce errors caused by time differences. The module captures the video stream from the perspective of each camera in real time and transmits it to the processing unit for subsequent processing.

[0160] Optionally, the received video stream is decoded, converting it from a compressed format into a processable frame sequence. This ensures the decoding process is synchronized with video acquisition, avoiding frame loss or delays. Image denoising algorithms are applied to remove noise and improve image quality, while preserving edge details and avoiding blurring important information during denoising. Since different cameras may have color deviations, color correction is necessary to ensure color consistency across different viewing angles. This can be achieved using color space conversion, histogram matching, or color mapping methods. Video frames are cropped to remove unnecessary edges as needed for subsequent processing. The frame size is adjusted to meet the requirements of subsequent fusion and depth estimation modules, maintaining the aspect ratio or adjusting it appropriately based on the specific algorithm. The video frame processing rate is adjusted according to system resources and application requirements to ensure real-time performance. Frame skipping and interpolation can be used to control the frame rate while maintaining video stream smoothness. A caching mechanism is implemented to address network latency or processing speed mismatches. A buffer is used to temporarily store acquired but unprocessed video frames, ensuring data stream continuity and stability.

[0161] Optionally, the real-time adaptive multi-view video fusion module uses a feature extraction model to extract features from video frames from each camera viewpoint. These features should contain enough information to distinguish different objects and regions in the scene. An attention mechanism is applied to the extracted feature map to dynamically identify key feature points between each camera viewpoint. These key points are usually prominent objects, edges, and corners in the scene, which can be identified in multiple viewpoints.

[0162] Optionally, a feature matching algorithm is used to match key points in different viewpoints, establish the correspondence between them, treat each camera viewpoint as a node in the graph, treat the matched key point pairs as edges between nodes, assign weights to each edge according to the matching degree and positional relationship of the key points, and construct a relationship graph between viewpoints.

[0163] Optionally, the real-time adaptive multi-view video fusion module analyzes the features of the current scene, including but not limited to lighting conditions, occlusion, and camera movement. This information can be obtained by analyzing the brightness, contrast, and motion vector features of video frames. Based on the scene features, the module uses a model to calculate the importance or credibility of each camera viewpoint in the current scene. These weights will be used in the subsequent video fusion process. Based on the calculated weights, the fusion weights between the views are dynamically adjusted. Views with higher importance are given greater weights, while views affected by changes in lighting, occlusion, or camera movement are given lower weights.

[0164] Optionally, the calculated fusion weights are used to perform pixel-level fusion of video frames from various camera perspectives. During the fusion process, it is important to maintain the spatiotemporal consistency between video frames to avoid flickering, blurring, or misalignment. This requires the application of some post-processing techniques. After the above steps, a high-quality fused video stream is generated. This video stream should contain complementary information from multiple perspectives while reducing the impact of adverse factors such as lighting changes, occlusion, and camera movement. Real-time performance must be ensured throughout the fusion process to meet the needs of real-time applications. At the same time, based on the actual application results, the steps of feature extraction, key point recognition, weight calculation, and fusion strategy are continuously optimized to improve the overall performance and robustness of the system.

[0165] Optionally, a feature extraction model is used to extract features from video frames at each camera's viewpoint. The feature extraction model shown below is used to extract features from video frames (video frame images) at each viewpoint, resulting in feature maps:

[0166]

[0167] Where F can be used to represent the feature extraction model; I ij This represents the j-th frame image from the perspective of the i-th camera; It can be used to represent a Gaussian model; M and N are used to represent the dimensions of a video frame; (μ x ,μ y ) can be used to represent the center coordinates of a video frame; σ x and σ y It can be used to represent the parameters controlling the Gaussian distribution corresponding to the Gaussian model; ω2·Convolution(I ij ω1 can be used to represent a convolutional model, used to perform convolution operations on video frames to extract features; ω2 can be used to represent the first weight; ω1 can be used to represent the second weight; F ij It can be used to represent the feature map obtained by the final weighted summation.

[0168] Optionally, an attention mechanism can be applied to the feature map to identify key feature points. The feature data can be determined through the attention mechanism shown below:

[0169]

[0170] Among them, K ij A can be used to represent the set of key feature points (feature data) of the j-th frame image from the i-th camera viewpoint after the attention mechanism; A can be used to represent the attention mechanism; L can be used to represent the number of feature data in the feature map.

[0171] Optionally, feature matching algorithms can be used to match key points from different perspectives. Feature similarity can be determined using the following formula:

[0172]

[0173] Where, k ip ,k jq It can be used to represent two feature data in a feature data pair; S(k) ip ,k jq α can be used to represent the feature similarity between two feature data in a feature data pair; α and β can be adjustable parameters, with values ​​ranging from 0 to 1; and They can represent the vectors corresponding to the two feature data respectively.

[0174] Alternatively, let P(k) ip )=(x ip ,y ip ) and P(k jq )=(x jq ,y jq Given the coordinates of keypoints in their respective viewpoint images, the positional relationship weights can be determined using the following formula:

[0175]

[0176] Among them, W pos (k ip ,k jq ) can be used to represent the positional relationship weight between two feature data; γ can be used to represent the adjustment parameter of the positional relationship; x ip and x jq It can be used to represent the x-coordinate of two feature data; y ip and y jq It can be used to represent the ordinate of two feature data.

[0177] Alternatively, the matching weight can be determined using the following formula:

[0178] M(k ip ,k jq )=λ×S(k ip ,k jq )+(1-λ)×W pos (k ip ,k jq )

[0179] Where M(k) ip ,k jq ) can be used to represent the matching weight between two feature data; λ can be used to represent the parameter that balances the weights of feature similarity and positional relationship. If M(kip ,k jq If the threshold (matching weight threshold) is exceeded, it indicates that the two feature data (k) in the feature data pair are... ip and k jq ) is the key point of matching.

[0180] Optionally, weights are assigned to each edge to construct a graph of relationships between viewpoints: Let the matching keypoint pairs extracted from two different camera viewpoints be (k1, k2), where k1 comes from viewpoint V1 and k2 comes from viewpoint V2.

[0181] The inner product similarity S can be calculated using the following formula. f :

[0182]

[0183] Among them, S f (k1,k2) can be used to represent the inner product similarity between k1 and k2; k1 and k2 can each represent two keypoints. It can be used to represent the eigenvector of k1; It can be used to represent the eigenvector of k2.

[0184] Alternatively, the Mahalanobis distance similarity S of the feature vectors can also be calculated using the following formula. m :

[0185]

[0186] Among them, S m (k1,k2) can be used to represent the Mahalanobis distance similarity between k1 and k2; ∑ can be used to represent the covariance matrix of the eigenvectors.

[0187] Optionally, let (x1, y1) and (x2, y2) be the coordinates of k1 and k2 in their respective viewpoint images, and the positional relationship weight W can be determined using the following formula. p :

[0188] W p (k1,k2)=exp(-θ((x1-x2) 2 +(y1-y2) 2 ))

[0189] Among them, W p (k1,k2) can be used to represent the positional relationship weight between k1 and k2; θ can be used to represent the adjustment parameter.

[0190] Alternatively, by combining feature similarity and positional relationships, the edge weights w can be constructed as shown below:

[0191]

[0192] Here, w(k1,k2) can be used to represent the edge weight between k1 and k2; ω1 and ω2 are used to represent the weight allocation parameters, and ω1+ω2=1.

[0193] Optionally, pixel-level fusion is performed on video frames from various camera perspectives. During the fusion process, care is taken to maintain the spatiotemporal consistency between video frames: Suppose there are n camera perspectives, and the pixel value of the video frame from the i-th camera perspective at coordinates (x, y) is P. i (x,y), the calculated fusion weight for this viewpoint is w. i (satisfies ∑) i =1 n w i =1).

[0194] Optionally, in the basic fusion calculation, a weighted sum can be performed to obtain the initial fused pixel value F(x,y):

[0195]

[0196] Optionally, considering spatiotemporal consistency, let t represent time (time information). For pixels at the same position in two consecutive frames, a spatiotemporal consistency factor λ(x,y,t) is defined. Assuming the fused pixel value of the previous frame is F(x,y,t-1), the adjusted fused pixel value F(x,y,t) can be obtained as follows:

[0197] F(x,y,t)=λ(x,y,t)F(x,y,t)+(1-λ(x,y,t))F(x,y,t-1).

[0198] Optionally, the depth estimation and 3D reconstruction module selects a depth estimation network for the current application scenario and trains it on a large amount of data to learn the ability to predict depth information from monocular or multi-view images. Each frame in the fused video stream is used as the input to the depth estimation network. Since the video stream is continuous, temporal information is used to enhance the accuracy of depth estimation. The depth estimation network processes the input frames and outputs the corresponding depth map. Each pixel value in the depth map represents the depth of the corresponding point in the scene. The predicted depth map is post-processed to remove noise, fill holes, and smooth edges. This can be achieved through filters or depth map repair algorithms.

[0199] Optionally, the intrinsic and extrinsic parameters of each camera are obtained. These parameters are necessary for 3D reconstruction. Using the camera intrinsic parameters and depth map, each pixel in the depth map is converted into a point cloud in 3D space. This usually involves converting pixel coordinates (u,v) and depth values ​​d into 3D coordinates (X,Y,Z). If multiple camera views are used, the point clouds from different views need to be fused. This can be achieved by converting the point cloud to a unified world coordinate system and removing duplicate or conflicting points. The fused point cloud is then optimized to remove noise, smooth the surface, and fill holes. This can be achieved through point cloud filtering and surface reconstruction techniques.

[0200] Optionally, the optimized point cloud can be converted into a surface mesh to construct a 3D scene model. Texture information from the original video frames can be mapped onto the 3D scene model to enhance its realism. This typically involves matching pixels in the video frames with vertices in the 3D model and applying texture mapping algorithms.

[0201] Optionally, ray casting is performed on the 3D scene model based on the points or paths input by the user. This involves converting the 2D input into rays in 3D space, detecting the intersections of the rays with the surface meshes in the 3D scene model, which can be achieved through a ray-triangle intersection test algorithm. The distance is calculated based on the 3D coordinates of the intersection and the starting point of the ray, which can be achieved through a simple 3D space distance formula. The calculated distance is then fed back to the user in real time.

[0202] Optionally, the optimization and feedback mechanism establishes evaluation criteria for the 3D reconstruction results. These criteria should be set based on application requirements. A performance monitoring module is implemented to evaluate the reconstruction results of the current system in real time or periodically, compare them with preset standards, analyze the differences between the reconstruction results and the preset standards, identify performance bottlenecks or deficiencies, and design an adaptive adjustment strategy, including: adjusting the attention mechanism: dynamically adjusting the attention mechanism based on which regions in the reconstruction results perform poorly, making the network pay more attention to these regions; optimizing graph neural network parameters: automatically adjusting the weights and parameters of the graph neural network using optimization algorithms based on performance differences; and depth estimation network training strategy: adjusting the learning rate, batch size, training cycle, and training parameters based on error feedback, or introducing new training data and methods. The above adjustment strategies are applied to the system, and the performance changes after adjustment are monitored.

[0203] Optionally, an intuitive and easy-to-use user interface can be designed to allow users to view reconstruction results, evaluate system performance, and provide feedback. The user interface can be used to collect user satisfaction with the reconstruction results, suggestions, and improvement opinions. The collected user feedback can be organized and analyzed to identify the performance indicators and common problems that users care about most. Based on user feedback, system parameters can be adjusted to better meet user needs. User feedback can be incorporated into the adaptive optimization algorithm to form a closed-loop feedback mechanism. The system can continuously optimize its performance based on user feedback and internal performance monitoring results.

[0204] Optionally, a periodic evaluation cycle can be set to comprehensively evaluate the system performance, including but not limited to reconstruction quality, computational efficiency, and user satisfaction. After the evaluation, the advantages and disadvantages of the current system can be summarized, the effectiveness of the adaptive optimization algorithm and user feedback mechanism can be reflected, and an improvement plan for the next stage can be formulated based on the evaluation results and user feedback. The system can be iteratively updated according to the improvement plan to continuously optimize system performance and improve user experience.

[0205] Compared with related technologies, in this embodiment of the invention, by employing multi-view video fusion technology combined with a real-time adaptive multi-view video fusion module, key feature points and their corresponding relationships between the viewpoints of each camera can be dynamically identified, and the fusion weights can be dynamically adjusted according to the current scene characteristics. This mechanism effectively addresses challenges such as lighting changes, occlusion, and camera movement, significantly improving the quality of the fused video stream. By introducing an optimization and feedback mechanism, establishing evaluation criteria for 3D reconstruction results, and evaluating the reconstruction results of the current system in real time or periodically, and comparing them with preset standards, this mechanism can promptly identify system performance bottlenecks or deficiencies, and design adaptive adjustment strategies for optimization. These adjustment strategies include adjusting the attention mechanism, optimizing graph neural network parameters, and adjusting the training strategy of the depth estimation network. In addition, the system allows users to view reconstruction results, evaluate system performance, and provide feedback through the user interface. This user feedback is incorporated into the adaptive optimization algorithm, forming a closed-loop feedback mechanism. This mechanism enables the system to continuously optimize its performance based on user feedback and internal performance monitoring results, constantly improving user experience and system performance.

[0206] The following explanation uses a conference room as the target scene to further illustrate the three-dimensional scene reconstruction method in this embodiment of the invention.

[0207] Four high-definition cameras are installed in different corners of the conference room to ensure comprehensive coverage of all angles and minimize blind spots. The camera resolution and frame rate parameters are configured to adapt to changes in indoor lighting, ensuring clear video streams under various lighting conditions. Software synchronization algorithms are used to ensure all cameras can start and stop recording simultaneously, reducing errors caused by time differences. The captured video is decoded in real-time, converting compressed video streams into processable frame sequences. The decoding process is synchronized with video capture to avoid frame drops or delays. Image denoising algorithms are applied to remove noise from the video, improving image quality while maintaining clear edge details and preventing blurring of important information. Color correction is performed using color space conversion, histogram matching, or color mapping methods to ensure consistent color across different viewing angles, improving fusion effects. Video frames are cropped and resized according to subsequent processing needs, maintaining the aspect ratio or adjusting appropriately based on specific algorithms. A caching mechanism is implemented to temporarily store captured but unprocessed video frames, ensuring the continuity and stability of the data stream.

[0208] Using a feature extraction model, features are extracted from video frames at each camera's perspective. Let I... ij For the j-th frame image from the i-th camera's perspective, feature extraction is performed: Where I represents a video frame image, F represents the feature extraction model, A represents the attention mechanism, ω1 and ω2 are weight parameters, and M and N are the dimensions of the image, (μ x ,μ y ) is the center coordinate of the image, σ x and σ y These are parameters that control the Gaussian distribution, Convolution(I ij () indicates that a convolution operation is performed on the image to extract features.

[0209] Optionally, the aforementioned features should contain sufficient information to distinguish different objects and regions in the scene. An attention mechanism is then applied to the extracted feature maps to dynamically identify key feature points between different camera viewpoints. Among them, K ij Let L represent the set of key feature points of the j-th frame image from the i-th camera viewpoint after passing through the attention mechanism, where L is the number of features in the feature map.

[0210] Optionally, the aforementioned key points are typically prominent objects, edges, or corners in the scene. Using a feature matching algorithm, key points from different viewpoints are matched to establish their correspondence. Each camera viewpoint is considered a node in the graph, and the matched key point pairs are considered edges between nodes. Based on the matching degree and positional relationship of the key points, weights are assigned to each edge to construct a viewpoint relationship graph. The lighting conditions and occlusion features of the current scene are analyzed, and the importance or credibility of each camera viewpoint in the current scene is calculated using a model. The fusion weights between the viewpoints are dynamically adjusted, and the calculated fusion weights are used to perform pixel-level fusion of video frames from each camera viewpoint.

[0211] Optionally, there are n camera viewpoints, and the pixel value of the video frame of the i-th camera viewpoint at coordinates (x, y) is P. i (x,y), the calculated fusion weight for this viewpoint is w. i (satisfies ∑) i =1 n w i =1), Basic fusion calculation: Perform weighted summation to obtain the preliminary fused pixel value F(x,y): To maintain spatiotemporal consistency between video frames, let t represent time. For pixels at the same position in two consecutive frames, define a spatiotemporal consistency factor λ(x,y,t). Assuming the pixel value after fusion in the previous frame is F(x,y,t-1), the adjusted fused pixel value is: F(x,y,t)=λ(x,y,t)F(x,y,t)+(1-λ(x,y,t))F(x,y,t-1). This avoids flickering, blurring, or misalignment. After fusion processing, a high-quality video stream containing complementary information from multiple perspectives is generated.

[0212] Optionally, a depth estimation network suitable for indoor scenes is selected, and depth estimation is performed using the fused video stream. Since the video stream is continuous, temporal information can be used to enhance the accuracy of depth estimation. The depth estimation network processes the input frames and outputs the corresponding depth map, where each pixel value represents the depth of the corresponding point in the scene. The predicted depth map is post-processed to remove noise, fill holes, and smooth edges to improve the quality of the depth map. The intrinsic and extrinsic parameters of each camera are obtained, which are necessary for 3D reconstruction. Using the camera intrinsic parameters and the depth map, each pixel in the depth map is converted into a point cloud in 3D space. If multiple camera views are used, the point clouds from different views need to be fused, and duplicate or conflicting points are removed. The fused point cloud is optimized to remove noise, smooth the surface, and fill holes. The optimized point cloud is converted into a surface mesh to construct a 3D scene model. The texture information in the original video frames is mapped onto the 3D scene model to enhance its realism.

[0213] Optionally, evaluation criteria should be designed based on application requirements, including reconstruction quality and computational efficiency. A performance monitoring module should be implemented to evaluate the reconstruction results of the current system in real time or periodically, compare them with preset standards, analyze the differences between the reconstruction results and the preset standards, identify performance bottlenecks or deficiencies, design adaptive adjustment strategies, including adjusting the attention mechanism, optimizing graph neural network parameters, and adjusting the training strategy of the depth estimation network. The above adjustment strategies should be applied to the system, and the performance changes after adjustment should be monitored. An intuitive and easy-to-use user interface should be designed to allow users to view reconstruction results, evaluate system performance, and provide feedback. User satisfaction, suggestions, and improvement opinions on the reconstruction results should be collected through the user interface.

[0214] In this embodiment of the invention, if it is necessary to construct a 3D model corresponding to a target scene, video acquisition devices deployed at multiple perspectives of the target scene can be used to acquire video data of the target scene, obtaining multiple video data corresponding to multiple perspectives. The video data can be analyzed to obtain corresponding feature data and the correlation between feature data. The video data is then fused using these correlations to obtain a fusion result. Depth information between the target scene and the video acquisition devices is extracted from the fusion result, and an initial 3D model is constructed using this depth information. The constructed initial 3D model is then evaluated to obtain an evaluation result. The initial 3D model whose evaluation result meets the evaluation result threshold is determined as the target 3D model of the target scene. In this embodiment, the multi-view video data fusion technology can dynamically identify the correlation between feature datasets of multiple perspectives. Therefore, under specific conditions (e.g., a perspective is occluded or there is insufficient lighting), the proportion of video data from that perspective in the fusion result can be automatically reduced, achieving the goal of improving the reliability and stability of the fusion result. This, in turn, improves the accuracy of the 3D model construction of the scene, solving the technical problem of low accuracy in 3D model construction of the scene.

[0215] Example 3

[0216] This invention provides a three-dimensional model construction device for a scene. It should be noted that the document generation device of this invention can be used to execute... Figure 1 The present invention provides a three-dimensional model construction device for a scene. The following describes the three-dimensional model construction device for a scene provided in the embodiments of the present invention.

[0217] Figure 3 This is a schematic diagram of a three-dimensional model construction device for a scene according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include: acquisition unit 302, fusion unit 304, construction unit 306, first determination unit 308 and second determination unit 310.

[0218] The acquisition unit 302 is used to acquire video data of the target scene using video acquisition devices deployed at multiple viewpoints, and obtain multiple video data.

[0219] The fusion unit 304 is used to acquire multiple feature data from multiple video data, and to fuse the multiple video data based on the correlation between the multiple feature data to obtain the fusion result.

[0220] The construction unit 306 is used to extract the depth information of the target scene from the fusion result and construct an initial three-dimensional model of the target scene based on the depth information. The depth information is used to represent the distance between the target scene and the video acquisition device.

[0221] The first determining unit 308 is used to determine the evaluation result of the initial three-dimensional model.

[0222] The second determining unit 310 is used to determine the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene.

[0223] The three-dimensional model construction device for a scene provided in this embodiment of the invention uses a collection unit 302 to collect video data of a target scene using video acquisition devices deployed at multiple viewpoints, obtaining multiple video data. A fusion unit 304 acquires multiple feature data from the multiple video data and fuses the multiple video data based on the correlation between the multiple feature data to obtain a fusion result. A construction unit 306 extracts depth information of the target scene from the fusion result and constructs an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to represent the distance between the target scene and the video acquisition devices. A first determination unit 308 determines the evaluation result of the initial three-dimensional model. A second determination unit 310 determines the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene, thereby solving the technical problem of low accuracy in the construction of the three-dimensional model of the scene and achieving the technical effect of improving the accuracy of the construction of the three-dimensional model of the scene.

[0224] The three-dimensional model building device for the above-mentioned scenario may also include a processor and a memory. All of the above-mentioned units are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0225] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, graceful shutdown of devices of the same type awaiting shutdown can be controlled.

[0226] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory, and the memory includes at least one memory chip.

[0227] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the efficiency of traders.

[0228] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0229] Example 4

[0230] According to an embodiment of the present invention, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, performs the construction of a three-dimensional model of a scene.

[0231] Example 5

[0232] According to an embodiment of the present invention, a processor is also provided, which is used to run a program, wherein the program performs the construction of a three-dimensional model of a scene during runtime.

[0233] Example 6

[0234] Figure 4 This is a schematic diagram of an electronic device for a method of constructing a three-dimensional model of a scene according to an embodiment of the present invention, as shown below. Figure 4 As shown in the embodiment of the present invention, an electronic device is also provided. The device includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the above-mentioned steps.

[0235] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0236] The present invention also provides a computer program product that, when executed on a data processing device, is adapted to execute a program that initializes the above-described method steps.

[0237] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0238] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0239] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0240] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0241] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0242] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.

[0243] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0244] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0245] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0246] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for constructing a 3D model of a scene, characterized in that, include: Video data of the target scene is collected using video acquisition devices deployed at multiple viewpoints to obtain multiple sets of video data; Multiple feature data are obtained from multiple video data sets, and the multiple video data sets are fused based on the correlation between the multiple feature data sets to obtain a fusion result; From the fusion result, the depth information of the target scene is extracted, and based on the depth information, an initial three-dimensional model of the target scene is constructed, wherein the depth information is used to represent the distance between the target scene and the video acquisition device; Determine the evaluation results of the initial 3D model; The initial 3D model whose evaluation results meet the evaluation result threshold is determined as the target 3D model of the target scene; The association is obtained as follows: Based on multiple feature data from different perspectives, the feature similarity between the multiple feature data is determined; based on the coordinate positions corresponding to the multiple feature data from different perspectives, the positional relationship data between the multiple feature data is determined; based on the feature similarity and the positional relationship data, the matching weight is determined, where the matching weight represents the degree of association between the multiple feature data; in response to the matching weight exceeding a matching weight threshold, feature data pairs are selected from the multiple feature data, where each feature data pair includes two feature data; feature vectors corresponding to the two feature data in the feature data pair are obtained, and the inner product similarity and distance similarity between the two feature vectors are determined; the coordinate positions of the two feature data in the feature data pair in their respective perspectives are obtained, and the positional relationship weight between the two coordinate positions is determined; based on the inner product similarity, the distance similarity, and the positional relationship weight, the association is constructed.

2. The construction method according to claim 1, characterized in that, Obtaining multiple feature data from multiple video data sets, including: An attention mechanism is used to identify multiple feature data from multiple video data sets.

3. The construction method according to claim 2, characterized in that, Using an attention mechanism, multiple feature data are identified from multiple video data sets, including: Using a feature extraction model, features are extracted from video frames in multiple video data to obtain multiple feature maps; Using the attention mechanism, multiple feature data are identified from multiple feature maps.

4. The construction method according to claim 3, characterized in that, The feature extraction model includes a Gaussian model and a convolutional model. The feature map includes a first feature map and a second feature map. The Gaussian model corresponds to a first weight and the first feature map, and the convolutional model corresponds to a second weight and the second feature map. The feature extraction model is used to extract features from video frames in multiple video data sets to obtain multiple feature maps, including: The video frame is input into the Gaussian model, and the Gaussian model is used to extract features from the video frame to obtain the first feature map. The video frame is then input into the convolutional model, and the convolutional model is used to extract features from the video frame to obtain the second feature map. The first weight is assigned to the first feature map, and the second weight is assigned to the second feature map. The weighted sum is then used to obtain the feature map.

5. The construction method according to claim 1, characterized in that, Based on the correlation between multiple feature data, the multiple video data are fused to obtain a fusion result, including: Based on the aforementioned relationship, the fusion weights corresponding to the plurality of video data are determined respectively; Based on the fusion weights, multiple video data are fused to obtain the fusion result.

6. The construction method according to claim 5, characterized in that, Based on the aforementioned correlation, determining the fusion weights corresponding to the plurality of video data respectively includes: The time information of the video data is acquired, and a spatiotemporal consistency factor is set based on the time information, wherein the spatiotemporal consistency factor includes the time information and the coordinate position of the feature data; The fusion weight is determined based on the spatiotemporal consistency factor and the correlation relationship.

7. The construction method according to claim 1, characterized in that, Based on the depth information, an initial 3D model of the target scene is constructed, including: The depth information is converted into three-dimensional coordinates, and the initial three-dimensional model is constructed based on the three-dimensional coordinates.

8. The construction method according to claim 1, characterized in that, The initial 3D model whose evaluation result meets the evaluation result threshold is determined as the target 3D model of the target scene, including: In response to the evaluation result exceeding the evaluation result threshold, the initial 3D model is determined as the target 3D model; In response to the evaluation result not exceeding the evaluation result threshold, the initial 3D model is adjusted, and the adjusted initial 3D model is determined as the target 3D model, wherein the evaluation result of the adjusted initial 3D model exceeds the evaluation result threshold.

9. A three-dimensional model construction device for a scene, characterized in that, include: The acquisition unit is used to acquire video data of the target scene using video acquisition devices deployed at multiple viewpoints, and obtain multiple video data sets. The fusion unit is used to acquire multiple feature data from multiple video data, and to fuse the multiple video data based on the correlation between the multiple feature data to obtain a fusion result; A construction unit is used to extract depth information of the target scene from the fusion result and construct an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to represent the distance between the target scene and the video acquisition device; The first determining unit is used to determine the evaluation result of the initial three-dimensional model; The second determining unit is used to determine the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene. The fusion unit is further configured to: determine the feature similarity between multiple feature data based on multiple feature data from different viewpoints; determine the positional relationship data between multiple feature data based on the coordinate positions corresponding to the multiple feature data from different viewpoints; determine the matching weight of the multiple feature data based on the feature similarity and the positional relationship data, wherein the matching weight is used to represent the degree of association between the multiple feature data; in response to the matching weight exceeding a matching weight threshold, filter feature data pairs from the multiple feature data, wherein the feature data pair includes two feature data; obtain the feature vectors corresponding to the two feature data in the feature data pair, and determine the inner product similarity and the distance similarity between the two feature vectors; obtain the coordinate positions of the two feature data in the feature data pair in their respective viewpoints, and determine the positional relationship weight between the two coordinate positions; and construct the association based on the inner product similarity, the distance similarity, and the positional relationship weight.

10. A processor, characterized in that, The processor is used to run a program, wherein the program is executed by the processor to perform a method for constructing a three-dimensional model of the scene according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform a method for constructing a three-dimensional model of the scene as described in any one of claims 1 to 8.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a method for constructing a three-dimensional model of the scene according to any one of claims 1 to 8.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements a method for constructing a three-dimensional model of a scene according to any one of claims 1 to 8.

Images