Scene three-dimensional model construction method and device and electronic equipment

Abstract

The invention discloses a three-dimensional model construction method and device of a scene and electronic equipment. The method comprises the following steps of: acquiring video data of a target scene by using video acquisition equipment deployed in a plurality of visual angles to obtain a plurality of pieces of video data; acquiring multiple pieces of feature data in the multiple pieces of video data, and fusing the multiple pieces of video data based on an association relationship among the multiple pieces of feature data to obtain a fusion result; depth information of the target scene is extracted from the fusion result, an initial three-dimensional model of the target scene is constructed based on the depth information, and the depth information is used for representing the distance between the target scene and the video acquisition equipment; determining an evaluation result of the initial three-dimensional model; and determining the initial three-dimensional model of which the evaluation result meets an evaluation result threshold as a target three-dimensional model of the target scene. According to the invention, the technical problem of low accuracy of three-dimensional model construction of a scene is solved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a method, device and electronic equipment for constructing a three-dimensional model of a scene. Background Art

[0002] At present, three-dimensional (3D) reconstruction technology has important application value and significance in many fields such as intelligent monitoring, virtual reality, three-dimensional measurement, robot navigation, disaster assessment, and urban planning.

[0003] In the related art, three-dimensional reconstruction based on video data from multiple perspectives mainly relies on deep learning. However, the construction of the above-mentioned three-dimensional model relies on manually designed feature points, which makes the process time-consuming, and the quality of the three-dimensional model construction depends on manual operation, which is difficult to meet the needs of real-time applications. In addition, the three-dimensional model construction process relies on manually designed parameters, the parameter adjustment process takes a long time, and the quality of the model construction depends on manual operation, which is difficult to meet the needs of real-time applications. The three-dimensional model construction process is highly dependent on the performance of deep learning networks. The improvement of deep learning network performance depends on the accumulation of a large number of data labels, and the current data labels are time-consuming and labor-intensive, and the quality is low, resulting in a long three-dimensional reconstruction process and poor results. Therefore, there is still a technical problem of low accuracy in the construction of three-dimensional models of scenes.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide a method, device and electronic device for constructing a three-dimensional model of a scene, so as to at least solve the technical problem of low accuracy in constructing a three-dimensional model of a scene.

[0006] According to one aspect of an embodiment of the present invention, a method for constructing a three-dimensional model of a scene is provided, comprising: respectively using video acquisition devices deployed at multiple viewing angles to acquire video data of a target scene to obtain multiple video data; acquiring multiple feature data in the multiple video data, and fusing the multiple video data based on the correlation between the multiple feature data to obtain a fusion result; extracting depth information of the target scene from the fusion result, and constructing an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to indicate the distance between the target scene and the video acquisition device; determining an evaluation result of the initial three-dimensional model; and determining the initial three-dimensional model whose evaluation result meets an evaluation result threshold as the target three-dimensional model of the target scene.

[0007] Optionally, obtaining multiple feature data from multiple video data includes: using an attention mechanism to respectively identify multiple feature data from the multiple video data.

[0008] Optionally, an attention mechanism is used to separately identify multiple feature data in multiple video data, including: using a feature extraction model to extract features from video frames in multiple video data to obtain multiple feature maps; using an attention mechanism to identify multiple feature data from multiple feature maps.

[0009] Optionally, the feature extraction model includes a Gaussian model and a convolution model, the feature map includes a first feature map and a second feature map, the Gaussian model corresponds to a first weight and a first feature map, the convolution model corresponds to a second weight and a second feature map, wherein the feature extraction model is used to extract features from video frames in multiple video data to obtain multiple feature maps, including: inputting the video frame into the Gaussian model, extracting features from the video frame using the Gaussian model to obtain a first feature map, and inputting the video frame into the convolution model, extracting features from the video frame using the convolution model to obtain a second feature map; assigning a first weight to the first feature map, and assigning a second weight to the second feature map, and performing weighted summation to obtain the feature map.

[0010] Optionally, the method also includes: determining feature similarities between multiple feature data under different perspectives; determining positional relationship data between multiple feature data based on coordinate positions respectively corresponding to the multiple feature data under different perspectives; determining matching weights of multiple feature data based on feature similarities and positional relationship data, wherein the matching weight is used to represent the degree of association of the association relationship between multiple feature data; in response to the matching weight exceeding a matching weight threshold, screening out feature data pairs from multiple feature data, wherein the feature data pairs include two feature data; and constructing association relationships corresponding to the feature data pairs.

[0011] Optionally, constructing an association relationship corresponding to the feature data pair includes: obtaining feature vectors corresponding to two feature data in the feature data pair, and determining the inner product similarity between the two feature vectors, and the distance similarity between the two feature vectors; obtaining the coordinate positions of the two feature data in the feature data pair in their respective perspectives, and determining the positional relationship weight between the two coordinate positions; and constructing an association relationship based on the inner product similarity, the distance similarity and the positional relationship weight.

[0012] Optionally, based on the correlation between multiple feature data, multiple video data are fused to obtain a fusion result, including: based on the correlation, determining fusion weights corresponding to the multiple video data respectively; based on the fusion weights, fusing the multiple video data to obtain a fusion result.

[0013] Optionally, based on the association relationship, fusion weights corresponding to multiple video data are determined, including: obtaining time information of acquired video data, and setting a spatiotemporal consistency factor based on the time information, wherein the spatiotemporal consistency factor includes the time information and the coordinate position of the feature data; based on the spatiotemporal consistency factor and the association relationship, the fusion weight is determined.

[0014] Optionally, constructing an initial three-dimensional model of the target scene based on the depth information includes: converting the depth information into three-dimensional coordinates, and constructing the initial three-dimensional model based on the three-dimensional coordinates.

[0015] Optionally, an initial three-dimensional model whose evaluation result satisfies an evaluation result threshold is determined as a target three-dimensional model of the target scene, including: in response to the evaluation result exceeding the evaluation result threshold, determining the initial three-dimensional model as the target three-dimensional model; in response to the evaluation result not exceeding the evaluation result threshold, adjusting the initial three-dimensional model, and determining the adjusted initial three-dimensional model as the target three-dimensional model, wherein the evaluation result of the adjusted initial three-dimensional model exceeds the evaluation result threshold.

[0016] According to another aspect of an embodiment of the present invention, a device for constructing a three-dimensional model of a scene is also provided, including: an acquisition unit, used to respectively use video acquisition devices deployed at multiple viewing angles to acquire video data of a target scene to obtain multiple video data; a fusion unit, used to obtain multiple feature data in the multiple video data, and fuse the multiple video data based on the correlation between the multiple feature data to obtain a fusion result; a construction unit, used to extract depth information of the target scene from the fusion result, and construct an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to indicate the distance between the target scene and the video acquisition device; a first determination unit, used to determine an evaluation result of the initial three-dimensional model; and a second determination unit, used to determine the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene.

[0017] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing any one of the above-mentioned methods for constructing a three-dimensional model of a scene.

[0018] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the above-mentioned methods for constructing a three-dimensional model of the scene.

[0019] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, it implements any one of the above-mentioned methods for constructing a three-dimensional model of a scene.

[0020] In an embodiment of the present invention, if it is necessary to construct a three-dimensional model corresponding to a target scene, the video data of the target scene can be collected by using video acquisition devices respectively deployed at multiple viewing angles of the target scene to obtain multiple video data corresponding to multiple viewing angles. The video data can be analyzed to obtain corresponding feature data, and the correlation between the feature data can be analyzed. The video data is fused through the correlation to obtain a fusion result. The depth information between the target scene and the video acquisition device is extracted from the fusion result, and the initial three-dimensional model is constructed using the depth information. The constructed initial three-dimensional model is evaluated to obtain an evaluation result. The initial three-dimensional model whose evaluation result meets the evaluation result threshold is determined as the target three-dimensional model of the target scene. In this embodiment, the video data fusion technology under multiple viewing angles can dynamically identify the correlation between the feature data sets of multiple viewing angles, so that under specific conditions (for example, a certain viewing angle is blocked or the light is insufficient), the proportion of the video data under the viewing angle in the fusion result can also be automatically reduced, thereby achieving the purpose of improving the reliability and stability of the fusion result, thereby achieving the technical effect of improving the accuracy of the three-dimensional model construction of the scene, and solving the technical problem of low accuracy of the three-dimensional model construction of the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 is a flow chart of a method for constructing a three-dimensional model of a scene according to an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a three-dimensional scene reconstruction and ranging system based on multi-video fusion according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of the structure of a three-dimensional model building device for a scene according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of an electronic device for a method for constructing a three-dimensional model of a scene according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] According to an embodiment of the present invention, an embodiment of building a three-dimensional model of a scene is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 is a flow chart of a method for constructing a three-dimensional model of a scene according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0031] Step S102, respectively using video acquisition devices deployed at multiple viewing angles to acquire video data of a target scene to obtain multiple video data.

[0032] In the technical solution provided in the above step S102 of the embodiment of the present invention, the target scene may be a scene for which a three-dimensional model is to be constructed (three-dimensional scene reconstruction), which may also be referred to as a three-dimensional scene, and may be determined according to the application field and function. For example, for automatic drivers and robot navigation, the target scene may be roads and indoor spaces; for virtual reality and augmented display, the target scene may be historical sites, museums, natural landscapes, etc.; for architecture and real estate, the target scene may be a building model; for cultural relics protection and restoration, the target scene may be cultural relics, architectural sites, etc.

[0033] It should be noted that the above target scenes are only examples and are not specifically limited here. As long as the three-dimensional scene can be corresponding to the three-dimensional model construction method in the embodiment of the present invention, it is within the protection scope of the embodiment of the present invention.

[0034] Optionally, a video acquisition device can be deployed at a corresponding viewing angle around the target scene to acquire video data of the target scene. The video acquisition device can be a camera, such as a high-definition camera, an infrared camera, a 3D camera, etc. This is only an example, and there is no specific restriction on the video acquisition device. It can be adaptively selected according to the target scene to be three-dimensionally reconstructed. The video data can be a video stream acquired by the video acquisition device, which can also be called a multi-view video, for example, it can be an acquired high-definition video of the target scene.

[0035] In this embodiment, if the target scene needs to be reconstructed in three dimensions, video acquisition devices can be deployed at multiple viewing angles that can capture a panoramic view of the target scene. The video acquisition devices can be controlled to obtain video data of the target scene at corresponding viewing angles.

[0036] Optionally, select appropriate video acquisition equipment according to the characteristics of the target scene and reconstruction requirements. Determine the installation position and angle of the video acquisition equipment to ensure that all important areas of the entire target scene are covered and blind spots and occlusions are reduced. View planning can take into account the geometric characteristics, lighting conditions, and dynamic elements of the target scene.

[0037] Optionally, in addition, when using video acquisition devices under multiple viewing angles to capture the target scene, the time consistency of the video data respectively captured by the multiple video acquisition devices can also be considered. Therefore, in order to ensure the time consistency of the video data under multiple viewing angles, a hardware synchronization signal or a software algorithm can be used to ensure that the video acquisition devices for the same target scene can start and stop video recording at the same time, thereby reducing the error caused by the time difference.

[0038] Optionally, in addition, the resolution, frame rate, exposure, white balance and other parameters of the image acquisition device may be configured according to lighting conditions and environmental changes to adapt to the diversity of target scenes and ensure the quality of video data.

[0039] Optionally, the video acquisition device captures the video stream of the target scene in real time, ensuring the details and clarity of the reconstruction result by capturing high-definition video. The acquired video data needs to be transmitted to the processing unit in real time, which may be done through a wired or wireless network. The data transmission process should ensure the integrity of the video stream, avoid delays and packet loss, and maintain data continuity and synchronization.

[0040] In the embodiment of the present invention, the video data is collected from multiple perspectives, which can ensure the comprehensiveness of the collection of various details in the target scene and the complementarity between different details in the video data under multiple perspectives. Video data from different perspectives can provide multi-angle details of the target scene, which is crucial for the accuracy of three-dimensional reconstruction. The video data collected in real time forms a continuous video stream, which is convenient for capturing the dynamic changes of the target scene in the time dimension, which is particularly important for processing dynamic scenes such as moving objects and lighting changes.

[0041] Step S104, obtaining multiple feature data from multiple video data, and fusing the multiple video data based on the association relationship between the multiple feature data to obtain a fusion result.

[0042] In the technical solution provided in the above step S104 of the embodiment of the present invention, the feature data can be used to represent the key local feature area in the video data, which can also be called key feature points (feature points / key points), which can be areas with complex textures or patterns in the target scene, such as carpet patterns, decorations on walls, etc., or physical edges and contours in the target scene, such as vehicle contours, tree branches, or significant signs in the target scene, such as billboards, road signs, etc. It should be noted that the above feature data is only for example, and no specific restrictions are made here, and can be determined according to the actual target scene to be three-dimensionally reconstructed. The association relationship can be a corresponding relationship between key feature points, which can be used to represent the association between multiple feature data in time, space, texture or structure. For example, the spatial association relationship can be used to represent the position relationship of the same feature point in three-dimensional space under different viewing angles, and the temporal association relationship can be used to represent the position change of the same feature point at different time points (continuous video frames). The texture association relationship can be used to represent the color or texture similarity based on the local area of ​​the feature point. The structural association relationship can be used to represent the relative position and form based on the feature point in the scene. This is only an example and is not a specific limitation. The fusion result can be used to represent a video obtained by fusing video data from multiple perspectives, for example, it can be used to generate a fused high-quality video stream by fusing multiple perspective videos.

[0043] In this embodiment, after video data of the target scene are collected by using video acquisition devices deployed at multiple viewing angles, multiple feature data can be obtained from the multiple video data. The association relationship between the multiple feature data is analyzed, and the multiple video data corresponding to the multiple viewing angles are fused according to the association relationship to obtain a fusion result.

[0044] Optionally, key feature points may be extracted from the video data at each viewing angle. In order to establish a corresponding relationship between feature data at different viewing angles, feature matching may be performed.

[0045] Optionally, in the process of feature matching, whether there is an association relationship between feature data can be determined by determining the similarity between feature data under different viewing angles. For example, if the similarity is high, there is an association relationship. By the above method, an association relationship between key feature points can be established.

[0046] Optionally, based on the established association relationship between key feature points, the video data from multiple perspectives are fused. For example, corresponding weights can be assigned to the video data from different perspectives according to the association relationship, so as to perform video data fusion by a weighted method. Feature points with larger weights have a greater impact on the fusion result. It should be noted that the above process and method for fusing video data from different perspectives are only for illustration and are not specifically limited here.

[0047] Optionally, in addition, in order to maintain the temporal and spatial consistency between video frames in the video data, the correlation relationship between the previous and next video frames may also be considered to ensure the temporal continuity and spatial stability of the fusion result.

[0048] In an embodiment of the present invention, through the above method, a fusion result that integrates feature data from multiple perspectives can be generated. The fusion result not only includes a comprehensive perspective of the target scene, but also effectively reduces the errors and information loss that may be caused by a single perspective by dynamically identifying and matching key feature points. The fusion result will then be used for subsequent three-dimensional reconstruction to provide more accurate and complete visual information for three-dimensional reconstruction. The above method is the core of real-time three-dimensional scene understanding, which enables accurate identification and tracking of key feature points in the target scene in a dynamic environment, which is very important for many advanced visual tasks (such as autonomous driving, robot navigation, augmented reality, etc.).

[0049] Step S106: extracting the depth information of the target scene from the fusion result, and constructing an initial three-dimensional model of the target scene based on the depth information.

[0050] In the technical solution provided in step S106 of the embodiment of the present invention, the depth information can be used to indicate the distance between the target scene and the video acquisition device, which can be the distance between the target scene and the video acquisition device, or can be called a depth value. The initial 3D model can be a 3D scene model, or can be called a reconstruction result (3D reconstruction result).

[0051] In this embodiment, after a plurality of video data from a plurality of perspectives are fused based on the correlation between the feature data and a fusion result is obtained, the depth information of the target scene can be extracted from the fusion result, and an initial three-dimensional model of the target scene can be constructed based on the depth information.

[0052] Optionally, depth information is extracted from the fusion result of the multi-view video streams, and an initial three-dimensional model of the target scene is constructed based on the depth information.

[0053] Optionally, the extraction of depth information can be based on a depth estimation method, in which a depth estimation network (e.g., a deep learning model) is used to predict a depth map from each video frame of the fused result in the fused video stream (i.e., the fused result). The depth map is a two-dimensional image in which the value of each pixel represents the depth of a feature point in the corresponding target scene, i.e., the distance between the feature point and the video acquisition device.

[0054] Optionally, after extracting the depth information, the principle of multi-view geometry can be applied to construct a 3D model. The depth maps at different viewing angles and the parameters of the video acquisition device can be used to convert the 2D pixel coordinates in the depth map into 3D space coordinates, thereby constructing an initial 3D model.

[0055] In the embodiment of the present invention, depth information can be extracted from the fused video stream to construct an initial 3D model of the target scene, providing a preliminary framework of the 3D structure of the target scene for subsequent model optimization using the evaluation results. Although the initial 3D model is initial, it already contains the spatial layout and depth information of the target scene, and is the basis for realizing advanced functions such as 3D scene understanding, object positioning, and scene ranging.

[0056] Step S108, determining the evaluation result of the initial three-dimensional model.

[0057] In the technical solution provided in the above step S108 of the embodiment of the present invention, the evaluation result can be used to indicate whether the initial three-dimensional model meets the requirements (preset standards / evaluation standards), and can also be used to indicate the satisfaction of relevant personnel with the initial three-dimensional model, and to determine whether the initial three-dimensional model needs to be adjusted.

[0058] In this embodiment, after constructing the initial three-dimensional model of the target scene based on the depth information, an evaluation result of the initial three-dimensional model may be determined.

[0059] Optionally, in an embodiment of the present invention, the above process is a stage of evaluating the initial three-dimensional model and optimizing the initial three-dimensional model using the evaluation results, the purpose of which is to ensure that the reconstructed three-dimensional model meets preset performance standards and application requirements, that is, to ensure that the final three-dimensional model meets the requirements.

[0060] Optionally, before evaluating the initial 3D model, a set of evaluation criteria may be established, which may be set based on application requirements and may include, but are not limited to, reconstruction accuracy, model details, computational efficiency, integrity and consistency of the 3D model, etc. For example, in autonomous driving applications, the accuracy of the 3D model (especially the precise location of roads and obstacles) is crucial; while in virtual reality or augmented reality applications, the details and texture quality of the 3D model may be the key indicators of concern.

[0061] It should be noted that the above-mentioned evaluation criteria are only for illustration and are not specifically limited here. As long as the content and process of formulating the evaluation criteria can be based on the requirements of the accuracy of the three-dimensional model in different target scenarios, they are within the scope of protection of the embodiments of the present invention.

[0062] Optionally, the quality of the constructed initial 3D model can be evaluated according to the evaluation criteria established by the above method. The evaluation can involve multiple aspects, including the geometric matching between the 3D model and the actual target scene, the smoothness and integrity of the model surface, the level of detail of the 3D model and the accuracy of texture mapping. The evaluation process can be automated, using pre-set algorithms and indicators, or semi-automated, combined with visual inspection and correction suggestions from relevant personnel.

[0063] It should be noted that the above-mentioned process and method of evaluating the initial three-dimensional model using the evaluation criteria are only for illustration and are not specifically limited here. As long as the initial three-dimensional model can be evaluated and the target scene can be accurately reconstructed in three dimensions, it is within the protection scope of the embodiments of the present invention.

[0064] Step S110: determining the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene.

[0065] In the technical solution provided in the above step S110 of the embodiment of the present invention, the evaluation result threshold may be a satisfaction threshold, which may be determined according to the requirements of the target scene to be three-dimensionally reconstructed in actual situations, or may be pre-set, for example, it may be pre-set to 98%. It should be noted that the setting method and value of the above evaluation result threshold are only for illustrative purposes and are not specifically limited here. As long as the process and method can be used to measure whether the initial three-dimensional model meets the requirements, they are within the protection scope of the embodiment of the present invention.

[0066] In this embodiment, after determining the evaluation result of the initial three-dimensional model, it can be determined whether the evaluation result meets the evaluation result threshold. If the evaluation result meets the evaluation result threshold, it can be shown that the relevant personnel are relatively satisfied with the initial three-dimensional model, and the corresponding initial three-dimensional model can be determined as the final target three-dimensional model. Otherwise, it can be shown that the relevant personnel are relatively dissatisfied with the initial three-dimensional model, and it is necessary to adjust the initial three-dimensional model, or re-determine the target three-dimensional model that satisfies the relevant personnel.

[0067] Optionally, based on the evaluation results, it can be determined whether the initial three-dimensional model needs to be adjusted. If the evaluation result of the initial three-dimensional model meets the evaluation result threshold, that is, the initial three-dimensional model meets the evaluation criteria (preset criteria), it can directly enter the subsequent application or perform slight optimization. If the evaluation result of the initial three-dimensional model does not meet the evaluation result threshold, that is, the initial three-dimensional model does not meet the preset criteria, the specific reasons can be analyzed (such as large depth estimation error, inaccurate feature point matching, fusion problems, etc.), and corresponding adjustment strategies can be set to adjust the initial three-dimensional model.

[0068] In the embodiment of the present invention, determining the evaluation results and determining the target 3D model according to the evaluation results is an important step in the 3D scene reconstruction process, which can not only ensure the quality of the final target 3D model, but also provide guidance for subsequent target 3D model optimization and 3D reconstruction improvement. The optimized target 3D model will be more accurate and more reliable, thus providing a solid foundation for using the final target 3D model for distance measurement, navigation, object recognition, etc.

[0069] In step S102 to step S110 of the embodiment of the present invention, if it is necessary to construct a three-dimensional model corresponding to the target scene, the video data of the target scene can be collected by using video acquisition devices respectively deployed at multiple viewing angles of the target scene to obtain multiple video data corresponding to multiple viewing angles. The video data can be analyzed to obtain corresponding feature data, and the correlation relationship between the feature data can be analyzed. The video data is fused through the correlation relationship to obtain a fusion result. The depth information between the target scene and the video acquisition device is extracted from the fusion result, and the initial three-dimensional model is constructed using the depth information. The constructed initial three-dimensional model is evaluated to obtain an evaluation result. The initial three-dimensional model whose evaluation result meets the evaluation result threshold is determined as the target three-dimensional model of the target scene. In this embodiment, the video data fusion technology under multiple perspectives is adopted, which can dynamically identify the correlation between the feature data sets of multiple perspectives. Therefore, under certain conditions (for example, a certain perspective is blocked or the light is insufficient), the proportion of video data under this perspective in the fusion result can be automatically reduced, thereby achieving the purpose of improving the reliability and stability of the fusion result, and further achieving the technical effect of improving the accuracy of building a three-dimensional model of the scene, and solving the technical problem of low accuracy in building a three-dimensional model of the scene.

[0070] The embodiment of the present invention is described in detail below in combination with the above steps.

[0071] As an optional embodiment, step S104, obtaining multiple feature data from multiple video data, includes: using an attention mechanism to separately identify multiple feature data from multiple video data.

[0072] In this embodiment, in the process of obtaining multiple feature data from multiple video data, the attention mechanism can be used to identify the multiple video data to obtain multiple feature data in each video data.

[0073] Optionally, in order to process multi-view video data more efficiently and intelligently and ensure that the feature data extracted from each view is a representative and informative local area in the target scene, an attention mechanism can be used to identify and extract more critical feature data from multiple video data.

[0074] Optionally, the attention mechanism is a computational model that mimics human visual attention selection, which can focus on certain parts (i.e., local areas with representativeness and information content) in the input data (i.e., multiple video data input to the computational model), while ignoring other unimportant or redundant parts. In the multi-view video fusion scenario, the attention mechanism can dynamically identify and focus on the feature points with recognition and information value in each video data, thereby improving the accuracy of feature extraction and the efficiency of the subsequent fusion process.

[0075] In an embodiment of the present invention, through the above method, feature data can be extracted from multi-view video streams more intelligently, and the feature data will be used for video fusion and three-dimensional reconstruction. The introduction of the attention mechanism not only improves the efficiency of feature extraction, but also enhances the system adaptability and robustness of the three-dimensional model construction method for processing scenes, so that the system can be more accurate and stable when processing complex and dynamic scenes. The above processing method is crucial to achieving high-quality video fusion and three-dimensional scene reconstruction, especially in real-time applications such as autonomous driving, robot navigation, augmented reality, etc., it can provide a more stable and reliable visual information foundation.

[0076] As an optional embodiment, an attention mechanism is used to separately identify multiple feature data in multiple video data, including: using a feature extraction model to extract features from video frames in multiple video data to obtain multiple feature maps; using an attention mechanism to identify multiple feature data from multiple feature maps.

[0077] In this embodiment, in the process of using the attention mechanism to respectively identify multiple feature data in multiple video data, a feature extraction model can be used to extract features from video frames in the multiple video data to obtain multiple feature maps. The attention mechanism can be used to identify corresponding feature data from the feature maps. The video frame can be each frame image in the video data.

[0078] Optionally, the process of processing video frames in multiple video data using an attention mechanism and a feature extraction model to identify and extract key feature data can be divided into two steps: feature extraction and feature recognition.

[0079] Optionally, a video frame is obtained from each video data. A video frame can be each frame of the video data. For real-time and efficiency, the video data is usually preprocessed, such as decoding, denoising, color correction, etc. A feature extraction model is selected, such as a convolutional neural network model based on deep learning. The above model is trained to automatically identify and extract representative features in the image, such as the edges, corners, textures, etc. of the object.

[0080] Optionally, for each video frame, a feature extraction model is applied, through which local features in the video frame can be identified and one or more feature maps can be generated. The feature map can be understood as an abstract representation of the original image, where each pixel or unit corresponds to the importance or descriptor of the feature, thereby retaining spatial location information and feature information.

[0081] Optionally, the attention mechanism is a computational method that focuses more on the important parts of the input data. In this embodiment, the attention mechanism is used to identify and focus on the most important feature data from the feature map, that is, those feature points that are valuable for subsequent processing (such as video fusion, three-dimensional reconstruction). The attention mechanism analyzes each feature point in the feature map, evaluates its importance and relevance in the current scene, and dynamically selects those significant feature points. These feature points may be object boundaries, areas with complex textures, key visual landmarks, etc. in the scene, which can be stably identified in video frames from different perspectives. The attention mechanism not only identifies key feature points, but also assigns a weight to each feature point. This weight reflects the importance of the feature point. Feature points with higher weights will be given greater attention and role in subsequent video fusion and three-dimensional reconstruction processes. According to the results of attention mechanism recognition and weight assignment, the highlighted feature data will be extracted from the feature map. The above data includes feature descriptors, location information, size, direction, etc. of feature points, which are the basis for subsequent processing.

[0082] As an optional embodiment, the feature extraction model includes a Gaussian model and a convolution model, the feature map includes a first feature map and a second feature map, the Gaussian model corresponds to a first weight and a first feature map, and the convolution model corresponds to a second weight and a second feature map, wherein the feature extraction model is used to extract features from video frames in multiple video data to obtain multiple feature maps, including: inputting the video frame into the Gaussian model, extracting features from the video frame using the Gaussian model to obtain a first feature map, and inputting the video frame into the convolution model, extracting features from the video frame using the convolution model to obtain a second feature map; assigning a first weight to the first feature map, and assigning a second weight to the second feature map, and performing weighted summation to obtain the feature map.

[0083] In this embodiment, in the process of extracting features from video frames in a plurality of video data using a feature extraction model, the video frames can be input into a Gaussian model in the feature extraction model, and the Gaussian model is used to extract features from the video frames, thereby obtaining a corresponding first feature map. The video frames can also be input into a convolution model in the feature extraction model, and the convolution model is used to extract features from the video frames to obtain a second feature map. The first feature map and the second feature map can be assigned corresponding first weights and second weights, respectively, and the final feature map can be obtained by weighted summation. Among them, the feature extraction model can include a Gaussian model and a convolution model. The feature map can include a first feature map and a second feature map. The first feature map can be a feature map extracted using a Gaussian model. The second feature map can be a feature map extracted using a convolution model. The first weight can be a weight assigned to the first feature map. The second weight can be a weight assigned to the second feature map.

[0084] Optionally, a feature extraction model is used to extract features from the video frames of each camera's perspective. The features should contain sufficient information to distinguish different objects and areas in the scene. An attention mechanism is applied to the extracted feature map to dynamically identify key feature points between the camera perspectives. The key feature points are usually significant objects, edges, and corners in the scene, which can be identified in multiple perspectives.

[0085] Optionally, feature extraction is performed on video frames (video frame images) at various viewing angles using a feature extraction model as shown below to obtain a feature map:

[0086]

[0087] Among them, F can be used to represent the feature extraction model; I ij Represents the j-th frame image from the i-th camera perspective; Can be used to represent Gaussian models; M and N are used to represent the dimensions of video frames; (μ x ,μ y ) can be used to represent the center coordinates of the video frame; σ x and σ y It can be used to represent the parameters of the Gaussian distribution corresponding to the control Gaussian model; ω2·Convolution(I ij ) can be used to represent a convolution model for performing a convolution operation on a video frame to extract features; ω1 can be used to represent a first weight; ω2 can be used to represent a second weight; F ij It can be used to represent the feature map obtained by the final weighted summation.

[0088] Optionally, after determining the above-mentioned feature map, the feature map can be input into an attention mechanism, and the attention mechanism is used to identify feature data from the feature map.

[0089] Optionally, feature data can be determined through an attention mechanism as shown below:

[0090]

[0091] Among them, K ij It can be used to represent the set of key feature points (feature data) of the j-th frame image of the i-th camera perspective after the attention mechanism; A can be used to represent the attention mechanism; L can be used to represent the number of feature data in the feature map.

[0092] As an optional embodiment, the method also includes: determining feature similarities between multiple feature data under different perspectives; determining positional relationship data between multiple feature data based on coordinate positions corresponding to the multiple feature data under different perspectives; determining matching weights of multiple feature data based on feature similarities and positional relationship data, wherein the matching weight is used to represent the degree of association of the association relationship between multiple feature data; in response to the matching weight exceeding the matching weight threshold, screening out feature data pairs from multiple feature data, wherein the feature data pairs include two feature data; and constructing association relationships corresponding to the feature data pairs.

[0093] In this embodiment, the feature similarity between multiple feature data can be determined; the position relationship data between the feature data can also be determined based on multiple coordinate positions corresponding to different feature numbers. According to the feature similarity and the position relationship data, the matching weights of the multiple feature data can be determined, and it can be judged whether the matching weight exceeds the matching weight threshold. If the matching weight exceeds the matching weight threshold, the feature data pair can be screened out from the multiple feature data, and the association relationship between the two feature data in the feature data pair can be constructed. Among them, the matching weight can be used to represent the degree of association of the association relationship between multiple feature data. The feature data pair can include two feature data with an association relationship. The position relationship data can also be called the position relationship weight.

[0094] Optionally, not only features are extracted from the multi-view video data, but also the relationship between the features can be further analyzed to determine which feature points are associated with each other, thereby constructing an association relationship between two feature data in a feature data pair.

[0095] Optionally, for multiple feature data extracted from different perspectives, each feature data contains a descriptor that quantifies the visual appearance of the feature point, such as color, texture, shape, etc. The descriptors of the feature data under different perspectives are compared to calculate the feature similarity between them. For example, the similarity measurement methods that can be used include but are not limited to Euclidean distance, cosine similarity, Mahalanobis distance, etc. If the feature descriptor exists in the form of a vector, the dot product or cosine similarity between the vectors corresponding to the two feature data can be calculated.

[0096] Optionally, during the feature extraction process, each feature data also contains its coordinate position in the original video frame, and this position information is crucial for understanding the relationship between feature points in three-dimensional space. By analyzing the coordinate positions of feature points under different viewing angles, the relative positional relationship between them is calculated, which includes information such as direction and distance. The calculation of the positional relationship can be achieved through simple geometric operations or more complex models, with the purpose of evaluating whether the feature points are in the same or similar positions under different viewing angles. The feature similarity and positional relationship data are combined together, and the matching weight between feature data is calculated through a certain weighted formula. The matching weight reflects the strength of the association between feature points and is an important basis for subsequent feature matching and video fusion processes. In order to ensure that only those matches with high confidence are selected, the system will set a matching weight threshold. Only when the matching weight exceeds this threshold, the two feature data will be considered to be matched. By comparing the matching weights between all feature data, those feature data pairs that exceed the threshold are screened out, and the feature points in the feature data pairs are considered to have consistent or similar corresponding relationships under different viewing angles.

[0097] Optionally, key points from different perspectives may be matched using a feature matching algorithm.

[0098] For example, feature similarity can be determined by the following formula:

[0099]

[0100] Among them, k ip ,k jq Can be used to represent two feature data in a feature data pair; S(k ip ,k jq ) can be used to represent the feature similarity between two feature data in a feature data pair; α and β can be adjustable parameters, with a value range between 0 and 1; and They can represent the vectors corresponding to the two feature data respectively.

[0101] For another example, let P(k ip )=(x ip ,y ip ) and P(k jq )=(x jq ,y jq ) are the coordinates of the key points in their respective view images, and the positional relationship weight can be determined by the following formula:

[0102]

[0103] Among them, W pos (k ip ,k jq) can be used to represent the weight of the positional relationship between two feature data; γ can be used to represent the adjustment parameter of the positional relationship; x ip and x jq Can be used to represent the horizontal coordinates of two feature data; y ip and jq Can be used to represent the vertical coordinate of two feature data.

[0104] As an optional example, the matching weight may be determined by the following formula:

[0105] M(k ip ,k jq )=λ×S(k ip ,k jq )+(1-λ)×W pos (k ip ,k jq )

[0106] Among them, M(k ip ,k jq ) can be used to represent the matching weight between two feature data; λ can be used to represent the parameter that balances feature similarity and position relationship weight. ip ,k jq ) exceeds the set threshold (matching weight threshold), it can be said that the two feature data (k ip and k jq ) is the key point of the match.

[0107] As an optional embodiment, constructing an association relationship corresponding to a feature data pair includes: obtaining feature vectors corresponding to two feature data in the feature data pair, and determining the inner product similarity between the two feature vectors, and the distance similarity between the two feature vectors; obtaining the coordinate positions of the two feature data in the feature data pair in their respective perspectives, and determining the positional relationship weight between the two coordinate positions; and constructing an association relationship based on the inner product similarity, the distance similarity and the positional relationship weight.

[0108] In this embodiment, in the process of constructing the association relationship corresponding to the feature data, the feature vectors corresponding to the two feature data in the feature data pair can be determined, and the inner product similarity between the two feature vectors and the distance similarity between the two feature vectors can be determined. The coordinate positions of the two feature data in the feature data pair in their respective perspectives can be obtained, and the corresponding position relationship weights can be determined. The association relationship is constructed based on the inner product similarity, the distance similarity and the position relationship weight.

[0109] Optionally, constructing the association relationship corresponding to the feature data is achieved by analyzing and synthesizing the inner product similarity, distance similarity and position relationship weight of the feature vectors. This process ensures the accuracy of feature point matching under different perspectives, thereby providing a solid foundation for subsequent three-dimensional scene reconstruction and ranging.

[0110] Optionally, a feature descriptor is extracted from each feature data, which is a vector that quantifies the visual properties of the feature point. Feature descriptors are usually output by feature extraction models and contain information such as color, texture, shape, etc. of the feature point. Each feature data corresponds to a feature vector, which represents the properties of the feature point. The dimension and element value of the vector depend on the design and training of the feature extraction model.

[0111] Optionally, for two feature data in a feature data pair, the system calculates the inner product similarity between their feature vectors. The inner product similarity is based on the measure of vector direction consistency and reflects the similarity of the two feature points in visual attributes.

[0112] For example, the inner product similarity S can be calculated by the following formula: f :

[0113]

[0114] Among them, S f (k1, k2) can be used to represent the inner product similarity between k1 and k2; k1 and k2 can represent two key points respectively; Can be used to represent the eigenvector of k1; Can be used to represent the eigenvector of k2.

[0115] Optionally, the Mahalanobis distance similarity S of the feature vector can also be calculated by the following formula: m :

[0116]

[0117] Among them, S m (k1, k2) can be used to represent the Mahalanobis distance similarity between k1 and k2; ∑ can be used to represent the covariance matrix of the feature vector.

[0118] Optionally, let (x1, y1) and (x2, y2) be the coordinates of k1 and k2 in their respective view images, and the positional relationship weight W can be determined by the following formula: p :

[0119] W p (k1,k2)=exp(-θ((x1-x2) 2 +(y1-y2) 2 ))

[0120] Among them, W p (k1, k2) can be used to represent the positional relationship weight of k1 and k2; θ can be used to represent the adjustment parameter.

[0121] Optionally, by combining feature similarity and position relationship, the edge weight w can be constructed as follows:

[0122]

[0123] Among them, w(k1, k2) can be used to represent the edge weight between k1 and k2; ω1 and ω2 are used to represent weight distribution parameters respectively, and ω1+ω2=1.

[0124] Optionally, match the key points in different perspectives and establish the corresponding relationship between them. Consider each camera perspective as a node in the graph, and consider the matched key point pairs as edges between nodes. According to the matching degree and position relationship of the key points, assign weights to each edge to construct the perspective relationship graph. Assign weights to each edge to construct the perspective relationship graph: Let the matching key point pairs extracted from two different camera perspectives be (k1, k2), where k1 is from perspective V1 and k2 is from perspective V2:

[0125] As an optional embodiment, step S106, based on the correlation relationship between multiple feature data, multiple video data are fused to obtain a fusion result, including: based on the correlation relationship, determining the fusion weights corresponding to the multiple video data respectively; based on the fusion weights, the multiple video data are fused to obtain a fusion result.

[0126] In this embodiment, based on the association relationship between multiple feature data, multiple video data are fused, and the fusion weights corresponding to each video data can be determined based on the association relationship. Based on the fusion weights, multiple video data can be fused to obtain corresponding fusion results.

[0127] Optionally, use the calculated fusion weights to perform pixel-level fusion on the video frames from each camera perspective. During the fusion process, pay attention to maintaining the temporal and spatial consistency between video frames to avoid flickering, blurring or misalignment. This requires the application of some post-processing techniques. After the above steps, a fused high-quality video stream is generated. The video stream should contain complementary information from multiple perspectives while reducing the impact of adverse factors such as lighting changes, occlusions, and camera movement. During the entire fusion process, real-time performance must be ensured. At the same time, based on the actual application effect, the feature extraction, key point recognition, weight calculation and fusion strategy steps are continuously optimized to improve the overall performance and robustness of the system.

[0128] Optionally, perform pixel-level fusion on the video frames from each camera perspective. During the fusion process, pay attention to maintaining the temporal and spatial consistency between the video frames: It can be assumed that there are n camera perspectives, and the pixel value of the video frame from the i-th camera perspective at the coordinate (x, y) is P i (x, y), the calculated fusion weight of this perspective is w i (Satisfy∑ i =1 n w i =1).

[0129] Optionally, in the basic fusion calculation, a weighted sum can be performed to obtain a preliminary fused pixel value F(x,y):

[0130]

[0131] As an optional embodiment, based on the association relationship, the fusion weights corresponding to multiple video data are determined, including: obtaining the time information of the acquired video data, and setting the spatiotemporal consistency factor based on the time information, wherein the spatiotemporal consistency factor includes the time information and the coordinate position of the feature data; based on the spatiotemporal consistency factor and the association relationship, the fusion weight is determined.

[0132] In this embodiment, in the process of determining the fusion weights corresponding to the plurality of video data respectively based on the association relationship, the time information of the video data can be collected, and the spatiotemporal consistency factor can be set based on the time information. The fusion weight is determined by using the spatiotemporal consistency factor and the association relationship, wherein the spatiotemporal consistency factor can include the time information and the coordinate position of the feature data.

[0133] Optionally, in the embodiment of the invention, the spatiotemporal consistency is considered, and t is assumed to represent time (time information). For pixels at the same position in the previous and next frames, a spatiotemporal consistency factor λ(x, y, t) is defined. Assuming that the pixel value after fusion of the previous frame is F(x, y, t-1), the adjusted fused pixel value F(x, y, t) shown below can be obtained:

[0134] F(x,y,t)=λ(x,y,t)F(x,y,t)+(1-λ(x,y,t))F(x,y,t-1).

[0135] Optionally, in a 3D scene reconstruction and ranging system based on multi-video fusion, considering the spatiotemporal consistency factor when determining the fusion weight of video data is an important technical optimization point, which plays a key role in improving the coherence of the fused video and the accuracy of 3D reconstruction.

[0136] Optionally, in view of the importance of time information, in real-time video stream fusion, video data from different perspectives usually have slight differences in time, which may be caused by hardware delays, network transmission time or processing speed differences. Consideration of time information helps the system maintain the timing correspondence between frames when fusing multi-perspective videos, avoiding scene dislocation or flickering caused by time inconsistency.

[0137] Optionally, for the setting of the spatiotemporal consistency factor, the spatiotemporal consistency factor includes not only the time information, but also takes into account the coordinate position of the feature data in each perspective. This means that when determining the fusion weight, the system not only focuses on the visual similarity of the feature points, but also ensures their continuity and consistency in spatial position. For example, in consecutive video frames, even if the visual appearance of the feature points changes slightly, as long as their positions are relatively stable, the system will still give them a higher fusion weight, which helps maintain the coherence of the video stream and the stability of the 3D reconstruction.

[0138] Optionally, for dynamic adjustment of fusion weights, the system can dynamically adjust the fusion weights of video data by using the correlation between the spatiotemporal consistency factor and the feature points. For those feature points that are consistent in time and continuous and stable in position, the video data of their perspective will be given a higher fusion weight. This shows that the system relies more on data with high spatiotemporal consistency when fusing videos, thereby improving the quality of the fused video and the accuracy of 3D reconstruction.

[0139] Optionally, for enhanced robustness and adaptability, by considering spatiotemporal consistency, the system can more accurately identify and match feature points under complex conditions such as lighting changes, occlusions, camera movement, etc. This means that even in dynamic and uncertain environments, the system can maintain a high degree of robustness and adaptability, generating stable, flicker-free fused videos, as well as accurate and detailed 3D scene models.

[0140] Optionally, for the optimization of video fusion and 3D reconstruction, a fusion weight calculation method based on spatiotemporal consistency factors is helpful for optimizing the video fusion and 3D reconstruction process. It ensures that the system not only focuses on the static similarity of feature points, but also fully considers the consistency of time series information and spatial position, thereby generating fused videos and 3D models that are more in line with real scene changes.

[0141] In the embodiment of the present application, the spatiotemporal consistency factor is set based on the time information of the video data and the coordinate position of the feature data, which is an important point in determining the video data fusion weight. The above method not only improves the visual quality of the fused video, but also significantly enhances the accuracy of 3D scene reconstruction, especially when dealing with dynamic and complex environments. By dynamically adjusting the fusion weight, the system can better cope with challenges such as lighting changes, occlusion and camera movement, ensuring the stability and robustness of 3D scene reconstruction and ranging functions.

[0142] As an optional embodiment, step S108, constructing an initial three-dimensional model of the target scene based on the depth information, includes: converting the depth information into three-dimensional coordinates, and constructing the initial three-dimensional model based on the three-dimensional coordinates.

[0143] In this embodiment, in the process of constructing the initial three-dimensional model of the target scene based on the depth information, the depth information may be converted into three-dimensional coordinates, and the initial three-dimensional model may be constructed based on the three-dimensional coordinates.

[0144] Optionally, a depth estimation network is used to obtain depth information from the fused video stream. Depth information is usually presented in the form of a depth map, where the value of each pixel represents the distance between the corresponding scene point and the camera. To convert the information in the depth map into three-dimensional coordinates, the system needs to know the intrinsic parameters (such as focal length, principal point coordinates) and extrinsic parameters (such as position, orientation) of the camera. Intrinsic parameters are used to convert image coordinates into normalized device coordinates, while extrinsic parameters are used to convert normalized device coordinates into three-dimensional coordinates in the world coordinate system.

[0145] Optionally, by combining the depth information d with the camera parameters, the two-dimensional coordinates (u, v) of each pixel in the two-dimensional image are converted into coordinate points (X, Y, Z) in three-dimensional space, thereby constructing an initial three-dimensional model of the target scene.

[0146] As an optional embodiment, step S110, determining the initial three-dimensional model whose evaluation result satisfies the evaluation result threshold as the target three-dimensional model of the target scene, includes: in response to the evaluation result exceeding the evaluation result threshold, determining the initial three-dimensional model as the target three-dimensional model; in response to the evaluation result not exceeding the evaluation result threshold, adjusting the initial three-dimensional model, and determining the adjusted initial three-dimensional model as the target three-dimensional model, wherein the evaluation result of the adjusted initial three-dimensional model exceeds the evaluation result threshold.

[0147] In this embodiment, in the process of determining the initial three-dimensional model whose evaluation result satisfies the evaluation result threshold as the target three-dimensional model, if the evaluation result exceeds the evaluation result threshold, the initial three-dimensional model can be determined as the target three-dimensional model. If the evaluation result does not exceed the evaluation result threshold, the initial three-dimensional model is adjusted, and the adjusted initial three-dimensional model that exceeds the evaluation result threshold can be determined as the target three-dimensional model.

[0148] Optionally, the process of determining the target 3D model of the target scene is based on comparing the evaluation result of the initial 3D model with a preset evaluation result threshold. The above method can ensure that a target 3D model that meets the quality standard can be generated, while also providing a way for iterative optimization and adjustment to address possible deficiencies in the initial 3D model.

[0149] Optionally, an evaluation result of the initial three-dimensional model is calculated, and the above evaluation may be based on multiple aspects, including but not limited to the geometric accuracy, texture quality, number of faces, computational efficiency and matching degree with the actual scene of the model. The evaluation result is calculated by comparing the difference between the model and the actual scene, the self-consistency and integrity of the model and other factors. A threshold value of the evaluation result is set to determine whether the initial three-dimensional model meets the quality requirements. If the evaluation result exceeds the preset threshold, it indicates that the quality of the model has reached the expected standard and can be directly determined as the target three-dimensional model for subsequent three-dimensional scene analysis and ranging functions.

[0150] Through the above process, the system of the present invention can ensure that the generated three-dimensional model meets specific application requirements and quality standards. This mechanism is crucial for application scenarios that require high-precision three-dimensional models, such as autonomous driving, virtual reality, augmented reality, and robot navigation. At the same time, by setting evaluation result thresholds and iterative adjustment processes, the system can continuously optimize the construction of the model when dealing with complex and dynamic environments, and improve the adaptability and reliability of the model.

[0151] In an embodiment of the present invention, if it is necessary to construct a three-dimensional model corresponding to a target scene, the video data of the target scene can be collected by using video acquisition devices respectively deployed at multiple viewing angles of the target scene to obtain multiple video data corresponding to multiple viewing angles. The video data can be analyzed to obtain corresponding feature data, and the correlation between the feature data can be analyzed. The video data is fused through the correlation to obtain a fusion result. The depth information between the target scene and the video acquisition device is extracted from the fusion result, and the initial three-dimensional model is constructed using the depth information. The constructed initial three-dimensional model is evaluated to obtain an evaluation result. The initial three-dimensional model whose evaluation result meets the evaluation result threshold is determined as the target three-dimensional model of the target scene. In this embodiment, the video data fusion technology under multiple viewing angles can dynamically identify the correlation between the feature data sets of multiple viewing angles, so that under specific conditions (for example, a certain viewing angle is blocked or the light is insufficient), the proportion of the video data under the viewing angle in the fusion result can also be automatically reduced, thereby achieving the purpose of improving the reliability and stability of the fusion result, thereby achieving the technical effect of improving the accuracy of the three-dimensional model construction of the scene, and solving the technical problem of low accuracy of the three-dimensional model construction of the scene.

[0152] Example 2

[0153] Another optional specific implementation is described in detail below.

[0154] At present, as three-dimensional scene reconstruction and ranging technology have shown great application potential in many fields, such as autonomous driving, virtual reality, augmented reality, and robot navigation, these technologies rely on high-precision three-dimensional model construction and real-time ranging capabilities to achieve accurate understanding and interaction of complex environments.

[0155] In related technologies, real-time adaptive multi-view video fusion modules face many challenges when dealing with complex scenes. They are unable to dynamically identify the key feature points and corresponding relationships between the camera perspectives, construct a relationship graph between perspectives, and dynamically adjust the fusion weights between perspectives according to the current scene characteristics. When realizing real-time fusion of multi-view videos, they often ignore the temporal and spatial consistency. Therefore, there is still a technical problem of low accuracy in constructing the three-dimensional model of the scene.

[0156] The present invention proposes a method for three-dimensional scene reconstruction and ranging based on multi-video fusion. If the method needs to construct a three-dimensional model corresponding to a target scene, the video data of the target scene can be collected by using video acquisition devices respectively deployed at multiple viewing angles of the target scene to obtain multiple video data corresponding to multiple viewing angles. The video data can be analyzed to obtain corresponding feature data, and the correlation between the feature data can be analyzed. The video data is fused through the correlation to obtain a fusion result. The depth information between the target scene and the video acquisition device is extracted from the fusion result, and the initial three-dimensional model is constructed using the depth information. The constructed initial three-dimensional model is evaluated to obtain an evaluation result. The initial three-dimensional model whose evaluation result meets the evaluation result threshold is determined as the target three-dimensional model of the target scene. In this embodiment, the video data fusion technology under multiple viewing angles is adopted, so that the correlation between the feature data sets between multiple viewing angles can be dynamically identified, so that under certain conditions, the proportion of the video data under the viewing angle in the fusion result can also be automatically reduced, so as to achieve the purpose of improving the reliability and stability of the fusion result, thereby achieving the technical effect of improving the accuracy of the three-dimensional model construction of the scene, and solving the technical problem of low accuracy of the three-dimensional model construction of the scene.

[0157] The method is further described below.

[0158] In this embodiment, Figure 2 is a schematic diagram of a 3D scene reconstruction and ranging system based on multi-video fusion according to an embodiment of the present invention, such as Figure 2As shown, the system may include: a multi-view video real-time acquisition and preprocessing module 202, a real-time adaptive multi-view video fusion module 204, a depth estimation and 3D reconstruction module 206, and an optimization and feedback mechanism module 208. Among them, the multi-view video real-time acquisition and preprocessing module 202 is used to use multiple high-definition cameras to synchronously acquire video streams from different perspectives, perform preliminary processing on the acquired videos, and prepare for subsequent fusion and depth estimation. The real-time adaptive multi-view video fusion module 204 is used to dynamically identify the key feature points and corresponding relationships between the perspectives of each camera through an attention mechanism, and dynamically adjust the fusion weights between the perspectives according to the current scene features by constructing a relationship diagram between perspectives to cope with the challenges of illumination changes, occlusion, and camera movement, so as to achieve real-time fusion of multi-view videos and generate a fused high-quality video stream. The depth estimation and 3D reconstruction module 206 is used to combine the depth estimation network in deep learning, directly extract a high-precision depth map from the fused video stream, and use the extracted depth map combined with the multi-view geometry principle to reconstruct a 3D point cloud, construct a 3D scene model, and realize the real-time ranging function in a 3D scene. The optimization and feedback mechanism module 208 is used to design an adaptive optimization algorithm. According to the difference between the reconstruction results and the preset standards, it dynamically adjusts the attention mechanism, GNN parameters and the training strategy of the depth estimation network to achieve continuous optimization of system performance. It introduces a user feedback mechanism to allow users to adjust system parameters according to actual needs.

[0159] Optionally, the multi-view video real-time acquisition and preprocessing module selects a certain number and type of high-definition cameras according to the requirements of the application scenario, determines the installation position and angle of the camera to ensure that different perspectives of the target scene can be covered and blind spots can be minimized as much as possible, configures the camera's resolution, frame rate, and exposure parameters to adapt to different lighting conditions and environmental changes, uses hardware synchronization signals or software synchronization algorithms, and all cameras can start and stop video recording at the same time to reduce errors caused by time differences, captures the video stream from each camera's perspective in real time, and transmits it to the processing unit for subsequent processing.

[0160] Optionally, decode the received video stream and convert it from a compressed format into a processable frame sequence, ensure that the decoding process is synchronized with the video acquisition to avoid frame loss or delay, apply an image denoising algorithm to remove noise from the video and improve image quality, pay attention to maintaining the edge details of the image, and avoid blurring important information during the denoising process. Since different cameras may have color deviations, color correction is required to make videos from different perspectives consistent in color. Color space conversion, histogram matching, and color mapping methods can be used for correction. According to the needs of subsequent processing, the video frames are cropped to remove unnecessary edge parts, and the size of the video frames is adjusted to meet the requirements of subsequent fusion and depth estimation modules. The aspect ratio needs to be kept unchanged, or it can be adjusted appropriately according to the specific algorithm. According to system resources and application requirements, the processing rate of the video frames is adjusted to ensure real-time performance. Frame skipping and interpolation methods can be used to control the frame rate while maintaining the smoothness of the video stream. A cache mechanism is set to cope with network delays or processing speed mismatches. A buffer is used to temporarily store video frames that have been collected but not yet processed to ensure the continuity and stability of the data stream.

[0161] Optionally, the real-time adaptive multi-view video fusion module uses a feature extraction model to extract features from the video frames of each camera perspective. These features should contain sufficient information to distinguish different objects and areas in the scene. An attention mechanism is applied to the extracted feature map to dynamically identify key feature points between the camera perspectives. These key points are usually salient objects, edges, and corners in the scene, which can be identified in multiple perspectives.

[0162] Optionally, a feature matching algorithm is used to match key points in different perspectives and establish a corresponding relationship between them. Each camera perspective is regarded as a node in the graph, and the matched key point pairs are regarded as edges between nodes. According to the matching degree and positional relationship of the key points, a weight is assigned to each edge to construct a relationship graph between perspectives.

[0163] Optionally, the real-time adaptive multi-view video fusion module analyzes the characteristics of the current scene, including but not limited to lighting conditions, occlusion conditions, and camera movement status. This information can be obtained by analyzing the brightness, contrast, and motion vector characteristics of the video frame. According to the scene characteristics, the model is used to calculate the importance or credibility of each camera perspective in the current scene. These weights will be used in the subsequent video fusion process. According to the calculated weights, the fusion weights between the perspectives are dynamically adjusted. For perspectives with higher importance, larger weights are assigned, and for perspectives affected by lighting changes, occlusions or camera movement, their weights are reduced.

[0164] Optionally, use the calculated fusion weights to perform pixel-level fusion on the video frames from each camera perspective. During the fusion process, pay attention to maintaining the temporal and spatial consistency between video frames to avoid flickering, blurring or misalignment. This requires the application of some post-processing techniques. After the above steps, a fused high-quality video stream is generated. The video stream should contain complementary information from multiple perspectives while reducing the impact of adverse factors such as lighting changes, occlusions, and camera movement. During the entire fusion process, real-time performance must be ensured to meet the needs of real-time applications. At the same time, based on the actual application effects, the feature extraction, key point recognition, weight calculation and fusion strategy steps are continuously optimized to improve the overall performance and robustness of the system.

[0165] Optionally, a feature extraction model is used to extract features from the video frames of each camera perspective. The feature extraction model shown below is used to extract features from the video frames (video frame images) at each perspective to obtain a feature map:

[0166]

[0167] Among them, F can be used to represent the feature extraction model; I ij Represents the j-th frame image from the i-th camera perspective; Can be used to represent Gaussian models; M and N are used to represent the dimensions of video frames; (μ x ,μ y ) can be used to represent the center coordinates of the video frame; σ x and σ y It can be used to represent the parameters of the Gaussian distribution corresponding to the control Gaussian model; ω2·Convolution(I ij ) can be used to represent a convolution model for performing a convolution operation on a video frame to extract features; ω1 can be used to represent a first weight; ω2 can be used to represent a second weight; F ij It can be used to represent the feature map obtained by the final weighted summation.

[0168] Optionally, an attention mechanism is applied on the feature map to identify key feature points. The feature data can be determined by the attention mechanism as shown below:

[0169]

[0170] Among them, K ij It can be used to represent the set of key feature points (feature data) of the j-th frame image of the i-th camera perspective after the attention mechanism; A can be used to represent the attention mechanism; L can be used to represent the number of feature data in the feature map.

[0171] Optionally, using a feature matching algorithm, key points from different perspectives can be matched. The feature similarity can be determined by the following formula:

[0172]

[0173] Among them, k ip ,k jq Can be used to represent two feature data in a feature data pair; S(k ip ,k jq ) can be used to represent the feature similarity between two feature data in a feature data pair; α and β can be adjustable parameters, with a value range between 0 and 1; and They can represent the vectors corresponding to the two feature data respectively.

[0174] Optionally, let P(k ip )=(x ip ,y ip ) and P(k jq )=(x jq ,y jq ) are the coordinates of the key points in their respective view images, and the positional relationship weight can be determined by the following formula:

[0175]

[0176] Among them, W pos (k ip ,k jq ) can be used to represent the weight of the positional relationship between two feature data; γ can be used to represent the adjustment parameter of the positional relationship; x ip and x jq Can be used to represent the horizontal coordinates of two feature data; y ip and jq Can be used to represent the vertical coordinate of two feature data.

[0177] Optionally, the matching weight is determined by the following formula:

[0178] M(k ip ,k jq )=λ×S(k ip ,k jq )+(1-λ)×W pos (k ip ,k jq )

[0179] Among them, M(k ip ,k jq ) can be used to represent the matching weight between two feature data; λ can be used to represent the parameter that balances feature similarity and position relationship weight.ip ,k jq ) exceeds the set threshold (matching weight threshold), it can be said that the two feature data (k ip and k jq ) is the key point of the match.

[0180] Optionally, a weight is assigned to each edge to construct a relationship graph between perspectives: let the matching key point pairs extracted from two different camera perspectives be (k1, k2), where k1 comes from perspective V1 and k2 comes from perspective V2.

[0181] The inner product similarity S can be calculated by the following formula f :

[0182]

[0183] Among them, S f (k1, k2) can be used to represent the inner product similarity between k1 and k2; k1 and k2 can represent two key points respectively; Can be used to represent the eigenvector of k1; Can be used to represent the eigenvector of k2.

[0184] Optionally, the Mahalanobis distance similarity S of the feature vector can also be calculated by the following formula: m :

[0185]

[0186] Among them, S m (k1, k2) can be used to represent the Mahalanobis distance similarity between k1 and k2; ∑ can be used to represent the covariance matrix of the feature vector.

[0187] Optionally, let (x1, y1) and (x2, y2) be the coordinates of k1 and k2 in their respective view images, and the positional relationship weight W can be determined by the following formula: p :

[0188] W p (k1,k2)=exp(-θ((x1-x2) 2 +(y1-y2) 2 ))

[0189] Among them, W p (k1, k2) can be used to represent the positional relationship weight of k1 and k2; θ can be used to represent the adjustment parameter.

[0190] Optionally, by combining feature similarity and position relationship, the edge weight w can be constructed as follows:

[0191]

[0192] Among them, w(k1, k2) can be used to represent the edge weight between k1 and k2; ω1 and ω2 are used to represent weight distribution parameters respectively, and ω1+ω2=1.

[0193] Optionally, perform pixel-level fusion on the video frames from each camera perspective. During the fusion process, pay attention to maintaining the temporal and spatial consistency between the video frames: Suppose there are n camera perspectives, and the pixel value of the video frame from the i-th camera perspective at the coordinate (x, y) is P i (x, y), the calculated fusion weight of this perspective is w i (Satisfy∑ i =1 n w i =1).

[0194] Optionally, in the basic fusion calculation, a weighted sum can be performed to obtain a preliminary fused pixel value F(x,y):

[0195]

[0196] Optionally, considering the spatiotemporal consistency, let t represent the time (temporal information), and define a spatiotemporal consistency factor λ(x, y, t) for the pixels at the same position in the previous and next frames. Assuming that the pixel value after fusion of the previous frame is F(x, y, t-1), the adjusted fused pixel value F(x, y, t) can be obtained as shown below:

[0197] F(x,y,t)=λ(x,y,t)F(x,y,t)+(1-λ(x,y,t))F(x,y,t-1).

[0198] Optionally, the depth estimation and 3D reconstruction module selects a depth estimation network for the current application scenario and is trained with a large amount of data to learn the ability to predict depth information from monocular or multi-camera images. Each frame in the fused video stream is used as input to the depth estimation network. Since the video stream is continuous, time information is used to enhance the accuracy of depth estimation. The depth estimation network processes the input frames and outputs the corresponding depth map. Each pixel value in the depth map represents the depth of the corresponding point in the scene. The predicted depth map is post-processed to remove noise, fill holes and smooth edges. This can be achieved through filters or depth map repair algorithms.

[0199] Optionally, the intrinsic and extrinsic parameters of each camera are obtained. These parameters are necessary for 3D reconstruction. The camera intrinsic parameters and depth map are used to convert each pixel in the depth map into a point cloud in 3D space. This usually involves converting the pixel coordinates (u, v) and depth value d into 3D coordinates (X, Y, Z). When using multiple camera perspectives, the point clouds from different perspectives need to be fused. This can be achieved by converting the point cloud to a unified world coordinate system and removing duplicate or conflicting points. The fused point cloud is optimized to remove noise, smooth the surface and fill holes. This can be achieved through point cloud filtering and surface reconstruction technology.

[0200] Optionally, the optimized point cloud is converted into a surface mesh to construct a 3D scene model, and texture information in the original video frame is mapped onto the 3D scene model to enhance its realism, which typically involves matching pixels in the video frame with vertices in the 3D model and applying a texture mapping algorithm.

[0201] Optionally, on the three-dimensional scene model, ray casting is performed according to the point or path input by the user, which involves converting the two-dimensional input into a ray in three-dimensional space, detecting the intersection of the ray with the surface mesh in the three-dimensional scene model, which can be achieved through an intersection test algorithm between the ray and the triangle, and calculating the distance based on the three-dimensional coordinates of the intersection and the starting point of the ray, which can be achieved through a simple three-dimensional space distance formula, and feeding the calculated distance back to the user in real time.

[0202] Optionally, the optimization and feedback mechanism establishes evaluation criteria for 3D reconstruction results. These criteria should be set based on application requirements. A performance monitoring module is implemented to evaluate the reconstruction results of the current system in real time or periodically and compare them with preset standards. The differences between the reconstruction results and the preset standards are analyzed to identify performance bottlenecks or deficiencies. A set of adaptive adjustment strategies are designed, including: Adjusting the attention mechanism: Dynamically adjusting the attention mechanism based on which areas in the reconstruction results perform poorly so that the network pays more attention to these areas. Optimizing graph neural network parameters: Based on performance differences, using optimization algorithms to automatically adjust the weights and parameters of the graph neural network. Depth estimation network training strategy: Based on error feedback, adjust the learning rate, batch size, training cycle training parameters, or introduce new training data and methods. Apply the above adjustment strategy to the system and monitor the performance changes after adjustment.

[0203] Optionally, an intuitive and easy-to-use user interface is designed to allow users to view the reconstruction results, evaluate system performance, and provide feedback. User satisfaction, suggestions, and improvement opinions on the reconstruction results are collected through the user interface. The collected user feedback is sorted and analyzed to identify the performance indicators and common problems that users are most concerned about. Based on user feedback, system parameters are adjusted to better meet user needs. User feedback is taken into consideration in the adaptive optimization algorithm to form a closed-loop feedback mechanism. The system continuously optimizes its own performance based on user feedback and internal performance monitoring results.

[0204] Optionally, set a regular evaluation cycle to conduct a comprehensive evaluation of system performance, including but not limited to reconstruction quality, computing efficiency, and user satisfaction. After the evaluation, summarize the strengths and weaknesses of the current system, reflect on the effectiveness of the adaptive optimization algorithm and user feedback mechanism, and formulate an improvement plan for the next stage based on the evaluation results and user feedback. According to the improvement plan, iteratively update the system to continuously optimize system performance and enhance user experience.

[0205] Compared with the related art, in the embodiment of the present invention, by adopting the multi-view video fusion technology and combining the real-time adaptive multi-view video fusion module, the key feature points and corresponding relationships between the camera perspectives can be dynamically identified, and the fusion weights can be dynamically adjusted according to the current scene characteristics. This mechanism effectively copes with the challenges of illumination changes, occlusion, and camera movement, and significantly improves the quality of the fused video stream. By introducing the optimization and feedback mechanism, the evaluation criteria for the three-dimensional reconstruction results are established, and the reconstruction results of the current system are evaluated in real time or regularly, and compared with the preset standards. This mechanism can timely discover the system performance bottleneck or shortcomings, and design adaptive adjustment strategies for optimization. These adjustment strategies include adjusting the attention mechanism, optimizing the parameters of the graph neural network, and adjusting the training strategy of the depth estimation network. In addition, the system also allows users to view the reconstruction results, evaluate the system performance and provide feedback through the user interface. These user feedbacks are taken into account by the adaptive optimization algorithm, forming a closed-loop feedback mechanism. This mechanism enables the system to continuously optimize its own performance based on user feedback and internal performance monitoring results, and continuously improve user experience and system performance.

[0206] The following uses a conference room as the target scene to further explain the three-dimensional scene reconstruction method in the embodiment of the present invention.

[0207] Install 4 high-definition cameras in different corners of the conference room to ensure that they can fully cover all angles of view in the entire conference room and minimize blind spots. Configure the camera resolution and frame rate parameters to adapt to changes in indoor lighting and ensure that clear video streams can be obtained under different lighting conditions. Use software synchronization algorithms to ensure that all cameras can start and stop video recording at the same time to reduce errors caused by time differences. Perform real-time decoding of the captured video and convert the compressed video stream into a processable frame sequence to ensure that the decoding process is synchronized with video capture to avoid frame loss or delay. Apply image denoising algorithms to remove noise from the video and improve image quality while keeping the edge details of the image clear and avoiding blurring of important information. Perform color correction and use color space conversion, histogram matching or color mapping methods to keep the colors of videos from different angles consistent and improve the fusion effect. Crop and resize the video frames according to subsequent processing requirements to keep the aspect ratio unchanged or make appropriate adjustments based on the specific algorithm. Set up a cache mechanism to temporarily store the captured but not yet processed video frames to ensure the continuity and stability of the data stream.

[0208] Using the feature extraction model, the feature extraction is performed on the video frame of each camera perspective. ij Represents the j-th frame image from the i-th camera perspective, and performs feature extraction: Among them, I is the video frame image, F is the feature extraction model, A is the attention mechanism, ω1 and ω2 are weight parameters, M and N are the dimensions of the image, (μ x ,μ y ) is the center coordinate of the image, σ x and σ y is the parameter that controls the Gaussian distribution, Convolution(I ij ) represents the convolution operation on the image to extract features.

[0209] Optionally, the above features should contain enough information to distinguish different objects and regions in the scene, and an attention mechanism is applied on the extracted feature map to dynamically identify key feature points between camera views: Among them, K ij It represents the set of key feature points of the j-th frame image of the i-th camera perspective after the attention mechanism, and L is the number of features in the feature map.

[0210] Optionally, the key points are usually significant objects, edges or corner points in the scene. The feature matching algorithm is used to match the key points in different perspectives, establish a correspondence between them, regard each camera perspective as a node in the graph, and regard the matched key point pairs as edges between the nodes. According to the matching degree and positional relationship of the key points, a weight is assigned to each edge, and a relationship graph between perspectives is constructed. The lighting conditions and occlusion characteristics of the current scene are analyzed, and the model is used to calculate the importance or credibility of each camera perspective in the current scene, and the fusion weights between the perspectives are dynamically adjusted. The calculated fusion weights are used to perform pixel-level fusion of the video frames of each camera perspective.

[0211] Optionally, there are n camera views, and the pixel value of the video frame of the i-th camera view at the coordinate (x, y) is P i (x, y), the calculated fusion weight of this perspective is w i (Satisfy∑ i =1 n w i =1), basic fusion calculation: perform weighted summation to obtain the initial fusion pixel value F(x,y): Pay attention to maintaining the spatiotemporal consistency between video frames. Let t represent time. For pixels in the same position in the previous and next frames, define a spatiotemporal consistency factor λ(x,y,t). Assuming that the pixel value of the previous frame after fusion is F(x,y,t-1), the adjusted fused pixel value is: F(x,y,t)=λ(x,y,t)F(x,y,t)+(1-λ(x,y,t))F(x,y,t-1). This avoids flickering, blurring or dislocation. After fusion processing, a high-quality video stream containing complementary information from multiple perspectives is generated.

[0212] Optionally, a depth estimation network suitable for indoor scenes is selected, and the fused video stream is used for depth estimation. Since the video stream is continuous, time information can be used to enhance the accuracy of depth estimation. The depth estimation network processes the input frame and outputs the corresponding depth map, where each pixel value represents the depth of the corresponding point in the scene. The predicted depth map is post-processed to remove noise, fill holes and smooth edges to improve the quality of the depth map. The intrinsic and extrinsic parameters of each camera are obtained. These parameters are necessary for three-dimensional reconstruction. Using the camera intrinsic parameters and the depth map, each pixel in the depth map is converted into a point cloud in three-dimensional space. When using multiple camera perspectives, it is necessary to fuse the point clouds from different perspectives and remove duplicate or conflicting points. The fused point cloud is optimized to remove noise, smooth the surface and fill holes. The optimized point cloud is converted into a surface mesh to construct a three-dimensional scene model, and the texture information in the original video frame is mapped to the three-dimensional scene model to enhance its realism.

[0213] Optionally, design evaluation criteria, which should be set based on application requirements, including reconstruction quality and computational efficiency. Implement a performance monitoring module to evaluate the reconstruction results of the current system in real time or periodically and compare them with preset standards, analyze the differences between the reconstruction results and the preset standards, identify performance bottlenecks or deficiencies, design adaptive adjustment strategies, including adjusting the attention mechanism, optimizing graph neural network parameters, and adjusting the training strategy of the depth estimation network. Apply the above adjustment strategies to the system and monitor the performance changes after adjustment. Design an intuitive and easy-to-use user interface that allows users to view the reconstruction results, evaluate system performance, and provide feedback. Collect user satisfaction, suggestions, and improvement opinions on the reconstruction results through the user interface.

[0214] In an embodiment of the present invention, if it is necessary to construct a three-dimensional model corresponding to a target scene, the video data of the target scene can be collected by using video acquisition devices respectively deployed at multiple viewing angles of the target scene to obtain multiple video data corresponding to multiple viewing angles. The video data can be analyzed to obtain corresponding feature data, and the correlation between the feature data can be analyzed. The video data is fused through the correlation to obtain a fusion result. The depth information between the target scene and the video acquisition device is extracted from the fusion result, and the initial three-dimensional model is constructed using the depth information. The constructed initial three-dimensional model is evaluated to obtain an evaluation result. The initial three-dimensional model whose evaluation result meets the evaluation result threshold is determined as the target three-dimensional model of the target scene. In this embodiment, the video data fusion technology under multiple viewing angles can dynamically identify the correlation between the feature data sets of multiple viewing angles, so that under specific conditions (for example, a certain viewing angle is blocked or the light is insufficient), the proportion of the video data under the viewing angle in the fusion result can also be automatically reduced, thereby achieving the purpose of improving the reliability and stability of the fusion result, thereby achieving the technical effect of improving the accuracy of the three-dimensional model construction of the scene, and solving the technical problem of low accuracy of the three-dimensional model construction of the scene.

[0215] Example 3

[0216] The embodiment of the present invention provides a three-dimensional model construction device for a scene. It should be noted that the document generation device of the embodiment of the present invention can be used to execute Figure 1 The three-dimensional model construction of the scene provided by the embodiment of the present invention is described below.

[0217] Figure 3 is a schematic diagram of a three-dimensional model construction device for a scene according to an embodiment of the present invention, such as Figure 3 As shown, the device may include: a collection unit 302 , a fusion unit 304 , a construction unit 306 , a first determination unit 308 , and a second determination unit 310 .

[0218] The acquisition unit 302 is used to acquire video data of a target scene by using video acquisition devices deployed at multiple viewing angles to obtain multiple video data.

[0219] The fusion unit 304 is used to obtain multiple feature data from multiple video data, and fuse the multiple video data based on the correlation relationship between the multiple feature data to obtain a fusion result.

[0220] The construction unit 306 is used to extract the depth information of the target scene from the fusion result, and to construct an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to indicate the distance between the target scene and the video acquisition device.

[0221] The first determining unit 308 is used to determine the evaluation result of the initial three-dimensional model.

[0222] The second determination unit 3110 is configured to determine the initial three-dimensional model whose evaluation result satisfies the evaluation result threshold as the target three-dimensional model of the target scene.

[0223] The three-dimensional model construction device of the scene provided by the embodiment of the present invention uses the video acquisition devices deployed at multiple viewing angles to acquire the video data of the target scene through the acquisition unit 302 to obtain multiple video data; obtains multiple feature data in the multiple video data through the fusion unit 304, and fuses the multiple video data based on the correlation between the multiple feature data to obtain a fusion result; extracts the depth information of the target scene from the fusion result through the construction unit 306, and constructs the initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to indicate the distance between the target scene and the video acquisition device; determines the evaluation result of the initial three-dimensional model through the first determination unit 308; and determines the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene through the second determination unit 310, thereby solving the technical problem of low accuracy in building the three-dimensional model of the scene and achieving the technical effect of improving the accuracy of building the three-dimensional model of the scene.

[0224] The three-dimensional model construction device of the above scene may also include a processor and a memory. The above units are all stored in the memory as program units, and the processor executes the above program units stored in the memory to implement corresponding functions.

[0225] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set to control the same device type of devices to be shut down to perform graceful shutdown by adjusting kernel parameters.

[0226] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0227] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the work efficiency of traders can be improved by adjusting the kernel parameters.

[0228] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0229] Example 4

[0230] According to an embodiment of the present invention, there is further provided a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the three-dimensional model of the scene is constructed.

[0231] Example 5

[0232] According to an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the program executes construction of a three-dimensional model of a scene when it is run.

[0233] Example 6

[0234] Figure 4 is a schematic diagram of an electronic device for a method for constructing a three-dimensional model of a scene according to an embodiment of the present invention, such as Figure 4 As shown, an embodiment of the present invention further provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. The above steps are implemented when the processor executes the program.

[0235] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0236] The invention also provides a computer program product which, when executed on a data processing device, is adapted to execute a program initialized with the above method steps.

[0237] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0238] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0239] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0240] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0241] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0242] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0243] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0244] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0245] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0246] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for constructing a three-dimensional model of a scene, characterized in that: include: Using video acquisition devices deployed at multiple viewing angles respectively, to acquire video data of a target scene, to obtain a plurality of video data; Acquire multiple feature data from the multiple video data, and fuse the multiple video data based on the association relationship between the multiple feature data to obtain a fusion result; Extracting depth information of the target scene from the fusion result, and constructing an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to indicate the distance between the target scene and the video acquisition device; Determining an evaluation result of the initial three-dimensional model; The initial three-dimensional model whose evaluation result satisfies the evaluation result threshold is determined as the target three-dimensional model of the target scene.

2. The construction method according to claim 1, characterized in that: Acquiring a plurality of feature data from a plurality of video data, including: By using the attention mechanism, the plurality of feature data in the plurality of video data are respectively identified.

3. The construction method according to claim 2, characterized in that: Using the attention mechanism, a plurality of feature data in the plurality of video data are respectively identified, including: Using a feature extraction model, extracting features from a plurality of video frames in the video data to obtain a plurality of feature maps; Utilizing the attention mechanism, a plurality of feature data are identified from a plurality of feature graphs.

4. The construction method according to claim 3, characterized in that: The feature extraction model includes a Gaussian model and a convolution model, the feature map includes a first feature map and a second feature map, the Gaussian model corresponds to a first weight and the first feature map, and the convolution model corresponds to a second weight and the second feature map, wherein the feature extraction model is used to extract features from a plurality of video frames in the video data to obtain a plurality of feature maps, including: Inputting the video frame into the Gaussian model, extracting features from the video frame using the Gaussian model to obtain the first feature map, and inputting the video frame into the convolutional model, extracting features from the video frame using the convolutional model to obtain the second feature map; The first weight is assigned to the first feature map, and the second weight is assigned to the second feature map, and the feature map is obtained by weighted summation.

5. The construction method according to claim 3, characterized in that: The method further comprises: Determining feature similarities between the plurality of feature data based on the plurality of feature data at different perspectives; Determine positional relationship data between the plurality of feature data based on coordinate positions respectively corresponding to the plurality of feature data at different viewing angles; Determine matching weights of the plurality of feature data based on the feature similarities and the position relationship data, wherein the matching weights are used to represent the degree of association of the association relationships between the plurality of feature data; In response to the matching weight exceeding a matching weight threshold, selecting a feature data pair from the plurality of feature data, wherein the feature data pair includes two of the feature data; The association relationship corresponding to the feature data pair is constructed.

6. The construction method according to claim 5, characterized in that: Constructing the association relationship corresponding to the feature data pair includes: Obtaining feature vectors corresponding to two feature data in the feature data pair, and determining an inner product similarity between the two feature vectors and a distance similarity between the two feature vectors; Obtaining coordinate positions of two feature data in the feature data pair in respective viewing angles, and determining a positional relationship weight between the two coordinate positions; The association relationship is constructed based on the inner product similarity, the distance similarity and the position relationship weight.

7. The construction method according to claim 1, characterized in that: Based on the association relationship between the plurality of feature data, the plurality of video data are fused to obtain a fusion result, including: Based on the association relationship, determining fusion weights corresponding to the plurality of video data respectively; Based on the fusion weight, the plurality of video data are fused to obtain the fusion result.

8. The construction method according to claim 7, characterized in that: Based on the association relationship, determining fusion weights corresponding to the plurality of video data respectively includes: Acquire time information of collecting the video data, and set a spatiotemporal consistency factor based on the time information, wherein the spatiotemporal consistency factor includes the time information and the coordinate position of the feature data; The fusion weight is determined based on the spatiotemporal consistency factor and the association relationship.

9. The construction method according to claim 1, characterized in that: Based on the depth information, constructing an initial three-dimensional model of the target scene includes: The depth information is converted into three-dimensional coordinates, and the initial three-dimensional model is constructed based on the three-dimensional coordinates.

10. The construction method according to claim 1, characterized in that: Determining the initial three-dimensional model whose evaluation result satisfies the evaluation result threshold as the target three-dimensional model of the target scene includes: In response to the evaluation result exceeding the evaluation result threshold, determining the initial three-dimensional model as the target three-dimensional model; In response to the evaluation result not exceeding the evaluation result threshold, the initial three-dimensional model is adjusted, and the adjusted initial three-dimensional model is determined as the target three-dimensional model, wherein the evaluation result of the adjusted initial three-dimensional model exceeds the evaluation result threshold.

11. A device for constructing a three-dimensional model of a scene, characterized in that: include: A collection unit, used to collect video data of a target scene using video collection devices deployed at multiple viewing angles to obtain a plurality of video data; A fusion unit, used for acquiring a plurality of feature data from a plurality of the video data, and fusing the plurality of the video data based on the association relationship between the plurality of feature data to obtain a fusion result; A construction unit, used to extract the depth information of the target scene from the fusion result, and construct an initial three-dimensional model of the target scene based on the depth information, wherein the depth information is used to indicate the distance between the target scene and the video acquisition device; A first determining unit, used to determine an evaluation result of the initial three-dimensional model; The second determining unit is used to determine the initial three-dimensional model whose evaluation result meets the evaluation result threshold as the target three-dimensional model of the target scene.

12. A processor, characterized in that: The processor is used to run a program, wherein the program, when run by the processor, executes the method for constructing a three-dimensional model of a scene as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for constructing a three-dimensional model of a scene as described in any one of claims 1 to 10.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for constructing a three-dimensional model of a scene as described in any one of claims 1 to 10.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for constructing a three-dimensional model of a scene according to any one of claims 1 to 10.

Images