Method, device, equipment, medium and program for reconstructing three-dimensional scene of confined space

Abstract

The application provides a method, device, equipment, medium and program for three-dimensional scene reconstruction in a confined space. The method comprises: constructing a local three-dimensional observation field of the confined space according to a visible area in panoramic video data in the confined space; generating a mask for an invisible area in the confined space according to a panoramic view line projection relationship, image quality degradation characteristics of the panoramic video data, boundary constraints of the confined space and a historical observation state; inputting characteristics of the local three-dimensional observation field, the mask for the invisible area, the historical observation state and structural prior characteristics of the confined space into a three-dimensional scene generation model as conditional inputs to obtain a candidate three-dimensional space structure for the invisible area; screening the candidate three-dimensional space structure to obtain a target three-dimensional space structure for the invisible area; and fusing the local three-dimensional observation field and the target three-dimensional space structure to obtain a three-dimensional scene reconstruction model of the confined space. The application can be applied to the field of mine shafts and mines, and can significantly improve the quality of the three-dimensional scene reconstruction model of the confined space.

Description

Technical Field

[0001] This application relates to the fields of computer vision and intelligent generation technology of confined space, and in particular to a method, apparatus, device, medium and program for reconstructing three-dimensional scenes in confined space. Background Technology

[0002] Confined spaces refer to enclosed or semi-enclosed spaces with clear spatial boundaries, limited access range, and complex observation conditions, such as coal mine longwall faces, coal mine roadways, underground utility tunnels, underground powerhouses, and enclosed equipment rooms. In such spaces, 3D scene perception and reconstruction are crucial foundations for realizing digital twins, remote inspection, robot navigation, safety monitoring, and emergency response. Existing technologies typically acquire spatial information using methods such as LiDAR, depth cameras, multi-view cameras, monocular video, or panoramic video, and combine these with point cloud registration, Simultaneous Localization and Mapping (SLAM), depth estimation, neural radiation fields, 3D Gaussian representation, or mesh reconstruction to reconstruct the 3D structure of the scene.

[0003] Panoramic video offers the advantage of a wide field of view, enabling the acquisition of environmental information over a large area from a single acquisition point. Therefore, it is increasingly being applied to tasks such as confined space inspection, panoramic walkthroughs, and 3D scene construction. However, panoramic video cannot completely eliminate invisible areas. In scenarios such as coal mine longwall faces, tunnels, and utility tunnels, factors such as equipment obstruction, personnel obstruction, dust and smoke, low-light environments, near-field blind spots, installation location limitations, and decreased resolution at long distances still prevent some areas from being directly and effectively observed. For these invisible areas, existing panoramic video 3D scene reconstruction methods typically treat them as hollow areas, low-confidence areas, or missing areas, making it difficult to further recover their 3D structure.

[0004] On the other hand, while existing point cloud completion, mesh repair, and 3D generation methods can complete the missing geometric structures to a certain extent, these methods lack the utilization of the structural rules of the confined space. This leads to the 3D scene reconstruction results of the invisible areas of the confined space being prone to not conforming to the actual spatial relationships, which in turn affects the quality of the 3D scene reconstruction model of the entire confined space. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program for reconstructing three-dimensional scenes in confined spaces, in order to solve the problem that existing methods for reconstructing three-dimensional scenes in confined spaces cannot guarantee the quality of the reconstructed three-dimensional scene model.

[0006] The first aspect of this application provides a method for reconstructing a three-dimensional scene in a confined space, comprising: Acquire panoramic video data within a confined space; Based on the visible area in the panoramic video data, a local three-dimensional observation field is constructed for the confined space. The local three-dimensional observation field is used to characterize the spatial structure in the confined space that has been effectively observed by the panoramic video data. Based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation status of the confined space, a mask is generated for the invisible areas in the confined space that are not effectively observed by the panoramic video data. The features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the structural prior features of the confined space are used as conditions to input the three-dimensional scene generation model to obtain the candidate three-dimensional spatial structure of the invisible region. The candidate three-dimensional spatial structures are screened to obtain the target three-dimensional spatial structure of the invisible region; The local three-dimensional observation field and the target three-dimensional spatial structure are fused to obtain a three-dimensional scene reconstruction model of the confined space.

[0007] A second aspect of this application provides a confined space three-dimensional scene reconstruction device, comprising: The acquisition module is used to acquire panoramic video data within a confined space. A construction module is used to construct a local three-dimensional observation field of the confined space based on the visible area in the panoramic video data. The local three-dimensional observation field is used to characterize the spatial structure in the confined space that has been effectively observed by the panoramic video data. The generation module is used to generate a mask for invisible areas in the confined space that are not effectively observed by the panoramic video data, based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation status of the confined space. The input module is used to input the features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the structural prior features of the confined space as conditions into the three-dimensional scene generation model to obtain the candidate three-dimensional spatial structure of the invisible region. The filtering module is used to filter the candidate three-dimensional spatial structures to obtain the target three-dimensional spatial structure of the invisible region; The fusion module is used to fuse the local three-dimensional observation field and the target three-dimensional spatial structure to obtain a three-dimensional scene reconstruction model of the confined space.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a confined space three-dimensional scene reconstruction method as described above.

[0009] A fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a confined space three-dimensional scene reconstruction method as described in any of the above.

[0010] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements a confined space three-dimensional scene reconstruction method as described in any of the above.

[0011] In this application, panoramic video data within a confined space is first acquired. Then, based on the visible areas in the panoramic video data, a local 3D observation field is constructed for the confined space, representing the spatial structure effectively observed by the panoramic video data. Next, based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation state of the confined space, a mask is generated for the invisible areas in the confined space that were not effectively observed by the panoramic video data. Then, the features of the local 3D observation field, the mask for the invisible areas, the historical observation state, and the prior structural features of the confined space are used as input conditions into a 3D scene generation model to obtain candidate 3D spatial structures for the invisible areas. The candidate 3D spatial structures are then filtered to obtain the target 3D spatial structure for the invisible areas. Finally, the local 3D observation field and the target 3D spatial structure are fused to obtain a 3D scene reconstruction model for the confined space. This method can utilize the characteristics of confined space (boundary constraints of confined space, historical observation state of confined space, and structural prior features of confined space) to generate and reconstruct 3D invisible areas, avoiding the problem that the 3D scene reconstruction results of invisible areas in confined space do not conform to the actual spatial relationship. Therefore, it can solve the problem that existing 3D scene reconstruction methods in confined space cannot guarantee the quality of 3D scene reconstruction models. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1This is a flowchart illustrating a method for reconstructing a three-dimensional scene in a confined space, as shown in an embodiment of this application.

[0014] Figure 2 This is a schematic diagram illustrating the principle of invisible region identification and 3D generation and reconstruction provided in the embodiments of this application.

[0015] Figure 3 This is a schematic diagram illustrating the implementation process of the confined space three-dimensional scene reconstruction method provided in the embodiments of this application.

[0016] Figure 4 This is a structural block diagram of a confined space three-dimensional scene reconstruction device shown in an embodiment of this application.

[0017] Figure 5 This is a schematic diagram of the physical structure of an electronic device as shown in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The method of this application can be implemented by electronic devices such as computers, servers, or 3D scene reconstruction equipment, or by a 3D scene reconstruction device installed in such electronic devices. This 3D scene reconstruction device can be implemented through software, hardware, or a combination of both. The following will use a 3D scene reconstruction device as an example to describe the confined space 3D scene reconstruction method of this application in detail.

[0020] The method of this application can be applied to scenarios with clear spatial boundaries and limited observation conditions, such as fully mechanized coal mining faces, coal mine roadways, underground pipe corridors, tunnels, underground powerhouses, and enclosed equipment rooms. For ease of explanation, the following embodiments are described using fully mechanized coal mining faces as an example, but this application is not limited to this application scenario.

[0021] Figure 1 This is a flowchart illustrating a method for reconstructing a three-dimensional scene in a confined space, as shown in an embodiment of this application. Figure 1 As shown, the confined space 3D scene reconstruction method of this application may include: Step S101: Obtain panoramic video data within the confined space.

[0022] In this embodiment, the confined space includes, but is not limited to, coal mine longwall mining faces, coal mine roadways, underground pipe corridors, tunnels, underground powerhouses, underground storage spaces, enclosed equipment compartments, or other enclosed or semi-enclosed spaces with spatial boundary constraints.

[0023] In this embodiment, panoramic video acquisition equipment can be deployed in a confined space to collect continuous panoramic video data within the confined space.

[0024] Among them, panoramic video acquisition equipment can be a fixed panoramic camera, a panoramic camera mounted on a mobile inspection device, or a panoramic acquisition device that works in conjunction with other sensors.

[0025] In practice, the type and installation method of the panoramic video acquisition device can be set according to the specific conditions of the confined space. For ease of explanation, the following embodiments will be described using a panoramic camera as an example of the panoramic video acquisition device.

[0026] Step S102: Based on the visible area in the panoramic video data, construct a local three-dimensional observation field for the confined space. The local three-dimensional observation field is used to characterize the spatial structure in the confined space that has been effectively observed by the panoramic video data.

[0027] In this embodiment, the spatial structure for effective observation refers to the spatial structure in the panoramic video data whose image quality meets the preset observation conditions and whose corresponding three-dimensional spatial position can be determined.

[0028] In this embodiment, the local three-dimensional observation field can be represented in a target representation form, which includes any of the following: point cloud, mesh, voxel, depth field, implicit neural field, and three-dimensional Gaussian set.

[0029] Step S103: Based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation status of the confined space, generate a mask for the invisible areas in the confined space that were not effectively observed by the panoramic video data.

[0030] In this embodiment, the invisible area includes areas formed by at least one of the following factors: equipment obstruction, personnel obstruction, dust obstruction, smoke obstruction, insufficient lighting, blind spots of the panoramic camera installation, insufficient vantage point resolution, motion blur, missing acquisition frames, and areas that have not been effectively observed for a long period of time. In this embodiment, the panoramic line-of-sight projection relationship refers to the geometrical relationship of the line of sight between the panoramic camera and various spatial locations within the confined space.

[0031] Specifically, the panoramic line-of-sight projection relationship can be obtained in the following way: Based on the panoramic camera's capture pose (including position and orientation) and the spherical viewing direction corresponding to each pixel coordinate in the panoramic video data, multiple spatial rays pointing from the panoramic camera into the confined space are established. The panoramic viewing projection relationship is obtained based on the geometric relationship between each spatial ray and its spatial position within the confined space. For example, the geometric relationship between each spatial ray and its spatial position within the confined space can be directly determined as the panoramic viewing projection relationship.

[0032] In this embodiment, image quality degradation features refer to the characteristics of image quality degradation in panoramic video data caused by factors such as dust, smoke, low light, strong reflection, motion blur, and image compression distortion.

[0033] Image quality degradation characteristics can be quantitatively evaluated by indicators such as sharpness, brightness, contrast, dust concentration, smoke, and motion blur of panoramic video data.

[0034] Boundary constraints of a confined space refer to the three-dimensional spatial range defined by the physical boundaries of the confined space. For example, taking a fully mechanized coal mining face as an example, the boundary constraints of the confined space can be jointly determined by the coal wall, roof, floor, and roadway boundaries.

[0035] The historical observation status of a confined space refers to the observation records of each three-dimensional spatial location within the confined space at historical moments. The historical observation status can include at least one of the following: the most recent valid observation time, the cumulative number of valid observations, historical observation reliability, and historical visibility status. The most recent valid observation time refers to the time point at which a certain three-dimensional spatial location was most recently and effectively observed by panoramic video data. The cumulative number of valid observations refers to the total number of times a certain three-dimensional spatial location has been effectively observed by panoramic video data within a historical time range. Historical observation reliability refers to the reliability of the historical observation results for a certain three-dimensional spatial location. Historical visibility status refers to the record of whether a certain three-dimensional spatial location is a visible area at each historical moment.

[0036] Historical observation status can be stored in a historical observation status database and updated as panoramic video data is continuously acquired.

[0037] In this embodiment, the number of invisible areas in the confined space is at least one.

[0038] Step S104: Input the features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the structural prior features of the confined space as conditions into the three-dimensional scene generation model to obtain the candidate three-dimensional spatial structure of the invisible region.

[0039] In this embodiment, the features of the local three-dimensional observation field include geometric features, semantic features, texture features, and temporal variation features.

[0040] Geometric features include, but are not limited to, the distribution of point clouds, planar orientation, boundary contours, and spatial continuity in the local three-dimensional observation field.

[0041] Semantic features include, but are not limited to, category information of each three-dimensional spatial unit in the local three-dimensional observation field, such as supports, coal walls, roof, floor, conveying equipment, pipelines, and roadway boundaries.

[0042] Texture features include, but are not limited to, surface textures, edge structures, and local image patterns corresponding to local three-dimensional observation fields.

[0043] Temporal variation characteristics include, but are not limited to, structural changes between adjacent image frames, equipment motion status, and observation stability. Equipment motion status refers to the positional changes and motion trends of equipment within a confined space between adjacent image frames, such as the direction and speed of movement of a coal mining machine, and the lifting and lowering status of a hydraulic support. Observation stability refers to the degree of consistency between the results observed at the same three-dimensional spatial location in multiple consecutive image frames. Higher observation stability indicates that the observation results corresponding to that three-dimensional spatial location are more reliable.

[0044] In this embodiment, the structural prior features include, but are not limited to, at least one of the following: spatial boundary prior features, planar continuity prior features, channel orientation prior features, channel width constraint prior features, wall or coal wall continuity prior features, roof and floor constraint prior features, equipment arrangement prior features, support structure topology prior features, pipeline orientation prior features, and equipment motion trajectory prior features.

[0045] For example, taking a fully mechanized coal mining face as an example, the a priori structural features may also include the topology of the hydraulic support array, support numbering, support spacing, continuous coal face, linear structure of the scraper conveyor, movement trajectory of the coal mining machine, and the direction of face advancement.

[0046] Structural prior features can be stored in a restricted spatial structural prior library and participate in the 3D generation and reconstruction of invisible regions in the form of parameterized rules, topological graphs, constraint matrices, structural labels, conditional vectors, or neural network embeddings.

[0047] In this embodiment, after obtaining the features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the prior structural features, the above information can be used as conditions to input the three-dimensional scene generation model to generate and reconstruct the invisible region in three dimensions, thereby obtaining the candidate three-dimensional spatial structure of the invisible region.

[0048] In this embodiment, the 3D scene generation model can be any of the following: conditional generation network, 3D diffusion generation model, implicit neural field generation model, 3D Gaussian generation model, voxel generation model, point cloud completion model, mesh completion model, or a combination of at least two of the above models.

[0049] In this embodiment, the candidate three-dimensional spatial structure can be represented as at least one of point cloud, mesh, voxel, occupancy field, signed distance field, implicit field, or three-dimensional Gaussian set. When performing step S106, the candidate three-dimensional spatial structure needs to be converted into a target representation consistent with the local three-dimensional observation field, as described above.

[0050] Step S105: Filter the candidate three-dimensional spatial structures to obtain the target three-dimensional spatial structure of the invisible area.

[0051] In this embodiment, each invisible region corresponds to a target three-dimensional spatial structure. Among at least one candidate three-dimensional spatial structure corresponding to each invisible region, the target three-dimensional spatial structure is closest to the true three-dimensional spatial structure of the invisible region.

[0052] Step S106: Fuse the local three-dimensional observation field and the target three-dimensional spatial structure to obtain a three-dimensional scene reconstruction model of the confined space.

[0053] In this embodiment, the target three-dimensional spatial structure corresponding to the invisible area is fused with the local three-dimensional observation field corresponding to the visible area to form a complete three-dimensional scene model of the confined space.

[0054] In this embodiment, firstly, panoramic video data within a confined space is acquired. Then, based on the visible areas in the panoramic video data, a local 3D observation field is constructed for the confined space. This local 3D observation field characterizes the spatial structure within the confined space that has been effectively observed by the panoramic video data. Next, based on the panoramic line-of-sight projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation state of the confined space, a mask is generated for the invisible areas within the confined space that have not been effectively observed by the panoramic video data. Then, the features of the local 3D observation field, the mask for the invisible areas, the historical observation state, and the prior structural features of the confined space are used as input conditions into the 3D scene generation model to obtain candidate 3D spatial structures for the invisible areas. The candidate 3D spatial structures are then filtered to obtain the target 3D spatial structure for the invisible areas. Finally, the local 3D observation field and the target 3D spatial structure are fused to obtain a 3D scene reconstruction model for the confined space. This method can utilize the characteristics of confined space (boundary constraints of confined space, historical observation state of confined space, and structural prior features of confined space) to generate and reconstruct 3D invisible areas, avoiding the problem that the 3D scene reconstruction results of invisible areas in confined space do not conform to the actual spatial relationships. Therefore, it can solve the problem that existing 3D scene reconstruction methods in confined space cannot guarantee the quality of 3D scene reconstruction models.

[0055] In one implementation, in conjunction with the above embodiments, step S102 may include: Step S1021: Obtain the spherical orientation vector corresponding to the pixel coordinates of each frame image in the panoramic video data.

[0056] In this embodiment, the spherical direction vector refers to the three-dimensional direction vector corresponding to the pixel coordinates of each frame of the panoramic video data under the spherical projection model, which is used to characterize the observation direction of each pixel on the panoramic sphere.

[0057] For example, for panoramic video data in equidistant cylindrical projection format, the pixel coordinates can be converted into corresponding spherical direction vectors based on the mapping relationship between the pixel coordinates and latitude and longitude of each frame image.

[0058] Step S1022: Transform the directional vectors of each sphere to the confined space coordinate system. The confined space coordinate system is a three-dimensional coordinate system established based on the physical structure of the confined space.

[0059] In this embodiment, one or more of the following can be combined: installation parameters of the panoramic camera, acquisition pose, inertial measurement information, odometer information, and visual pose estimation results, to uniformly transform the spherical orientation vectors corresponding to each frame of the panoramic video data to a constrained spatial coordinate system.

[0060] The confined space coordinate system is a three-dimensional coordinate system established based on the physical structure of the confined space. For example, taking a fully mechanized coal mining face as an example, a fixed reference point on the face floor can be used as the origin, with the roadway direction as the X-axis, the face advance direction as the Y-axis, and the direction perpendicular to the floor plane and upwards as the Z-axis, to establish the confined space coordinate system. In actual implementation, the origin position and the directions of each coordinate axis of the confined space coordinate system can be set according to the specific structure of the confined space; this embodiment does not impose specific restrictions on this.

[0061] Step S1023: Perform three-dimensional spatial positioning on the coordinates of each pixel in the visible area to obtain the three-dimensional spatial position corresponding to each pixel coordinate.

[0062] In this embodiment, the depth value corresponding to each pixel coordinate in the visible area can be obtained by one or more of the following methods: monocular depth estimation, multi-view depth estimation, multi-frame geometric consistency verification, or depth sensor measurement. Based on the spherical direction vector and depth value corresponding to each pixel coordinate, the three-dimensional spatial position of each pixel coordinate in the confined spatial coordinate system can be calculated.

[0063] Step S1024: Generate three-dimensional spatial units corresponding to the visible areas based on each three-dimensional spatial location. The three-dimensional spatial unit is the smallest spatial element that represents the spatial structure that has been effectively observed in the confined space.

[0064] In this embodiment, not all three-dimensional spatial locations obtained in step S1023 meet the requirements for effective observation. Therefore, the validity of each three-dimensional spatial location is first screened: if the confidence level of the depth value corresponding to a certain three-dimensional spatial location is higher than the confidence level threshold, and the reprojection error of the three-dimensional spatial location between multiple frames of images is lower than the error threshold, the three-dimensional spatial location is determined as a valid observation three-dimensional spatial location; otherwise, the three-dimensional spatial location is discarded. Further, the valid observation three-dimensional spatial locations belonging to the same spatial region in consecutive frames of images are registered and fused according to spatial coordinates or nearest neighbor relationships to obtain the three-dimensional spatial units corresponding to the visible region.

[0065] Step S1025: Combine the three-dimensional spatial units to obtain a local three-dimensional observation field.

[0066] In this embodiment, the spatial coordinates, color information, semantic category, observation time, observation source, and observation reliability of each three-dimensional spatial unit are recorded. The recorded three-dimensional spatial units are then combined according to their spatial positional relationships to form a local three-dimensional observation field representing the spatial structure of the currently visible area.

[0067] In this embodiment, the pixel coordinates of each frame of the panoramic video data are mapped to spherical direction vectors and transformed to a restricted spatial coordinate system. Then, through depth positioning, validity screening, and multi-frame fusion, three-dimensional spatial units are generated. This enables the accurate extraction of the three-dimensional spatial structure of the visible area from the panoramic video data, providing a reliable data foundation for the subsequent identification of invisible areas and reconstruction of three-dimensional scenes.

[0068] In conjunction with the above embodiments, in one implementation, step S103 may include: Step S1031: Determine the target space range that needs to be effectively observed and judged based on the boundary constraints of the confined space.

[0069] In this embodiment, the target spatial range refers to the three-dimensional spatial range defined by the physical boundary of the confined space, that is, the entire spatial area that needs to be determined whether it can be effectively observed by the panoramic video data.

[0070] Step S1032: Within the target space, determine the spatial occlusion relationship in the local three-dimensional observation field based on the panoramic line-of-sight projection relationship; determine the mask for occlusion-type invisible areas based on the spatial occlusion relationship; determine the mask for blind zone-type invisible areas based on the effective observation coverage of the panoramic camera in the confined space; determine the mask for degradation-type invisible areas based on image quality degradation characteristics; and determine the mask for historically missing-type invisible areas based on historical observation status. Specifically, occlusion-type invisible areas include spatial locations that cannot be effectively observed due to obstructions blocking the panoramic line of sight; blind zone-type invisible areas include spatial locations not within the effective observation coverage; degradation-type invisible areas include spatial locations where image quality does not meet the conditions for effective observation; and historically missing-type invisible areas include spatial locations that have not been effectively observed within a preset time range.

[0071] In this embodiment, for each three-dimensional spatial unit within the target space, masks for occlusion-type invisible regions, blind-zone-type invisible regions, degradation-type invisible regions, and historically missing-type invisible regions are generated based on whether the panoramic view is truncated by an entity, whether it is within the effective observation coverage of the panoramic camera, whether the corresponding image quality meets the effective observation conditions, and whether there are valid historical observation records within a preset time range. These masks for each type of invisible region are logically merged to obtain the mask for the invisible regions, and one or more type tags for the corresponding invisible regions are saved for each three-dimensional spatial unit.

[0072] Figure 2 This is a schematic diagram illustrating the principle of invisible region identification and 3D generation and reconstruction provided in an embodiment of this application. Figure 2 As shown, panoramic cameras have a large field of view, but various invisible areas may still exist in confined spaces. The identification process for each type of invisible area is explained below.

[0073] For occlusion-type invisible areas, determining the spatial occlusion relationship in the local three-dimensional observation field based on the panoramic line-of-sight projection relationship can include the following steps: Step A1: Based on the panoramic camera's acquisition pose and the spherical direction vectors corresponding to the pixel coordinates in the panoramic video data, establish multiple spatial rays pointing from the panoramic camera into the confined space.

[0074] In this embodiment, based on the panoramic camera's acquisition pose and the spherical viewing direction corresponding to the pixel coordinates of each frame in the panoramic video data, multiple spatial rays pointing from the panoramic camera into the confined space are established. The starting point of each spatial ray is the acquisition position of the panoramic camera, and the direction is determined by transforming the spherical direction vector of the corresponding pixel coordinates into the confined space coordinate system according to the panoramic camera's acquisition pose.

[0075] Step A2: Along each spatial ray, determine the first effective entity surface reached on each spatial ray in the local three-dimensional observation field, and identify the effective entity surface as the obstruction on the corresponding spatial ray.

[0076] In this embodiment, for each spatial ray, the effective entity surface that is first reached on the spatial ray is determined by combining at least one of the following: depth value, depth mutation result, semantic segmentation result, multi-frame geometric consistency result, and existing space occupancy state, and the effective entity surface is determined as the occupant on the current spatial ray.

[0077] Among these, depth mutation results refer to the detection results of significant jumps in depth values ​​between adjacent pixels along the spatial ray direction. Semantic segmentation results refer to the semantic categories to which each pixel belongs after semantic segmentation of each frame of panoramic video data, such as equipment, coal wall, support, personnel, etc. Multi-frame geometric consistency results refer to the verification results of whether the depth value and spatial position of the same 3D spatial location remain consistent across multiple consecutive frames. Existing spatial occupancy status refers to the spatial occupancy status determined based on the spatial position information of each 3D spatial unit in the constructed local 3D observation field, i.e., whether a certain spatial location has been occupied by a known 3D spatial unit. Valid entity surface refers to the surface that belongs to a real physical entity.

[0078] The obstruction can be equipment, personnel, vehicles, pipelines, support structures, or other entities within the confined space.

[0079] Step A3: Determine the location of the space behind the obstruction and within the target space as the obstructed space location.

[0080] In this embodiment, a three-dimensional spatial unit located behind an obstruction but still within the target space and which cannot be directly observed by other effective spatial rays or other effective observation perspectives at the current moment is identified as an obstructed spatial location.

[0081] In determining the location of an occluded space, errors in depth estimation, the thickness of the occluding object, and the discretized representation of space can lead to misidentification of 3D spatial units immediately behind the surface of the occluding object as occluded locations. To eliminate these misidentifications, a preset spatial tolerance can be set behind the surface of the occluding object. 3D spatial units within this tolerance range will be excluded from the list of occluded locations, and only those exceeding the tolerance range but meeting the occlusion criteria will be identified as occluded locations.

[0082] Furthermore, to avoid misjudgments of occlusion relationships due to occasional errors in depth estimation within a single frame, temporal verification of the recognition results can be performed. Specifically, when a 3D spatial unit is determined to be behind the same occluder in a consecutive preset number of frames, the 3D spatial unit is marked as an occluded spatial location; when the 3D spatial unit is determined to be behind an occluder only in a few frames, it is considered a misjudgment caused by occasional errors in depth estimation, and the 3D spatial unit is not marked as an occluded spatial location. When the observation perspective changes in subsequent panoramic video data, allowing the 3D spatial unit to be directly and effectively observed, the occlusion status of the 3D spatial unit is canceled, and the data is updated based on the new observation results.

[0083] Step A4: Establish the spatial occlusion relationship between each occluder and its corresponding occluded spatial location.

[0084] In this embodiment, a spatial occlusion relationship is established between each occluder and its corresponding occluded spatial location based on the judgment result.

[0085] Furthermore, masks for occluded invisible regions can be generated based on spatial occlusion relationships. Simultaneously, for each spatial location within an occluded invisible region, the corresponding occlusion type, occlusion start time, occlusion duration, and occlusion state reliability are recorded, providing a basis for the subsequent generation and reliability evaluation of candidate 3D spatial structures.

[0086] In this embodiment, a spatial ray is established pointing from the panoramic camera into the confined space, and the spatial occlusion relationship is determined along the ray direction, which can accurately identify occlusion-type invisible areas.

[0087] For blind-spot-type invisible areas, the three-dimensional effective coverage area of ​​the panoramic camera can be established based on the camera's installation location, effective imaging range, minimum near-field imaging distance, occlusion range of the installation structure, observation angle, and spatial sampling resolution. The three-dimensional effective coverage area refers to the three-dimensional spatial region that the panoramic camera can effectively observe.

[0088] For each three-dimensional spatial unit within the target space, determine whether it is within the effective three-dimensional coverage area. For three-dimensional spatial units that are not within the effective three-dimensional coverage area, they can be identified as blind-zone type invisible areas.

[0089] Specifically, a three-dimensional spatial unit not being within the effective three-dimensional coverage area includes at least one of the following situations: First, the three-dimensional spatial unit is located outside the effective imaging area of ​​the panoramic camera and cannot be projected onto the effective pixel area in the panoramic video data. Second, the three-dimensional spatial unit is located within the minimum imaging distance of the panoramic camera in the near field, or is blocked by the panoramic camera's bracket, protective cover, or the acquisition device itself, which makes it impossible for the panoramic camera to image it. Third, the observation angle between the three-dimensional spatial unit and the panoramic camera exceeds the preset angle threshold, resulting in the observation quality failing to meet the effective observation conditions. Fourth, the spatial sampling resolution corresponding to the three-dimensional spatial unit is lower than the preset resolution threshold, resulting in the inability to obtain observation information with sufficient accuracy.

[0090] Among them, invisible areas that have been identified as being formed by external entities are preferentially marked as occlusion-type invisible areas.

[0091] For degraded invisible areas, the influence of factors such as dust, smoke, weak light, strong reflection, motion blur, or image compression distortion on image quality can be used to identify degraded invisible areas within the observation range of the panoramic camera but where the image quality cannot meet the effective observation conditions.

[0092] In this embodiment, based on the spatial position of each three-dimensional spatial unit in the constrained spatial coordinate system and the acquisition pose of the panoramic camera, each three-dimensional spatial unit is projected onto the pixel plane of each frame of the panoramic video data to determine the image region corresponding to each three-dimensional spatial unit. For the image region corresponding to each three-dimensional spatial unit, various image quality indicators, including sharpness, brightness, contrast, dust or smoke characterization, motion blur degree, and compression distortion degree, are calculated. The above image quality indicators are normalized to the range of 0 to 1 and weighted and summed to obtain the comprehensive image quality score Q corresponding to the three-dimensional spatial unit. If the comprehensive image quality score Q corresponding to a certain three-dimensional spatial unit is lower than a preset effective observation threshold, it indicates that the image quality of the image region corresponding to the three-dimensional spatial unit cannot meet the effective observation conditions. To avoid misjudgment caused by occasional image quality fluctuations in a single frame, if this state is maintained continuously in a preset number of consecutive frame images, the three-dimensional spatial unit is determined as a degraded invisible region. Furthermore, based on the specific indicators in each image quality index that are below the corresponding threshold, the causes of degradation can be marked as dust, smoke, weak light, strong reflection, motion blur, or compression distortion, so that corresponding 3D generation and reconstruction strategies can be adopted for different causes of degradation in the future.

[0093] For historically missing invisible regions, based on historical observations, if at least one of the following conditions is met, the corresponding three-dimensional spatial unit is identified as a historically missing invisible region: First, the three-dimensional spatial unit is not effectively observed at the current moment, and the time interval between the current moment and the most recent effective observation moment recorded in the historical observation state database for the three-dimensional spatial unit exceeds a preset time threshold. Second, the credibility of historical observations recorded by the three-dimensional spatial unit in the historical observation state database is consistently lower than the preset credibility threshold. Third, the three-dimensional spatial unit has no valid observation record in the historical observation state database, meaning it has not been effectively observed within the preset time range.

[0094] Step S1033: Based on the obtained masks of each invisible region, generate a mask for the invisible regions in the confined space that are not effectively observed by the panoramic video data.

[0095] In this embodiment, the masks for occlusion-type invisible regions, blind-zone-type invisible regions, degradation-type invisible regions, and history-missing invisible regions identified in step S1032 are logically merged to obtain a mask for invisible regions in the confined space. The same three-dimensional spatial unit may belong to multiple invisible region types simultaneously; when generating the mask for the invisible region, the type labels of all invisible regions corresponding to that three-dimensional spatial unit are retained.

[0096] In this embodiment, the target space range is determined based on the boundary constraints of the confined space. Within the target space range, four types of invisible regions—occlusion type, blind spot type, degradation type, and historical missing type—are identified based on the panoramic line-of-sight projection relationship, effective observation coverage range, image quality degradation characteristics, and historical observation status. This can effectively improve the identification accuracy of invisible regions in the confined space and the quality of the 3D scene reconstruction model.

[0097] In conjunction with the above embodiments, in one implementation, step S105 may include: Step S1051: Determine the confidence value corresponding to each candidate three-dimensional spatial structure.

[0098] In this embodiment, the confidence value is used to characterize the degree of matching between the candidate three-dimensional spatial structure and the actual spatial structure in the confined space.

[0099] The confidence value ranges from 0 to 1. The higher the confidence value, the closer the candidate three-dimensional spatial structure is to the actual spatial structure in the confined space.

[0100] Step S1052: Based on the target constraints, judge each candidate three-dimensional spatial structure and obtain the judgment result. The target constraints include at least one of the following: boundary constraints of the confined space, physical accessibility constraints, neighborhood geometric continuity constraints, historical observation consistency constraints, and semantic topological constraints. Physical accessibility constraints mean that the candidate three-dimensional spatial structure must conform to the actual physical layout of the confined space. Neighborhood geometric continuity constraints mean that the candidate three-dimensional spatial structure must maintain geometric continuity with the adjacent visible area. Historical observation consistency constraints mean that the candidate three-dimensional spatial structure must be consistent with the historical three-dimensional spatial structure that has been effectively observed. Semantic topological constraints mean that the candidate three-dimensional spatial structure must be consistent with the local three-dimensional observation field in terms of semantic category and spatial location.

[0101] For example, taking a fully mechanized coal mining face as an example, the boundary constraints of the confined space require that the candidate three-dimensional spatial structure does not exceed the spatial range defined by the coal wall, roof, floor, and roadway boundaries; the physical accessibility constraints require that the candidate three-dimensional spatial structure conforms to the layout range of hydraulic supports, the movement trajectory of the coal mining machine, and the width limitations of the passageway; the neighborhood geometric continuity constraints require that the candidate three-dimensional spatial structure maintain geometric continuity with the coal wall surface, roof surface, or floor surface of the adjacent visible area; the historical observation consistency constraints require that the candidate three-dimensional spatial structure be consistent with the three-dimensional spatial structure at the same spatial location that has been effectively observed in history in terms of geometry, semantic category, and topological relationship; the semantic topology constraints require that the semantic category and relative spatial position relationship between the spatial objects in the candidate three-dimensional spatial structure conform to the actual layout of the confined space, for example, the hydraulic supports should be located between the roof and floor, and the scraper conveyor should be arranged linearly along the working face direction, etc.

[0102] In this embodiment, when judging candidate three-dimensional spatial structures based on historical observation consistency constraints, the following two cases can be distinguished: When valid observation records exist in an invisible region at a historical moment, the candidate 3D spatial structure is matched with the real 3D spatial structure that was validly observed at that historical moment, and the geometric error, semantic error, and topological error between the two are calculated. If none of the above errors are greater than the corresponding preset error threshold, the candidate 3D spatial structure is judged to satisfy the historical observation consistency constraint; if at least one of the above errors is greater than the corresponding preset error threshold, the candidate 3D spatial structure is judged to not satisfy the historical observation consistency constraint.

[0103] When there are no valid historical observation records in the invisible region within a preset time range, it is impossible to judge the candidate 3D spatial structure based on the historical observation consistency constraint. In this case, the weight of other constraints in the judgment process can be increased to compensate for the lack of historical observation information.

[0104] Step S1053: For each candidate three-dimensional spatial structure, if the judgment result indicates that the target constraint condition is not met, the target operation is performed on the candidate three-dimensional spatial structure. The target operation includes at least one of the following: deletion operation, reducing the corresponding confidence value, and optimization operation.

[0105] In this embodiment, the deletion operation refers to eliminating candidate three-dimensional spatial structures that do not meet the target constraints.

[0106] Lowering the corresponding credibility value means reducing the credibility value of the candidate 3D spatial structure based on the degree to which the candidate 3D spatial structure does not meet the target constraints.

[0107] Optimization refers to making local adjustments to the candidate three-dimensional spatial structure so that it meets the target constraints after adjustment.

[0108] For example, if a candidate 3D spatial structure exceeds the boundary constraints of the confined space, it can be deleted; if the geometric continuity deviation between a candidate 3D spatial structure and the adjacent visible region is small, it can be optimized to adjust the connection relationship; if a candidate 3D spatial structure has a certain deviation from the historically observed 3D spatial structure but is not completely contradictory, its confidence value can be reduced.

[0109] In addition, target operations may also include temporary storage operations, which temporarily store candidate 3D spatial structures that do not meet the target constraints as low-confidence candidate results, so that when the corresponding invisible area is re-observed in subsequent panoramic video data, the re-observed real 3D spatial structure can be used to verify and update the low-confidence candidate results.

[0110] Step S1054: Among the candidate three-dimensional spatial structures obtained after performing the target operation, determine the target three-dimensional spatial structure of the invisible region.

[0111] In this embodiment, among the candidate three-dimensional spatial structures obtained after performing the target operation, the target three-dimensional spatial structure of the invisible region is determined according to the confidence value corresponding to each candidate three-dimensional spatial structure.

[0112] For example, when there are multiple candidate 3D spatial structures in the same invisible region, the candidate 3D spatial structure with the highest confidence value can be selected as the target 3D spatial structure of the invisible region. Alternatively, multiple candidate 3D spatial structures with confidence values ​​all higher than a preset confidence threshold can be weighted and fused to obtain the target 3D spatial structure of the invisible region.

[0113] In this embodiment, the candidate 3D spatial structure is judged and processed based on the boundary constraints, physical accessibility constraints, neighborhood geometric continuity constraints, historical observation consistency constraints, and semantic topological constraints of the confined space. This ensures that the target 3D spatial structure matches the actual spatial structure of the confined space, thereby improving the reliability of the 3D generation and reconstruction results of the invisible region.

[0114] In conjunction with the above embodiments, in one implementation, step S106 may include: Step S1061: Fuse the local three-dimensional observation field and the target three-dimensional spatial structure according to the target representation form to obtain the initial three-dimensional scene reconstruction model of the confined space. The target representation form includes any of the following: point cloud, mesh, voxel, depth field, implicit field and three-dimensional Gaussian set.

[0115] In this embodiment, the target three-dimensional spatial structure corresponding to the invisible area and the local three-dimensional observation field corresponding to the visible area are fused according to a unified target representation form to obtain an initial three-dimensional scene reconstruction model of the confined space.

[0116] During the fusion process, the original observation results are retained first in the real observation area (local 3D observation field), and the target 3D spatial structure is connected and completed with the real observation area based on the confidence value and spatial consistency. The resulting initial 3D scene reconstruction model includes not only the real observation area obtained directly from the panoramic video data, but also the reconstructed 3D spatial structure of the invisible areas.

[0117] In actual implementation, the target representation can be set according to actual needs, and this embodiment does not impose specific restrictions on it.

[0118] Step S1062: In the initial 3D scene reconstruction model, add distinguishing markers to the local 3D observation field and the target 3D spatial structure, and add confidence markers to each target 3D spatial structure to obtain the 3D scene reconstruction model.

[0119] In this embodiment, the distinguishing markers are used to explicitly differentiate spatial structures from different sources in the initial 3D scene reconstruction model. Specifically, the spatial structures of the local 3D observation field are marked as the real observation region, and the target 3D spatial structures are marked as the generated reconstruction region.

[0120] In this embodiment, the confidence level marker is used to characterize the confidence level of each target's three-dimensional spatial structure. Based on the confidence level value corresponding to each target's three-dimensional spatial structure determined in step S1051, the generated reconstruction region can be divided into a high-confidence generated region, a medium-confidence region to be verified, and a low-confidence candidate region. Specifically, the confidence level of the high-confidence generated region is greater than a preset first confidence level threshold, and it can directly participate in the expression of the three-dimensional scene reconstruction model; the confidence level of the medium-confidence region to be verified is greater than a preset second confidence level threshold but less than or equal to the first confidence level threshold, and it can be retained as a temporary generation result and verified and updated when the corresponding region is re-observed in subsequent panoramic video data; the confidence level of the low-confidence candidate region is less than or equal to the preset second confidence level threshold, and it does not participate in the determination of the three-dimensional scene reconstruction model, or it is only saved as auxiliary candidate information.

[0121] In this embodiment, the local three-dimensional observation field and the target three-dimensional spatial structure are fused in a unified representation form, and a distinguishing marker and a confidence marker are added to the fused three-dimensional scene reconstruction model. This enables the explicit differentiation between the real observation area and the generated reconstruction area while realizing the complete three-dimensional scene reconstruction in a confined space. It also provides confidence level information for the generated reconstruction area, allowing downstream applications to adopt differentiated processing strategies based on the source and confidence level of different areas, thereby improving the reliability of the confined space three-dimensional scene reconstruction model.

[0122] In conjunction with the above embodiments, in one implementation, step S1051 may include: Step B1: For each candidate 3D spatial structure, obtain multiple credibility evaluation factors. These factors include several of the following: the type of invisible region, the image quality corresponding to the panoramic video data, the duration during which the invisible region has not been effectively observed, the consistency between the candidate 3D spatial structure and historical 3D spatial structures that have been effectively observed, the geometric continuity between the candidate 3D spatial structure and adjacent visible regions in the local 3D observation field, the consistency between the candidate 3D spatial structure and the prior features of the structure, the consistency between the candidate 3D spatial structure and the local 3D observation field in terms of semantic topological relationships, the uncertainty of the output of the 3D scene generation model, the consistency between multiple candidate 3D spatial structures, and the degree of consistency between the candidate 3D spatial structure and real observation data at different times or from different perspectives.

[0123] Step B2: For each candidate three-dimensional spatial structure, determine the credibility score of each credibility assessment factor, and perform weighted fusion of the credibility scores of each credibility assessment factor to obtain the credibility value of the candidate three-dimensional spatial structure.

[0124] Specifically, the credibility scores of each credibility assessment factor are weighted and fused to obtain the credibility value of the candidate three-dimensional spatial structure, which may include: Step B21: Weight and fuse the credibility scores of each credibility assessment factor to obtain the fusion result.

[0125] Step B22: Adjust the fusion result according to the correction coefficient corresponding to the type of invisible region and the joint gating value corresponding to the boundary constraints and physical accessibility constraints of the restricted space to obtain the credibility value of the candidate three-dimensional spatial structure. When the candidate three-dimensional spatial structure does not satisfy the boundary constraints or physical accessibility constraints of the restricted space, the joint gating value is zero.

[0126] In this embodiment, candidate three-dimensional spatial structure credibility value Calculate as follows: ; in, Representing candidate three-dimensional spatial structure The credibility value; Indicates the first The credibility score of each credibility assessment factor; Indicates the first The weights corresponding to the credibility assessment factors; Indicate the first The credibility assessment factor is determined by whether it exists; if it exists, the value is 1, and if it does not exist, the value is 0. This indicates the correction factor corresponding to the type of invisible region. This represents the joint gating value corresponding to the boundary constraints and physical reachability constraints of the confined space.

[0127] If the candidate three-dimensional spatial structure exceeds the boundary constraints of the confined space, or violates physical accessibility constraints such as equipment layout, support boundaries, access range, or movement trajectory, Set the value to 0, and discard or mark the candidate 3D spatial structure as an invalid candidate.

[0128] In this embodiment, multiple credibility evaluation factors are obtained and the credibility scores of each factor are weighted and fused. At the same time, an invisible region type correction coefficient and a joint gating value of boundary and physical reachability are introduced for adjustment. This can effectively intercept candidate three-dimensional spatial structures that do not meet the requirements and improve the reliability of the final target three-dimensional spatial structure.

[0129] In conjunction with the above embodiments, in one implementation, step S101 may include: Step S1011: Obtain the original panoramic video data within the confined space.

[0130] In this embodiment, raw panoramic video data refers to unprocessed continuous panoramic video data directly acquired by the panoramic video acquisition device within a confined space.

[0131] Step S1012: Perform preprocessing operations on the raw panoramic video data to obtain panoramic video data. The preprocessing operations include at least one of the following: Distortion correction is used to eliminate optical distortion introduced by panoramic lenses; Spherical mapping is used to map the pixel coordinates of each frame in panoramic video data to the corresponding spherical direction vector; Temporal alignment is used to align frames from different times in panoramic video data to a unified time reference. Acquire pose estimation to determine the pose of the panoramic camera in a confined space corresponding to each frame of the panoramic video data.

[0132] In this embodiment, distortion correction can be performed on each frame of the original panoramic video data according to the calibration parameters of the panoramic lens to eliminate the influence of optical distortions such as barrel distortion, pincushion distortion or fisheye distortion on the subsequent three-dimensional spatial positioning accuracy.

[0133] Spherical mapping can convert the pixel coordinates of each frame of an image into corresponding spherical orientation vectors based on the projection model of a panoramic camera, in order to establish the correspondence between pixels and spatial observation directions.

[0134] Timing alignment can align images from different times to a unified time base based on timestamps, frame rates, or synchronization signals, thereby ensuring timing consistency among multiple images.

[0135] Acquisition pose estimation can determine the acquisition pose of the panoramic camera in the confined space corresponding to each frame of the image based on inertial measurement information, odometry information, calibration point information, or visual pose estimation results, so as to facilitate the subsequent unified conversion of the observation data of each frame of the image to the confined space coordinate system.

[0136] In this embodiment, preprocessing operations such as distortion correction, spherical mapping, temporal alignment, and acquisition pose estimation are performed on the original panoramic video data, which can significantly improve the reliability of the panoramic video data and thus improve the accuracy of the 3D scene reconstruction model.

[0137] When the method of this application is applied to a fully mechanized coal mining face, panoramic cameras can be deployed in roadways, at the face end, on inspection robots, or on mobile inspection platforms. Due to the influence of factors such as hydraulic supports, coal mining machines, scraper conveyors, dust, and low light, areas such as the coal wall, roof, floor, areas behind supports, or areas obscured by equipment may not be directly and effectively observed by panoramic video data. In this case, prior structural features such as adjacent visible coal walls, adjacent hydraulic support structures, support numbers and spacing, coal mining machine movement trajectory, historical observation status, and the direction of face advancement can be used to generate and reconstruct 3D models of the invisible areas, and output a complete 3D scene reconstruction model of the working face with confidence markers. This 3D scene reconstruction model can be used for digital twin display, remote inspection, robot navigation path planning, safety risk identification, and emergency response decision support.

[0138] When the method of this application is applied to underground utility tunnels or other similar scenarios, the prior structural features of the confined space may include the tunnel boundary, passageway orientation, pipeline arrangement, wall continuity, ceiling and floor constraints, and equipment installation locations. For invisible areas caused by pipeline obstruction, limited viewing angles of inspection equipment, insufficient lighting, or missing historical data, candidate 3D spatial structures can be generated based on the local 3D observation field and prior structural features. The selected target 3D spatial structures are then fused with the local 3D observation field to obtain a complete 3D scene reconstruction model with credibility labels.

[0139] Figure 3 This is a schematic diagram illustrating the implementation process of the confined space three-dimensional scene reconstruction method provided in the embodiments of this application. Figure 3 In this context, steps 1-10 are a detailed elaboration of steps S101-S106 from the preceding text. For example... Figure 3As shown, firstly, panoramic video data within the confined space is acquired (Step 1), and preprocessed using methods such as distortion correction, spherical mapping, temporal alignment, and acquisition pose estimation (Step 2). Next, a local 3D observation field is constructed based on the visible area (Step 3). Then, four types of invisible regions—occlusion type, blind spot type, degradation type, and historical missing type—are identified, and masks for these invisible regions are generated (Step 4). Next, features of the local 3D observation field (including geometric features, semantic features, texture features, and temporal variation features) and prior structural features of the confined space (including spatial boundaries, equipment arrangement, and passageway orientation) are extracted (Step 5). Finally, this information is combined with a historical observation state database and a priori structural database of the confined space. The data is input into the 3D scene generation model (step 6) to generate candidate 3D spatial structures for invisible regions (step 7); the candidate 3D spatial structures are screened, optimized, and evaluated for credibility based on boundary constraints, neighborhood geometric continuity constraints, historical observation consistency constraints, semantic topological constraints, and physical reachability constraints (step 8); the local 3D observation field is fused with the screened target 3D spatial structures to obtain the initial 3D scene reconstruction model (step 9); finally, distinguishing and credibility markers are added to mark each spatial region as a real observation region, a high-credibility generation region, a medium-credibility region to be verified region, or a low-credibility candidate region to obtain the final 3D scene reconstruction model (step 10).

[0140] In summary, the method of this application includes at least the following technical effects: First, this application identifies invisible areas in the confined space that have not been effectively observed by panoramic video data based on the panoramic line-of-sight projection relationship, image quality degradation characteristics, boundary constraints of the confined space, and historical observation status. The invisible areas are classified into four types: occlusion type, blind spot type, degradation type, and historical missing type, and are labeled accordingly. This enables comprehensive coverage and classification of invisible areas in the confined space caused by different reasons, avoiding the problem of existing panoramic video 3D reconstruction methods simply treating invisible areas as holes or missing areas.

[0141] Second, this application extracts the structural prior features of the confined space and uses the features of the local three-dimensional observation field, the mask of the invisible area, the historical observation state, and the structural prior features as conditions to input the three-dimensional scene generation model. This enables the three-dimensional generation and reconstruction of the invisible area, allowing the generation results to make full use of the inherent structural laws of the confined space. This solves the problem that existing point cloud completion, mesh repair, and three-dimensional generation methods lack the utilization of the structural laws of the confined space and the generation results are prone to not conforming to the actual spatial relationships.

[0142] Third, this application uses boundary constraints, physical accessibility constraints, neighborhood geometric continuity constraints, historical observation consistency constraints, and semantic topological constraints of the confined space to screen and optimize candidate 3D spatial structures. This enables multi-dimensional constraint verification of the generated results, ensuring that the final determined target 3D spatial structure conforms to the actual spatial relationship of the confined space, and improving the reliability of the 3D generation and reconstruction results of invisible areas.

[0143] Fourth, this application adds distinguishing markers to the local 3D observation field and the target 3D spatial structure in the 3D scene reconstruction model, and adds credibility markers to each target 3D spatial structure. The generated reconstruction region is divided into high credibility generated region, medium credibility unverified region and low credibility candidate region. This enables explicit distinction between the real observation region and the generated reconstruction region, as well as credibility classification of the generated reconstruction region. This allows downstream applications to clearly distinguish which spatial structures come from real observation and which come from model inference, solving the problem that existing 3D completion methods lack explicit distinction between real observation and generated reconstruction and credibility markers.

[0144] Fifth, this application introduces historical observation status and historical observation consistency constraints. During the generation and screening of candidate three-dimensional spatial structures, it uses real three-dimensional spatial structures that have been effectively observed in historical moments for constraint and verification. When invisible areas are re-observed in subsequent panoramic video data, the generated reconstruction results are updated. This enables the three-dimensional scene reconstruction model to be continuously updated with observation data, improving the timeliness and reliability of the three-dimensional scene reconstruction model in confined space under long-term operating conditions.

[0145] The confined space three-dimensional scene reconstruction device provided in this application is described below. The confined space three-dimensional scene reconstruction device described below and the confined space three-dimensional scene reconstruction method described above can be referred to in correspondence.

[0146] Figure 4 This is a structural block diagram of a confined space three-dimensional scene reconstruction device shown in an embodiment of this application. Figure 4 As shown, the confined space three-dimensional scene reconstruction device 400 of this application includes: Acquisition module 401 is used to acquire panoramic video data within a confined space; The construction module 402 is used to construct a local three-dimensional observation field of the confined space based on the visible area in the panoramic video data. The local three-dimensional observation field is used to characterize the spatial structure in the confined space that has been effectively observed by the panoramic video data. The generation module 403 is used to generate a mask for invisible areas in the confined space that are not effectively observed by the panoramic video data, based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation status of the confined space. Input module 404 is used to input the features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the structural prior features of the confined space as conditions into the three-dimensional scene generation model to obtain the candidate three-dimensional spatial structure of the invisible region; The filtering module 405 is used to filter the candidate three-dimensional spatial structures to obtain the target three-dimensional spatial structure of the invisible area; The fusion module 406 is used to fuse the local three-dimensional observation field and the target three-dimensional spatial structure to obtain a three-dimensional scene reconstruction model of the confined space.

[0147] According to the confined space three-dimensional scene reconstruction device 400 provided in this application, the construction module 402 is used to: obtain the spherical direction vectors corresponding to the pixel coordinates of each frame image in the panoramic video data; convert each of the spherical direction vectors to a confined space coordinate system, wherein the confined space coordinate system is a three-dimensional coordinate system established according to the physical structure of the confined space; perform three-dimensional spatial positioning on each of the pixel coordinates in the visible area to obtain the three-dimensional spatial position corresponding to each of the pixel coordinates; generate a three-dimensional spatial unit corresponding to the visible area based on each of the three-dimensional spatial positions, wherein the three-dimensional spatial unit is the smallest spatial element representing the spatial structure that has been effectively observed in the confined space; and combine the three-dimensional spatial units to obtain the local three-dimensional observation field.

[0148] According to the confined space 3D scene reconstruction device 400 provided in this application, the generation module 403 is used to: determine the target space range that needs to be effectively observed based on the boundary constraints of the confined space; within the target space range, determine the spatial occlusion relationship in the local 3D observation field based on the panoramic view projection relationship, determine the mask of the occlusion-type invisible area based on the spatial occlusion relationship, determine the mask of the blind zone-type invisible area based on the effective observation coverage range of the panoramic camera in the confined space, determine the mask of the degradation-type invisible area based on the image quality degradation characteristics, and determine the mask of the historical missing-type invisible area based on the historical observation status, wherein the occlusion-type invisible area includes spatial locations that cannot be effectively observed due to obstruction of the panoramic view, the blind zone-type invisible area includes spatial locations that are not within the effective observation coverage range, the degradation-type invisible area includes spatial locations whose image quality does not meet the effective observation conditions, and the historical missing-type invisible area includes spatial locations that have not been effectively observed within a preset time range; and generate a mask of the invisible areas in the confined space that have not been effectively observed by the panoramic video data based on the obtained masks of each invisible area.

[0149] According to the confined space three-dimensional scene reconstruction device 400 provided in this application, the generation module 403 is used to: establish multiple spatial rays pointing from the panoramic camera to the interior of the confined space based on the acquisition pose of the panoramic camera and the spherical direction vector corresponding to the pixel coordinates in the panoramic video data; determine the effective entity surface first reached on each spatial ray in the local three-dimensional observation field along each spatial ray, and determine the effective entity surface as the occlusion object on the corresponding spatial ray; determine the spatial position located behind the occlusion object and within the target space range as the occluded spatial position; and establish the spatial occlusion relationship between each occlusion object and the corresponding occluded spatial position.

[0150] According to the confined space 3D scene reconstruction device 400 provided in this application, the screening module 405 is used to: determine the confidence value corresponding to each of the candidate 3D spatial structures; judge each of the candidate 3D spatial structures based on target constraints to obtain a judgment result, wherein the target constraints include at least one of the following: boundary constraints, physical accessibility constraints, neighborhood geometric continuity constraints, historical observation consistency constraints, and semantic topological constraints of the confined space, wherein the physical accessibility constraints indicate that the candidate 3D spatial structure must conform to the actual physical layout of the confined space, the neighborhood geometric continuity constraints indicate that the candidate 3D spatial structure must maintain geometric continuity with adjacent visible areas, the historical observation consistency constraints indicate that the candidate 3D spatial structure must be consistent with the historical 3D spatial structure that has been effectively observed, and the semantic topological constraints indicate that the candidate 3D spatial structure must be consistent with the local 3D observation field in terms of semantic category and spatial location; for each of the candidate 3D spatial structures, if the judgment result indicates that the target constraints are not met, perform a target operation on the candidate 3D spatial structure, wherein the target operation includes at least one of the following: deletion operation, reducing the corresponding confidence value, and optimization operation. Among the candidate three-dimensional spatial structures obtained after performing the target operation, the target three-dimensional spatial structure of the invisible region is determined.

[0151] According to the confined space 3D scene reconstruction device 400 provided in this application, the fusion module 406 is used to: fuse the local 3D observation field and the target 3D spatial structure according to the target representation form to obtain an initial 3D scene reconstruction model of the confined space, wherein the target representation form includes any one of the following: point cloud, mesh, voxel, depth field, implicit field and 3D Gaussian set; in the initial 3D scene reconstruction model, add distinguishing markers to the local 3D observation field and the target 3D spatial structure, and add confidence markers to each of the target 3D spatial structures to obtain the 3D scene reconstruction model.

[0152] According to the confined space 3D scene reconstruction device 400 provided in this application, the screening module 405 is used to: acquire multiple credibility evaluation factors for each candidate 3D spatial structure, the multiple credibility evaluation factors including multiple of the following factors: the type of the invisible region, the image quality corresponding to the panoramic video data, the duration during which the invisible region has not been effectively observed, the consistency between the candidate 3D spatial structure and the historical 3D spatial structure that has been effectively observed, the geometric continuity between the candidate 3D spatial structure and the adjacent visible region in the local 3D observation field, the consistency between the candidate 3D spatial structure and the prior features of the structure, the consistency between the candidate 3D spatial structure and the local 3D observation field in terms of semantic topological relationship, the uncertainty of the output of the 3D scene generation model, the consistency among multiple candidate 3D spatial structures, and the degree of consistency between the candidate 3D spatial structure and real observation data at different times or from different perspectives; for each candidate 3D spatial structure, determine the credibility score of each credibility evaluation factor, and perform weighted fusion of the credibility scores of each credibility evaluation factor to obtain the credibility value of the candidate 3D spatial structure.

[0153] According to the confined space 3D scene reconstruction device 400 provided in this application, the screening module 405 is used to: perform weighted fusion of the credibility scores of each credibility evaluation factor to obtain a fusion result; adjust the fusion result according to the correction coefficient corresponding to the type of the invisible region and the joint gating value corresponding to the boundary constraint and physical accessibility constraint of the confined space to obtain the credibility value of the candidate 3D space structure, wherein the joint gating value is zero when the candidate 3D space structure does not satisfy the boundary constraint or the physical accessibility constraint of the confined space.

[0154] According to the confined space 3D scene reconstruction device 400 provided in this application, the acquisition module 401 is used to: acquire original panoramic video data within the confined space; perform preprocessing operations on the original panoramic video data to obtain the panoramic video data, wherein the preprocessing operations include at least one of the following: distortion correction, used to eliminate optical distortion introduced by the panoramic lens; spherical mapping, used to map the pixel coordinates of each frame image in the panoramic video data to the corresponding spherical direction vector; temporal alignment, used to align each frame image at different times in the panoramic video data to a unified time reference; and acquisition pose estimation, used to determine the pose of the panoramic camera corresponding to each frame image in the panoramic video data in the confined space.

[0155] Figure 5 This is a schematic diagram of the physical structure of an electronic device shown in an embodiment of this application, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a method for reconstructing a three-dimensional scene in a confined space.

[0156] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute a confined space three-dimensional scene reconstruction method provided by the above methods.

[0158] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform a confined space three-dimensional scene reconstruction method provided by the above methods.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for reconstructing a three-dimensional scene in a confined space, characterized in that, include: Acquire panoramic video data within a confined space; Based on the visible area in the panoramic video data, a local three-dimensional observation field is constructed for the confined space. The local three-dimensional observation field is used to characterize the spatial structure in the confined space that has been effectively observed by the panoramic video data. Based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation status of the confined space, a mask is generated for the invisible areas in the confined space that are not effectively observed by the panoramic video data. The features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the structural prior features of the confined space are used as conditions to input the three-dimensional scene generation model to obtain the candidate three-dimensional spatial structure of the invisible region. The candidate three-dimensional spatial structures are screened to obtain the target three-dimensional spatial structure of the invisible region; The local three-dimensional observation field and the target three-dimensional spatial structure are fused to obtain a three-dimensional scene reconstruction model of the confined space.

2. The method for reconstructing a three-dimensional scene in a confined space according to claim 1, characterized in that, The step of constructing a local three-dimensional observation field for the confined space based on the visible area in the panoramic video data includes: Obtain the spherical orientation vector corresponding to the pixel coordinates of each frame in the panoramic video data; The spherical direction vectors are transformed to a confined space coordinate system, which is a three-dimensional coordinate system established based on the physical structure of the confined space. The three-dimensional spatial positioning of each pixel coordinate in the visible area is performed to obtain the three-dimensional spatial position corresponding to each pixel coordinate; Based on each of the three-dimensional spatial positions, a three-dimensional spatial unit corresponding to the visible area is generated. The three-dimensional spatial unit is the smallest spatial element that represents the spatial structure that has been effectively observed in the confined space. The three-dimensional spatial units are combined to obtain the local three-dimensional observation field.

3. The method for reconstructing a three-dimensional scene in a confined space according to claim 1, characterized in that, The step of generating a mask for invisible areas in the confined space that were not effectively observed by the panoramic video data, based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation state of the confined space, includes: Based on the boundary constraints of the confined space, the target space range that needs to be effectively observed and judged is determined; Within the target space, spatial occlusion relationships in the local three-dimensional observation field are determined based on the panoramic view projection relationship. Masks for occlusion-type invisible areas are determined based on the spatial occlusion relationships. Masks for blind-zone-type invisible areas are determined based on the effective observation coverage of the panoramic camera in the confined space. Masks for degraded-type invisible areas are determined based on the image quality degradation characteristics. Masks for historically missing-type invisible areas are determined based on the historical observation status. The occlusion-type invisible areas include spatial locations that cannot be effectively observed due to obstructions blocking the panoramic view. The blind-zone-type invisible areas include spatial locations not within the effective observation coverage. The degraded-type invisible areas include spatial locations where the image quality does not meet the effective observation conditions. The historically missing-type invisible areas include spatial locations that have not been effectively observed within a preset time range. Based on the obtained masks of each invisible region, a mask is generated for the invisible regions in the confined space that are not effectively observed by the panoramic video data.

4. The method for reconstructing a three-dimensional scene in a confined space according to claim 3, characterized in that, Determining the spatial occlusion relationship in the local three-dimensional observation field based on the panoramic line-of-sight projection relationship includes: Based on the acquisition pose of the panoramic camera and the spherical direction vector corresponding to the pixel coordinates in the panoramic video data, multiple spatial rays pointing from the panoramic camera into the confined space are established. Along each of the spatial rays, determine the first effective entity surface reached on each of the spatial rays in the local three-dimensional observation field, and define the effective entity surface as the obstruction on the corresponding spatial ray; The spatial location located behind the obstruction and within the target space is defined as the obstructed spatial location. Establish the spatial occlusion relationship between each of the aforementioned occluding objects and the corresponding occluded spatial location.

5. The method for reconstructing a three-dimensional scene in a confined space according to claim 1, characterized in that, The step of filtering the candidate three-dimensional spatial structures to obtain the target three-dimensional spatial structure of the invisible region includes: Determine the confidence value corresponding to each of the candidate three-dimensional spatial structures; The candidate 3D spatial structures are judged based on the target constraints to obtain the judgment result. The target constraints include at least one of the following: boundary constraints of the restricted space, physical accessibility constraints, neighborhood geometric continuity constraints, historical observation consistency constraints, and semantic topological constraints. The physical accessibility constraints indicate that the candidate 3D spatial structure must conform to the actual physical layout of the restricted space. The neighborhood geometric continuity constraints indicate that the candidate 3D spatial structure must maintain geometric continuity with the adjacent visible areas. The historical observation consistency constraints indicate that the candidate 3D spatial structure must be consistent with the historical 3D spatial structure that has been effectively observed. The semantic topological constraints indicate that the candidate 3D spatial structure must be consistent with the local 3D observation field in terms of semantic category and spatial location. For each of the candidate three-dimensional spatial structures, if the judgment result indicates that the target constraint condition is not met, a target operation is performed on the candidate three-dimensional spatial structure. The target operation includes at least one of the following: deletion operation, reducing the corresponding confidence value, and optimization operation. Among the candidate three-dimensional spatial structures obtained after performing the target operation, the target three-dimensional spatial structure of the invisible region is determined.

6. The method for reconstructing a three-dimensional scene in a confined space according to claim 5, characterized in that, Determining the confidence value corresponding to each of the candidate three-dimensional spatial structures includes: For each of the candidate 3D spatial structures, multiple credibility evaluation factors are obtained. These multiple credibility evaluation factors include several of the following factors: the type of the invisible region, the image quality corresponding to the panoramic video data, the duration during which the invisible region has not been effectively observed, the consistency between the candidate 3D spatial structure and the historical 3D spatial structure that has been effectively observed, the geometric continuity between the candidate 3D spatial structure and the adjacent visible regions in the local 3D observation field, the consistency between the candidate 3D spatial structure and the prior features of the structure, the consistency between the candidate 3D spatial structure and the local 3D observation field in terms of semantic topological relationship, the uncertainty of the output of the 3D scene generation model, the consistency among multiple candidate 3D spatial structures, and the degree of consistency between the candidate 3D spatial structure and the real observation data at different times or from different perspectives. For each of the candidate three-dimensional spatial structures, the credibility score of each credibility evaluation factor is determined, and the credibility scores of each credibility evaluation factor are weighted and fused to obtain the credibility value of the candidate three-dimensional spatial structure.

7. A confined space three-dimensional scene reconstruction device, characterized in that, include: The acquisition module is used to acquire panoramic video data within a confined space. A construction module is used to construct a local three-dimensional observation field of the confined space based on the visible area in the panoramic video data. The local three-dimensional observation field is used to characterize the spatial structure in the confined space that has been effectively observed by the panoramic video data. The generation module is used to generate a mask for invisible areas in the confined space that are not effectively observed by the panoramic video data, based on the panoramic view projection relationship corresponding to the panoramic video data, the image quality degradation characteristics of the panoramic video data, the boundary constraints of the confined space, and the historical observation status of the confined space. The input module is used to input the features of the local three-dimensional observation field, the mask of the invisible region, the historical observation state, and the structural prior features of the confined space as conditions into the three-dimensional scene generation model to obtain the candidate three-dimensional spatial structure of the invisible region. The filtering module is used to filter the candidate three-dimensional spatial structures to obtain the target three-dimensional spatial structure of the invisible region; The fusion module is used to fuse the local three-dimensional observation field and the target three-dimensional spatial structure to obtain a three-dimensional scene reconstruction model of the confined space.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements a confined space three-dimensional scene reconstruction method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a confined space three-dimensional scene reconstruction method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a confined space three-dimensional scene reconstruction method as described in any one of claims 1 to 6.

Images