Method and device for three-dimensional inventory of traffic infrastructure based on sparse images

Abstract

The application discloses a traffic infrastructure three-dimensional asset inventory method and device based on sparse images. The method comprises the following steps: acquiring panoramic images and corresponding pose data, extracting visual semantic features and geometric ray features of each observation target; correlating cross-view observation targets to generate target observation chains; performing high-precision geometric calculation on the target observation chains to obtain three-dimensional center coordinates and three-dimensional physical dimensions of the observation targets, and performing uncertainty evaluation to obtain an uncertainty score; executing a hierarchical response according to the uncertainty score; selecting a frame of image with the best visual quality in the target observation chain, generating a normalized three-dimensional grid model based on the image, performing spatial transformation on the normalized three-dimensional grid model based on the three-dimensional center coordinates and the three-dimensional physical dimensions, and obtaining a high-fidelity digital twin entity; and constructing a multi-dimensional state fingerprint for the high-fidelity digital twin entity, and performing whole-life-cycle operation and maintenance monitoring. The application can construct a high-fidelity entity three-dimensional model under sparse view angles.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus, storage medium and electronic device for three-dimensional asset inventory of transportation infrastructure based on sparse images. Background Technology

[0002] With the rapid development of smart city and digital twin technologies, constructing a high-precision, semantically rich, and comprehensive 3D database of urban infrastructure (such as streetlights, traffic signs, and manhole covers) has become the cornerstone of refined urban management. Currently, automated surveying and reconstruction based on sequenced street view images acquired by vehicle-mounted mobile measurement systems (MMS) is the mainstream approach. However, when faced with the pressure of massive data storage and transmission, engineering practice often adopts a large-interval sparse acquisition mode (e.g., taking one frame every 10-20 meters). This sparse observation condition poses a severe challenge to the existing technical system, mainly in the following four aspects: (1) Cross-perspective associations lack strong constraints in physical dimensions, which can easily lead to topological errors. Existing multi-view target tracking and matching algorithms typically rely on weighted fusion of visual features (such as ReID and CLIP features) or simple geometric distances (such as IoU and Euclidean distance). However, in urban road scenarios, there are numerous visually similar and spatially adjacent interfering targets (e.g., consecutively arranged streetlights of the same type, or multiple adjacent traffic signs on a lamppost). Relying solely on visual similarity and geometric distance makes mismatching highly likely. Existing fusion strategies often lack a consistency verification mechanism for the intrinsic physical properties of the target (such as true height and width), resulting in insufficient robustness of the algorithm when facing visually similar objects with significant scale differences (such as courtyard lights and high-pole lights).

[0003] (2) It is difficult to construct high-fidelity solid 3D models from a sparse perspective. Traditional multi-view stereo vision (MVS) or neural radiation field (NeRF) reconstruction algorithms heavily rely on dense imagery with high overlap. In sparse street scenes, these methods fail to match feature points due to excessive parallax, resulting in reconstructions filled with holes or completely collapsed. On the other hand, single-view... Figure 3 While generative models (such as various 3D AIGC algorithms) can generate realistic 3D meshes from single images, their outputs are typically located in a normalized virtual coordinate system, lacking true physical meter-level scale and absolute world geographic coordinates. Current industry technology has not yet developed an effective pipeline that organically combines the "realism" of generative AI with the "accuracy" of photogrammetry, resulting in generated models that can only be used for display purposes and cannot be used for engineering measurement and spatial analysis.

[0004] (3) The “acquisition-processing” link is fragmented and lacks a closed-loop feedback mechanism based on quality assessment. Current infrastructure surveys typically employ an open-loop "collect first, process later" approach. Data collection vehicles blindly follow fixed routes, making it impossible to assess data quality in real time. When subsequent processing stages discover excessively high uncertainty in the geometric calculations of certain key targets due to obstructions, lighting conditions, or poor viewing angles, it is often too late, forcing the acceptance of low-precision results or incurring significant costs to reschedule vehicles for supplementary data collection. Existing technologies lack a closed-loop feedback mechanism that uses geometric topological quality (such as ray intersection angles and reprojection residuals) to assess data availability in real time and automatically guide manual verification or trigger targeted supplementary data collection.

[0005] (4) Lack of full lifecycle status monitoring methods based on multidimensional fingerprints Current facility operation and maintenance mainly rely on manual inspections or simple image change detection. Pure image comparison is easily affected by changes in lighting, season, and shooting angle, resulting in a large number of false alarms. Moreover, two-dimensional images are difficult to quantify and perceive structural changes in the condition of facilities (such as a light pole tilting by 5 degrees or a road sign dropping by 20 centimeters). Existing technologies lack a "multi-dimensional state fingerprint" system that integrates precise geometric dimensions, visual semantic features, and relative spatial topology, making it difficult to achieve refined condition monitoring and anomaly diagnosis of infrastructure throughout its entire lifecycle, from "construction" to "abandonment."

[0006] In summary, there is an urgent need for a new method that can effectively solve technical challenges such as multimodal robust correlation under a sparse perspective, spatial metric quantification of generative models, closed-loop feedback of perception-acquisition, and precise operation and maintenance throughout the entire lifecycle. Summary of the Invention

[0007] This invention provides a method, apparatus, storage medium, and electronic device for three-dimensional asset inventory of transportation infrastructure based on sparse images. It can solve the problems of existing technologies under sparse street view image conditions, such as the lack of strong physical constraints on cross-view association of transportation infrastructure, the lack of real physical measurement in generative three-dimensional models, the fragmentation of data acquisition and processing links, and the lack of full life cycle status monitoring methods.

[0008] This invention provides a method for three-dimensional asset inventory of transportation infrastructure based on sparse images, comprising: Acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic images, and extract the visual semantic features and geometric ray features of each observed target; Construct a physical-gated multimodal cascaded association graph to associate cross-view observation targets and generate target observation chains; High-precision geometric calculations are performed on the target observation chain to obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and uncertainty assessment is performed to obtain an uncertainty score; A graded response is executed based on the uncertainty score; Select the frame image with the best visual quality in the target observation chain, generate a normalized three-dimensional mesh model based on the image, and perform spatial transformation on the normalized three-dimensional mesh model based on the three-dimensional center coordinates and the three-dimensional physical dimensions to obtain a high-fidelity digital twin entity; A multi-dimensional state fingerprint is constructed for the high-fidelity digital twin entity, and full lifecycle operation and maintenance monitoring is performed based on the multi-dimensional fingerprint.

[0009] Furthermore, according to the above-mentioned method for 3D asset inventory of transportation infrastructure based on sparse images, a multimodal cascaded association graph based on physical gating is constructed to associate cross-view observation targets and generate target observation chains, including: Visual similarity between observation pairs is calculated based on the visual semantic features, ray-to-plane distance between observation pairs is calculated based on the geometric ray features, and comprehensive affinity score of observation pairs is calculated based on visual similarity and ray-to-plane distance to generate candidate matching pairs. Physical consistency verification is performed on the candidate matching pairs. The assumed physical size of the observed target is calculated by using sparse ray geometry. Candidate matching pairs whose assumed physical size difference exceeds a preset consistency tolerance threshold are judged as mismatches and are forcibly disconnected to generate a target observation chain that has passed physical strong verification.

[0010] Furthermore, according to the above-mentioned method for three-dimensional asset inventory of transportation infrastructure based on sparse images, the formula for calculating the comprehensive affinity score is as follows:

[0011] in, To calculate the overall affinity score, Let be the cosine similarity function of the visual semantic feature vectors. Let be the normalized scoring function for the skew distance between the two rays. Targets for observation and Visual semantic features Targets for observation and The axial distance of the rays, The adaptive weighting factor is calculated using the following formula:

[0012] in, It is the Sigmoid activation function. To detect the area of ​​the frame, This is the preset baseline area threshold.

[0013] Furthermore, according to the above-mentioned method for three-dimensional asset inventory of transportation infrastructure based on sparse images, high-precision geometric calculation is performed on the target observation chain to obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and uncertainty assessment is performed to obtain an uncertainty score, including: Based on the target observation chain, a geometric energy function is constructed. By minimizing the geometric energy function, the optimal three-dimensional center coordinates and the corresponding three-dimensional physical dimensions are obtained. The uncertainty score for each observed target is calculated based on geometric topology.

[0014] Furthermore, according to the above-mentioned method for 3D asset inventory of transportation infrastructure based on sparse images, the geometric energy function is:

[0015] in, The geometric energy function, Point To the Observation rays vertical distance, For robust kernel functions; The formula for calculating the uncertainty score is as follows:

[0016] in, Score the uncertainty. Let N be the maximum angle between any two rays in the observation chain, and N be the number of observation frames. The optimized average reprojection residual, These are the normalized weighting coefficients. and This is the normalized parameter for controlling the decay rate of the function.

[0017] Furthermore, according to the above-described method for 3D asset inventory of transportation infrastructure based on sparse images, the hierarchical response based on the uncertainty score includes: When the uncertainty score is greater than the first preset threshold, it is determined to be a high-risk blind zone. The optimal reshoot viewpoint that maximizes the ray intersection angle is calculated in reverse based on the three-dimensional center coordinates of the observed target, and a collection guidance instruction is generated to trigger the current or subsequent fixed-point encrypted reshoot. When the uncertainty score is less than or equal to the first preset threshold and greater than the second preset threshold, it is determined to be a suspected target. The observed target is projected back to the original image, a set of multi-view ROI slices are extracted, and a manual verification work order is generated. When the uncertainty score is less than or equal to the second preset threshold, it is determined to be a high-confidence target.

[0018] Furthermore, according to the above-mentioned method for 3D asset inventory of transportation infrastructure based on sparse images, the formula for spatial transformation is:

[0019] Wherein, scaling factor ,in For normalized 3D mesh models The height of the enclosure box, The rotation matrix represents the actual physical height in the three-dimensional physical dimensions. The yaw angle is determined by the angle between the principal direction of the observed ray and the north direction of the world coordinate system, and the model is positioned so that its front faces the road. The translation vector is then used. , The coordinates are the three-dimensional center coordinates.

[0020] Furthermore, according to the above-mentioned method for three-dimensional asset inventory of transportation infrastructure based on sparse images, the multidimensional state fingerprint is:

[0021] in, For multidimensional state fingerprints, These represent absolute coordinates, physical height, average visual feature vector, and neighborhood topological relationships, respectively.

[0022] The present invention also provides a 3D asset inventory device for transportation infrastructure based on sparse images, comprising: The multimodal feature extraction module is used to acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic image, and extract the visual semantic features and geometric ray features of each observed target. The multimodal association module is used to construct a physical-gated multimodal cascade association graph to associate cross-view observation targets and generate target observation chains. The geometric solution and evaluation module is used to perform high-precision geometric solution on the target observation chain, obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and perform uncertainty evaluation to obtain an uncertainty score; A closed-loop feedback control module is used to execute a graded response based on the uncertainty score; The model anchoring and quantization module is used to select the frame image with the best visual quality in the target observation chain, generate a normalized three-dimensional mesh model based on the image, and perform spatial transformation on the normalized three-dimensional mesh model based on the three-dimensional center coordinates and the three-dimensional physical dimensions to obtain a high-fidelity digital twin entity. The intelligent operation and maintenance monitoring module is used to construct a multi-dimensional state fingerprint for the high-fidelity digital twin entity and perform full life-cycle operation and maintenance monitoring based on the multi-dimensional fingerprint.

[0023] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described methods for three-dimensional asset inventory of transportation infrastructure based on sparse images.

[0024] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the sparse image-based three-dimensional asset inventory method for transportation infrastructure described in any of the preceding claims.

[0025] This invention provides a method, apparatus, storage medium, and electronic device for 3D asset inventory of transportation infrastructure based on sparse images. The invention employs a "physical gating" mechanism, adding strong verification of physical size consistency based on the fusion of visual and geometric features. This effectively solves the problem of mismatching visually similar but different scaled objects (such as streetlights of different heights) in dense scenes, significantly reducing the topology error rate. Specifically, this invention includes the following beneficial effects: (1) The engineering application of generative models has been realized: This invention solves the problem that the output of existing single-view generative models is normalized coordinates and lacks real physical scale by using the spatial anchoring strategy of "geometric skeleton constraining generative flesh". The AIGC technology has been successfully applied to the fields of high-precision engineering surveying and digital twins.

[0026] (2) A closed-loop data quality control mechanism was constructed: This invention uses geometric parameters to quantitatively evaluate the reconstruction uncertainty and uses this to guide the optimal viewpoint supplementary collection, breaking the traditional "blind collection and open-loop processing" mode. While ensuring the low cost advantage of sparse collection, it effectively guarantees the reconstruction accuracy of key and difficult targets.

[0027] (3) Improved the level of precision in facility operation and maintenance: Based on the precise back-calculated physical geometric parameters, the present invention constructs a multi-dimensional state fingerprint, which, compared with the traditional pure image comparison, can more keenly detect structural hidden dangers of facilities (such as pole tilting, settlement, breakage, etc.), and realizes precise management of the entire life cycle. Attached Figure Description

[0028] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0029] Figure 1 A flowchart of a method for three-dimensional asset inventory of transportation infrastructure based on sparse images, provided in an embodiment of the present invention.

[0030] Figure 2This is a schematic diagram of the structure of a 3D asset inventory device for transportation infrastructure based on sparse images, provided in an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention provides a method, apparatus, storage medium, and electronic device for three-dimensional asset inventory of transportation infrastructure based on sparse images. The apparatus for three-dimensional asset inventory of transportation infrastructure based on sparse images provided in this invention can be integrated into an electronic device, which can be a terminal, server, or other such device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0034] Please see Figure 1 , Figure 1 The flowchart illustrates a method for three-dimensional asset inventory of transportation infrastructure based on sparse images, provided in an embodiment of the present invention. This method, applied in electronic devices, includes the following steps: S1. Acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic image, and extract the visual semantic features and geometric ray features of each observed target.

[0035] First, acquire sparse sequence panoramic images collected by the vehicle system. The image acquisition interval is set to a sparse mode of 10-20 meters. Then, a pre-trained target detection network (such as YOLO or Mask R-CNN) is used to extract two-dimensional bounding boxes of infrastructure targets (such as streetlights and traffic signs) in the image. (in These represent the horizontal and vertical coordinates of the top-left corner of the bounding box, respectively. (representing the width and height of the bounding box, respectively) and semantic mask. For each detection target Two sets of features are extracted in parallel: one is geometric ray features, which are derived from the camera intrinsic parameter matrix. and extrinsic rotation matrix Calculate from the camera optical center Pointing to pixel coordinates unit observation direction vector ,Will The first is geometric ray features; the second is visual semantic features, which involves inputting the image region within the detection box into the CLIP (Contrastive Language-Image Pre-training) image encoder to extract high-dimensional visual feature vectors. ,Will As a visual semantic feature.

[0036] S2, construct a physical-gated multimodal cascaded association graph to associate cross-view observation targets and generate target observation chains.

[0037] In one embodiment, step S2 includes the following steps: S21. Calculate the visual similarity between observation pairs based on visual semantic features, calculate the ray-to-plane distance between observation pairs based on geometric ray features, and calculate the comprehensive affinity score of observation pairs based on visual similarity and ray-to-plane distance to generate candidate matching pairs.

[0038] Specifically, visual similarity and ray-to-plane distance are first calculated, and adaptive weighted fusion is performed. The formula for calculating the overall affinity score is as follows:

[0039] in, To calculate the overall affinity score, Let be the cosine similarity function of the visual semantic feature vectors. This is a normalized scoring function for the distance between two skew rays; the smaller the distance, the higher the score. Targets for observation and Visual semantic features Targets for observation and The axial distance of the rays, The adaptive weighting factor is calculated using the following formula:

[0040] in, It is the Sigmoid activation function. To detect the area of ​​the frame, This is the preset baseline area threshold.

[0041] This mechanism makes it possible when the target is small or ambiguous. Automatic reduction relies more on geometric features; when the target is clear... Enlargement depends more on visual features.

[0042] S22, perform physical consistency verification on candidate matching pairs, use sparse ray geometry to back-calculate the assumed physical size of the observed target, and determine candidate matching pairs whose assumed physical size difference exceeds the preset consistency tolerance threshold as mismatches and forcibly disconnect the association, so as to generate a target observation chain that has passed physical strong verification.

[0043] Specifically, regarding the overall affinity score Candidate matching pairs exceeding the initial screening threshold undergo a physical veto check. The intersection point of the two observed rays is initially calculated using triangulation, and the assumed physical heights of the two observed targets are calculated using the principle of perspective projection. and Based on the calculation assumption of physical height and Physical difference rate The formula is as follows:

[0044] Execute the gating logic to determine: if the physical difference rate Greater than the preset consistency tolerance threshold (For example, in this embodiment, the value is 0.2), then it is directly determined. and If it is a false match, the association is forcibly broken; otherwise, the association is retained and the target observation chain is generated.

[0045] S3 performs high-precision geometric calculations on the target observation chain to obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and performs uncertainty assessment to obtain an uncertainty score.

[0046] In one embodiment, step S3 includes the following steps: S31. Construct a geometric energy function based on the target observation chain, minimize the geometric energy function, and calculate the optimal three-dimensional center coordinates and the corresponding three-dimensional physical dimensions.

[0047] First, global energy optimization is performed: a geometric energy function is constructed for the correlated target observation chain. ,in The three-dimensional center coordinates are to be determined, and the formula is as follows:

[0048] in, The geometric energy function, Point To the Observation rays vertical distance, Use robust kernel functions (such as the Huber kernel) to suppress outliers.

[0049] By minimizing this energy function, the optimal three-dimensional center coordinates can be calculated. and the corresponding precise three-dimensional physical dimensions (including height) ,width ).

[0050] S32 calculates the uncertainty score for each observed target based on geometric topology.

[0051] Simultaneously, based on geometric topology, a comprehensive reconstruction uncertainty score U is calculated for each target to quantify the credibility of the reconstruction results. The calculation formula is as follows:

[0052] in, Score the uncertainty. The maximum angle between any two rays in the observation chain represents the intersection geometric quality; N is the number of observation frames, representing the observation redundancy. The optimized average reprojection residual characterizes the model fitting quality; Normalized weighting coefficients (satisfying) ), and This is the normalized parameter for controlling the decay rate of the function.

[0053] S4, execute a graded response based on the uncertainty score.

[0054] In one embodiment, step S4 includes: When the uncertainty score is greater than the first preset threshold, it is determined to be a high-risk blind zone. The optimal reshoot viewpoint that maximizes the ray intersection angle is calculated in reverse based on the three-dimensional center coordinates of the observed target, and a data acquisition guidance instruction is generated to trigger the current or subsequent fixed-point encrypted reshoot. When the uncertainty score is less than or equal to the first preset threshold and greater than the second preset threshold, it is determined to be a suspected target. The observed target is projected back to the original image, a set of multi-view ROI slices is extracted, and a manual verification work order is generated. When the uncertainty score is less than or equal to the second preset threshold, it is determined to be a high-confidence target.

[0055] Specifically, if (First preset threshold) is determined to be a high-risk blind zone, based on the approximate location of the observed target, i.e., the three-dimensional center coordinates. Using the current vehicle trajectory as a reference, a position that maximizes the angle between the new ray and the existing ray is calculated in reverse as the "Next-Best-View," and the coordinates of this viewpoint are sent to the acquisition terminal to trigger the current or subsequent fixed-point encrypted acquisition; if If identified as a suspected target, the target is automatically projected back into the original image, a set of multi-view ROI slices is extracted, and a manual verification work order is generated; if (Second preset threshold) is determined to be a high confidence target and automatically proceeds to step S5.

[0056] S5. Select the frame with the best visual quality in the target observation chain, generate a normalized three-dimensional mesh model based on the image, and perform spatial transformation on the normalized three-dimensional mesh model based on the three-dimensional center coordinates and three-dimensional physical dimensions to obtain a high-fidelity digital twin entity.

[0057] First, normalization generation is performed: the frame with the best visual quality in the target observation chain is selected as input and fed into the pre-trained single-view image. Figure 3 A 3D generative model (such as SAM 3D or TripoSR) outputs a normalized 3D mesh model of the observed target. At this point, the model is in a virtual unit space and has no real scale.

[0058] Then, a spatial anchoring transformation is performed: using the actual physical height calculated in step S3. and three-dimensional center coordinates For normalized 3D mesh models Application of spatial transformation The formula is as follows:

[0059] Wherein, scaling factor ,in For normalized 3D mesh models The height of the enclosure box, The rotation matrix represents the actual physical height in the three-dimensional physical dimensions. The yaw angle is determined by the angle between the principal direction of the observed ray and the north direction of the world coordinate system. The model is then positioned so that its front faces the road, and the translation vector is used. , The coordinates are the three-dimensional center coordinates.

[0060] right After applying the above transformation, a high-fidelity quantized solid model in the world coordinate system is obtained. .

[0061] S6 constructs multi-dimensional state fingerprints for high-fidelity digital twin entities and performs full lifecycle operation and maintenance monitoring based on multi-dimensional fingerprints.

[0062] First, fingerprint construction is performed: a "multi-dimensional state fingerprint" F is constructed for each high-fidelity digital twin entity, defined as follows:

[0063] in, For multidimensional state fingerprints, These represent absolute coordinates, physical height, average visual feature vector, and neighborhood topological relationships, respectively.

[0064] Then perform incremental diagnostics: in subsequent inspection tasks, the newly calculated fingerprints will be... With historical fingerprint database Perform a comparison. If the positions are close, then... ( (The distance threshold is set, but the height is significantly reduced.) ( The ratio (e.g., 0.8) is used to determine whether the pole is "broken" or "severely tilted"; if the locations are close but the visual features are similar... ( (Based on a preset visual change threshold), it is judged as "appearance damage" (such as rust, graffiti); if in No matching fingerprints were found nearby, and the case was determined to be "assets lost".

[0065] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a sparse image-based 3D asset inventory device for transportation infrastructure. Specifically, the sparse image-based 3D asset inventory device for transportation infrastructure can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0066] Please see Figure 2 , Figure 2 This invention specifically describes a sparse image-based three-dimensional asset inventory device for transportation infrastructure, which is applied in electronic devices. The sparse image-based three-dimensional asset inventory device for transportation infrastructure may include: The multimodal feature extraction module is used to acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic image, and extract the visual semantic features and geometric ray features of each observed target. The multimodal association module is used to construct a physical-gated multimodal cascade association graph to associate cross-view observation targets and generate target observation chains. The geometric solution and evaluation module is used to perform high-precision geometric solution on the target observation chain, obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and perform uncertainty evaluation to obtain an uncertainty score; A closed-loop feedback control module is used to execute a graded response based on the uncertainty score; The model anchoring and quantization module is used to select the frame image with the best visual quality in the target observation chain, generate a normalized three-dimensional mesh model based on the image, and perform spatial transformation on the normalized three-dimensional mesh model based on the three-dimensional center coordinates and the three-dimensional physical dimensions to obtain a high-fidelity digital twin entity. The intelligent operation and maintenance monitoring module is used to construct a multi-dimensional state fingerprint for the high-fidelity digital twin entity and perform full life-cycle operation and maintenance monitoring based on the multi-dimensional fingerprint.

[0067] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0068] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps in any embodiment of the sparse image-based three-dimensional asset inventory method for transportation infrastructure provided by this invention. Therefore, it can achieve the beneficial effects that any sparse image-based three-dimensional asset inventory method for transportation infrastructure provided by this invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0069] Figure 3 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the sparse image-based three-dimensional asset inventory method for transportation infrastructure provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0070] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0071] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control. Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0072] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0073] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles operating devices, user interfaces, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0074] The electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to the various components. In some embodiments, the power supply may be logically connected to the processor 580 via a power management device, thereby enabling functions such as managing charging, discharging, and power consumption through the power management device. The power supply 590 may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0075] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic images, and extract the visual semantic features and geometric ray features of each observed target; Construct a physical-gated multimodal cascaded association graph to associate cross-view observation targets and generate target observation chains; High-precision geometric calculations are performed on the target observation chain to obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and uncertainty assessment is performed to obtain an uncertainty score; A graded response is executed based on the uncertainty score; Select the frame image with the best visual quality in the target observation chain, generate a normalized three-dimensional mesh model based on the image, and perform spatial transformation on the normalized three-dimensional mesh model based on the three-dimensional center coordinates and the three-dimensional physical dimensions to obtain a high-fidelity digital twin entity; A multi-dimensional state fingerprint is constructed for the high-fidelity digital twin entity, and full lifecycle operation and maintenance monitoring is performed based on the multi-dimensional fingerprint.

[0076] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0077] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the sparse image-based three-dimensional asset inventory method for transportation infrastructure provided by the present invention.

[0078] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0079] Since the instructions stored in the storage medium can execute the steps in any embodiment of the sparse image-based three-dimensional asset inventory method for transportation infrastructure provided in the embodiments of the present invention, the beneficial effects that any sparse image-based three-dimensional asset inventory method for transportation infrastructure provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0080] The foregoing has provided a detailed description of a method, apparatus, storage medium, and electronic device for three-dimensional asset inventory of transportation infrastructure based on sparse images, as provided in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for three-dimensional asset inventory of transportation infrastructure based on sparse images, characterized in that, The method includes: Acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic images, and extract the visual semantic features and geometric ray features of each observed target; A physical-gated multimodal cascaded association graph is constructed to associate cross-view observation targets and generate target observation chains. This includes: calculating visual similarity between observation pairs based on the visual semantic features; calculating ray-to-plane distance between observation pairs based on the geometric ray features; and calculating a comprehensive affinity score for the observation pairs based on the visual similarity and the ray-to-plane distance to generate candidate matching pairs. The formula for calculating the comprehensive affinity score is as follows: in, To calculate the overall affinity score, Let be the cosine similarity function of the visual semantic feature vectors. Let be the normalized scoring function for the skew distance between the two rays. Targets for observation and Visual semantic features Targets for observation and The axial distance of the rays, The adaptive weighting factor is calculated using the following formula: in, It is the Sigmoid activation function. To detect the area of ​​the frame, This is a preset baseline area threshold; Physical consistency verification is performed on the candidate matching pairs. The assumed physical size of the observed target is calculated by using sparse ray geometry. Candidate matching pairs whose assumed physical size difference exceeds a preset consistency tolerance threshold are judged as mismatches and are forcibly disconnected to generate a target observation chain that has passed physical strong verification. High-precision geometric calculations are performed on the target observation chain to obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed targets. Uncertainty assessment is then performed to obtain an uncertainty score, including: constructing a geometric energy function based on the target observation chain, minimizing the geometric energy function, and calculating the optimal three-dimensional center coordinates and corresponding three-dimensional physical dimensions; and calculating the uncertainty score for each observed target based on the geometric topology. The geometric energy function is: in, The geometric energy function, Point To the Observation rays vertical distance, For robust kernel functions; The formula for calculating the uncertainty score is as follows: in, Score the uncertainty. Let N be the maximum angle between any two rays in the observation chain, and N be the number of observation frames. The optimized average reprojection residual, These are the normalized weighting coefficients. and This is a normalized parameter for controlling the decay rate of the control function; The system executes a tiered response based on the uncertainty score, including: when the uncertainty score is greater than a first preset threshold, it is determined to be a high-risk blind zone; the optimal re-shooting viewpoint that maximizes the ray intersection angle is calculated in reverse based on the three-dimensional center coordinates of the observed target; and a data acquisition guidance instruction is generated to trigger the current or subsequent fixed-point encrypted re-shooting. When the uncertainty score is less than or equal to the first preset threshold and greater than a second preset threshold, it is determined to be a suspected target; the observed target is projected back to the original image; a set of multi-view ROI slices is extracted; and a manual verification work order is generated. When the uncertainty score is less than or equal to the second preset threshold, it is determined to be a high-confidence target. Select the image with the best visual quality from the target observation chain, generate a normalized 3D mesh model based on the image, and perform a spatial transformation on the normalized 3D mesh model based on the 3D center coordinates and the 3D physical dimensions to obtain a high-fidelity digital twin entity; the process of obtaining the high-fidelity digital twin entity includes: Normalized generation: Select the frame with the best visual quality in the target observation chain as input, feed it into a pre-trained single-view 3D generation model, and output a normalized 3D mesh model of the observed target. ; Perform spatial anchoring transformation: using the calculated actual physical height and three-dimensional center coordinates For normalized 3D mesh models Application of spatial transformation The formula is as follows: Wherein, scaling factor ,in For normalized 3D mesh models The height of the enclosure box, The rotation matrix represents the actual physical height in the three-dimensional physical dimensions. The yaw angle is determined by the angle between the principal direction of the observed ray and the north direction of the world coordinate system, and the model is positioned so that its front faces the road. The translation vector is then used. , The coordinates of the three-dimensional center; right After applying the above transformation, a high-fidelity quantized solid model in the world coordinate system is obtained. ; A multi-dimensional state fingerprint is constructed for the high-fidelity digital twin entity, and full lifecycle operation and maintenance monitoring is performed based on the multi-dimensional state fingerprint, including: Fingerprint construction: Construct a multi-dimensional state fingerprint F for each high-fidelity digital twin entity, defined as follows: in, For multidimensional state fingerprints, These represent absolute coordinates, physical height, average visual feature vector, and neighborhood topological relation, respectively. Perform incremental diagnostics: In subsequent inspection tasks, the newly calculated fingerprints will be... With historical fingerprint database Compare the positions; if they are close, then... However, the height decreased significantly. If the pole is determined to be broken or severely tilted, the location may be close but the visual characteristics are similar. It is judged as external damage; if in No matching fingerprints were found nearby, indicating that the asset was lost; among them, The preset distance threshold, This is the proportionality coefficient. This is the preset visual change threshold.

2. A sparse image-based three-dimensional asset inventory device for transportation infrastructure, wherein the sparse image-based three-dimensional asset inventory device for transportation infrastructure is used to implement the sparse image-based three-dimensional asset inventory method for transportation infrastructure as described in claim 1, characterized in that, include: The multimodal feature extraction module is used to acquire panoramic images and corresponding pose data, extract the two-dimensional bounding box and semantic mask of the panoramic image, and extract the visual semantic features and geometric ray features of each observed target. The multimodal association module is used to construct a physical-gated multimodal cascade association graph to associate cross-view observation targets and generate target observation chains. The geometric solution and evaluation module is used to perform high-precision geometric solution on the target observation chain, obtain the three-dimensional center coordinates and three-dimensional physical dimensions of the observed target, and perform uncertainty evaluation to obtain an uncertainty score; A closed-loop feedback control module is used to execute a graded response based on the uncertainty score; The model anchoring and quantization module is used to select the frame image with the best visual quality in the target observation chain, generate a normalized three-dimensional mesh model based on the image, and perform spatial transformation on the normalized three-dimensional mesh model based on the three-dimensional center coordinates and the three-dimensional physical dimensions to obtain a high-fidelity digital twin entity. The intelligent operation and maintenance monitoring module is used to construct a multi-dimensional state fingerprint for the high-fidelity digital twin entity and perform full life-cycle operation and maintenance monitoring based on the multi-dimensional state fingerprint.

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions adapted for loading by a processor to execute the sparse image-based three-dimensional asset inventory method for transportation infrastructure as described in claim 1.

Images