Method and device for constructing three-dimensional digital large model of open set traffic infrastructure

Abstract

The invention discloses an open set traffic infrastructure three-dimensional digital large model construction method and device. The method comprises the following steps: acquiring street scene pictures containing traffic infrastructures, point cloud data and prompt information corresponding to the street scene pictures; performing target detection on the streetscape picture based on the prompt information to obtain a target detection frame; inputting the target detection box into a segmentation cutting model to obtain a two-dimensional mask; projecting the two-dimensional mask to a three-dimensional space based on a geometrical relationship between the two-dimensional mask and the point cloud data to obtain a three-dimensional mask candidate; grouping the three-dimensional mask candidates based on a communication coverage judgment strategy to obtain a three-dimensional mask candidate group; and calculating scores of points in the three-dimensional mask candidate group, determining a segmentation threshold through an Otsu threshold segmentation algorithm, taking the points of which the scores are higher than the segmentation threshold as foreground points, and taking a set of the foreground points as a target three-dimensional mask. The method has high adaptability, and traffic infrastructures in different urban environments can be accurately identified.

Description

Technical Field

[0001] This application relates to the field of computer vision, and particularly to a method, device, storage medium, and electronic device for constructing a three-dimensional digital large model of open-set traffic infrastructure. Background Art

[0002] Traffic infrastructure is a key support for urban operation, and its digital transformation is crucial for improving management efficiency, ensuring traffic safety, and promoting urban sustainable development. Governments around the world have actively invested. For example, the US government has made huge investments in the modernization of the transportation system, and China has also introduced policies to promote the digital upgrade of highway and waterway traffic infrastructure, highlighting its important position in modern society.

[0003] The core of traffic infrastructure digitization lies in accurate asset inventory, and this process highly depends on efficient data processing and analysis technologies. Traditional traffic infrastructure survey and identification methods mainly work based on the road scene point clouds and panoramic images obtained by mobile mapping systems. However, these traditional methods face many difficult challenges in practical applications.

[0004] Although the image-based recognition method can capture rich texture and semantic information, it has obvious limitations in obtaining the accurate three-dimensional geographical location of the target. Due to the lack of depth information in the image data itself, relying solely on images for positioning, its geographical accuracy is difficult to meet the actual needs, resulting in the inability to provide accurate spatial location data for traffic infrastructure management in practical applications. In the case of point cloud-based methods, when dealing with targets with complex shapes, due to the limited ability to describe manual features, it is difficult to comprehensively and accurately depict the target features, thus making its generalization and robustness poor. In addition, point cloud data cannot directly provide texture information, which poses an obstacle to understanding the multi-dimensional attributes and deep meanings of traffic infrastructure.

[0005] To overcome the limitations of the above single-modal methods, methods combining images and point clouds have emerged. Such methods attempt to integrate the advantages of both and have made progress to a certain extent, but there are still many deficiencies. In open-set detection, they have poor adaptability to new category targets and are difficult to cope with the continuously increasing and changing types of traffic infrastructure. The utilization of multi-view information is also insufficient, and the valuable spatial information contained in repeated observations has not been fully explored, resulting in a waste of data resources. Moreover, these methods can often only obtain the basic attributes of the target, such as category and location, and cannot effectively obtain richer attribute information such as the meaning of the target, shape details, and material composition. This makes the description of traffic infrastructure incomplete and in-depth, and it is difficult to meet the needs of refined management and decision-making. Summary of the Invention

[0006] The present application provides a method, apparatus, storage medium and electronic device for constructing a three-dimensional digital large model of open-set traffic infrastructure, which has high adaptability, can accurately identify traffic infrastructure in different urban environments, supports rich category expansion and fine-grained attribute extraction, and improves the accuracy of traffic infrastructure identification.

[0007] The present application provides a method for constructing a three-dimensional digital large model of open-set traffic infrastructure, including: Obtain street view images, point cloud data containing traffic infrastructure, and prompt information corresponding to the street view images; Perform object detection on the street view images based on the prompt information to obtain object detection boxes; Input the object detection boxes into the Segment Anything Model to obtain two-dimensional masks; Based on the geometric relationship between the two-dimensional masks and the point cloud data, project the two-dimensional masks into three-dimensional space to obtain three-dimensional mask candidates; Group the three-dimensional mask candidates based on the intersection coverage judgment strategy to obtain groups of three-dimensional mask candidates; Calculate the scores of the points in the groups of three-dimensional mask candidates, and determine the segmentation threshold through the Otsu threshold segmentation algorithm. Take the points with scores higher than the segmentation threshold as foreground points, and the set of the foreground points is the target three-dimensional mask.

[0008] Further, in the above method for constructing a three-dimensional digital large model of open-set traffic infrastructure, the prompt information includes text prompt information and visual prompt information. The performing object detection on the street view images based on the prompt information to obtain object detection boxes includes: Input the text prompt information and the street view images into an open-set object detection model to obtain object detection boxes; Input the visual prompt information and the street view images into an open-set object detection model to obtain object detection boxes.

[0009] Further, in the above method for constructing a three-dimensional digital large model of open-set traffic infrastructure, inputting the text prompt information and the street view images into an open-set object detection model to obtain object detection boxes includes: Obtain the object detection boxes through the first formula, and the first formula is:

[0010] Where is an open-set object detection model, is the panoramic image of the street view, is the text prompt describing the target category, is the object detection box, is a set of bounding boxes of the target, is the corresponding semantic category; Said inputting the visual cue information and the street view image into an open-set object detection model to obtain object detection bounding boxes includes: Obtaining the object detection bounding boxes through a second formula, where the second formula is:

[0011] where, is another open-set object detection model, is the visual cue information, is the object detection bounding box, and are respectively the detected bounding box and the corresponding semantic category.

[0012] Furthermore, in the above method for constructing an open-set traffic infrastructure three-dimensional digital model, wherein, characterized in that, said inputting the object detection bounding boxes into a Segment Anything model to obtain a two-dimensional mask includes: Obtaining the two-dimensional mask through a third formula, where the third formula is:

[0013] where, and respectively represent the number of object detection bounding boxes obtained using text cue information and visual cue information, represents the Segment Anything model, is the input image, is the set of bounding boxes of object detection, is the output two-dimensional mask.

[0014] Furthermore, in the above method for constructing an open-set traffic infrastructure three-dimensional digital model, wherein, said projecting the two-dimensional mask into three-dimensional space based on the geometric relationship between the two-dimensional mask and the point cloud data to obtain a three-dimensional mask candidate includes: Projecting the point cloud data onto the image plane according to the extrinsic matrix and the intrinsic matrix of the camera to obtain a number of projection points; Taking the projection points falling within each mask range of the two-dimensional mask as three-dimensional mask candidate points; Taking the set of the three-dimensional mask candidate points as the three-dimensional mask candidate.

[0015] Furthermore, in the above method for constructing an open-set traffic infrastructure three-dimensional digital model, wherein, said grouping the three-dimensional mask based on a communication coverage judgment strategy to obtain a three-dimensional mask candidate group includes: Calculating the confidence of the three-dimensional mask candidate; Calculate the 3D mask candidate with the highest confidence and the coverage rate between it and other 3D mask candidates that intersect with the 3D mask candidate with the highest confidence. If the coverage rate is greater than the coverage rate threshold, the two 3D mask candidates are divided into the same group of 3D mask candidates. Specifically, in the above method for constructing an open-set traffic infrastructure 3D digital model, the formula for calculating the coverage rate is as follows:

[0016] Specifically, in the above method for constructing an open-set traffic infrastructure 3D digital model, the formula for calculating the coverage rate is as follows:

[0017] where represents the dilation operation on the sparse pixels of the point cloud is the 2D mask obtained by projecting the intersection point cloud of two 3D mask candidates to be judged onto their respective images is the corresponding 2D mask is the number of pixel points The definition of the group of 3D mask candidates is as follows:

[0018] where is the group of 3D mask candidates represents the coverage rate judgment threshold is the 3D mask candidate is the convolution operation

[0019] Specifically, in the above method for constructing an open-set traffic infrastructure 3D digital model, calculating the scores of the points in the group of 3D mask candidates includes: Calculating the scores through the following formula:

[0020]

[0021] where represents the number of 3D mask candidates containing the point is the 3D mask candidate is the function for calculating the absolute value is the observation angle closest to orthogonal in multiple perspectives

[0022] Specifically, in the above method for constructing an open-set traffic infrastructure 3D digital model, after the step of using the set of foreground points as the target 3D mask, it includes: Determine the target detection box corresponding to the target 3D mask; ​Crop the target detection box to obtain a cropped image; Input the cropped image and a preset attribute information group into a multi-modal pre-trained model to obtain the matching degree between the cropped image and each piece of attribute information in the attribute information group; Select the piece of attribute information with the highest matching degree as the additional attribute of the corresponding traffic infrastructure.

[0023] This application also provides an open-set traffic infrastructure three-dimensional digital large model construction device, including: An acquisition module, configured to acquire a street view image, point cloud data including traffic infrastructure, and prompt information corresponding to the street view image; A target detection module, configured to perform target detection on the street view image based on the prompt information to obtain a target detection box; A two-dimensional mask generation module, configured to input the target detection box into a segment-anything model to obtain a two-dimensional mask; A three-dimensional mask candidate generation module, configured to project the two-dimensional mask into three-dimensional space based on the geometric relationship between the two-dimensional mask and the point cloud data to obtain a three-dimensional mask candidate; A grouping module, configured to group the three-dimensional mask candidates based on a communication coverage judgment strategy to obtain a three-dimensional mask candidate group; A target three-dimensional mask generation module, configured to calculate the scores of the points in the three-dimensional mask candidate group, determine a segmentation threshold through an Otsu threshold segmentation algorithm, use the points with scores higher than the segmentation threshold as foreground points, and set the set of the foreground points as the target three-dimensional mask.

[0024] This application also provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions are suitable for being loaded by a processor to execute any one of the above open-set traffic infrastructure three-dimensional digital large model construction methods.

[0025] This application also provides an electronic device, including a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any one of the above open-set traffic infrastructure three-dimensional digital large model construction methods.

[0026] The method, device, storage medium and electronic device for constructing a large-scale three-dimensional digital model of open-set traffic infrastructure provided by this application. First, this application uses open-set object detection technology based on text and visual cues to identify objects in street view images. Secondly, using the object detection bounding box as a cue, a two-dimensional mask of the object is generated by the SAM segmentation-all model. Then, the two-dimensional mask is projected onto the 3D point cloud to obtain three-dimensional mask candidates. Through a 2D-3D spatial projection strategy with point cloud binding, and using multi-view repeated observation information and viewing angles, the three-dimensional mask candidates are grouped, scored and segmented, thus generating high-quality three-dimensional masks of the objects. Finally, a multi-modal model based on image-text contrast learning is used to extract multi-dimensional attribute information of the objects, including fine semantics, geometric shapes and material features. This application has high adaptability, can accurately identify traffic infrastructure in different urban environments, supports rich category expansion and fine-grained attribute extraction, improves the accuracy of traffic infrastructure identification, and further enhances the decision-making efficiency of infrastructure management and urban planning. Description of the Drawings

[0027] The following, in conjunction with the drawings, through a detailed description of the specific embodiments of this application, will make the technical solutions and other beneficial effects of this application obvious.

[0028] Figure 1 It is a flowchart of the method for constructing a large-scale three-dimensional digital model of open-set traffic infrastructure provided by an embodiment of this application.

[0029] Figure 2 It is another flowchart of the method for constructing a large-scale three-dimensional digital model of open-set traffic infrastructure provided by an embodiment of this application.

[0030] Figure 3 It is the object detection results of traffic signs and traffic lights using "Traffic Sign" and "Signal Light" provided by an embodiment of this application.

[0031] Figure 4 It is the visual cue used for roadside stone barriers and the object detection results according to this visual cue provided by an embodiment of this application.

[0032] Figure 5 It is partial two-dimensional masks of traffic infrastructure obtained by using the SAM model and the object detection bounding box as cues provided by an embodiment of this application.

[0033] Figure 6 It is the identification results of partial category traffic infrastructure provided by an embodiment of this application.

[0034] Figure 7 It is the identification results of attributes of partial category traffic infrastructure provided by an embodiment of this application.

[0035] Figure 8 This is a schematic structural diagram of an open-set traffic infrastructure three-dimensional digital large model construction device provided by an embodiment of the present application.

[0036] Figure 9 This is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0037] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0038] An embodiment of the present application provides an open-set traffic infrastructure three-dimensional digital large model construction method, device, storage medium, and electronic device. An open-set traffic infrastructure three-dimensional digital large model construction device provided by an embodiment of the present application can be integrated in an electronic device, and the electronic device can be a device such as a terminal or a server. Among them, the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a microprocessing box, or other devices, etc.

[0039] Please refer to Figure 1 and Figure 2 , Figure 1 This is a flowchart of an open-set traffic infrastructure three-dimensional digital large model construction method provided by an embodiment of the present application. Figure 2 This is another flowchart of an open-set traffic infrastructure three-dimensional digital large model construction method provided by an embodiment of the present application, which is applied to an electronic device. The open-set traffic infrastructure three-dimensional digital large model construction method includes the following steps: S1. Obtain street view pictures, point cloud data, and prompt information corresponding to the street view pictures that contain traffic infrastructure.

[0040] S2. Perform object detection on the street view pictures based on the prompt information to obtain object detection frames.

[0041] Specifically, the prompt information includes text prompt information and visual prompt information. Step S2 includes the following steps: S21. Input the text prompt information and the street view pictures into an open-set object detection model to obtain object detection frames.

[0042] Specifically, a large object detection model using text prompts is employed to achieve object detection based on natural language descriptions. By inputting text prompt information (such as "traffic sign"), the model can identify various objects with the same semantic category without relying on specific appearance features. This process can be expressed as:

[0043] Among them, represents an open-set object detection model, specifically a large object detection model using text prompts, is a panoramic image of a street scene, is a text prompt describing the object category, is the object detection box, and the output is a set of bounding boxes of the objects, is the corresponding semantic category. Figure 3 This is the object detection result of traffic signs and traffic lights provided by the embodiment of the present application using "Traffic Sign" and "SignalLight".

[0044] S22. Input the visual prompt information and the street scene image into the open-set object detection model to obtain the object detection box.

[0045] When it is difficult to accurately describe the object in language, a large object detection model using visual prompts can be used to achieve object detection based on visual prompts (such as a selected area or a point). In this embodiment, the region of interest is annotated using visual prompts in the image to achieve precise recognition of complex objects with minimal prompts. The object detection process is as follows:

[0046] Among them, is another open-set object detection model, specifically a large object detection model using visual prompts, is a visual prompt (such as a selected area or a point), is the object detection box, and are the detected bounding box and the corresponding semantic category respectively. Figure 4 This is the visual prompt used for roadside stone isolation piers and the object detection result according to this visual prompt provided by the embodiment of the present application.

[0047] S3. Input the object detection box into the Segment Anything Model to obtain a 2D mask.

[0048] The Segment Anything Model (SAM) is used to generate a pixel-level 2D mask of the object. The SAM model takes the detected bounding box as the prompt input and generates a high-quality pixel-level object segmentation mask. The process can be expressed as:

[0049] Among them, and respectively represent the number of object detection bounding boxes obtained using text prompt information and visual prompt information. represents the SAM model. is the input image. is the set of bounding boxes for object detection. is the output pixel-level two-dimensional mask. This process provides high-quality 2D segmentation input for subsequent point cloud processing, which can significantly improve the accuracy and reliability of segmentation. Figure 5 This is a partial two-dimensional mask of traffic infrastructure obtained by using the SAM model and object detection bounding boxes provided in the embodiments of this application as prompts.

[0050] S4. Based on the geometric relationship between the two-dimensional mask and the point cloud data, project the two-dimensional mask into the three-dimensional space to obtain a three-dimensional mask candidate.

[0051] In one embodiment, step S4 includes the following steps: S41. According to the extrinsic matrix and intrinsic matrix of the camera, project the point cloud data onto the image plane to obtain a number of projection points. S42. Take the projection points falling within each mask range in the two-dimensional mask as three-dimensional mask candidate points. S43. Take the set of three-dimensional mask candidate points as the three-dimensional mask candidate.

[0052] Specifically, after obtaining the two-dimensional mask , according to the extrinsic and intrinsic matrices of the camera, project the three-dimensional points in the point cloud data onto the image plane. For each mask in the two-dimensional mask , the corresponding three-dimensional mask candidate for the two-dimensional mask is composed of the points in the point cloud whose two-dimensional coordinates projected onto the image fall within the range of . Each two-dimensional mask corresponds to a generated three-dimensional mask candidate which is expressed as:

[0053] Among them, is the input point cloud, is the point whose pixel row and column numbers projected onto the panoramic image according to the exposure extrinsic parameters.

[0054] By calculating the three-dimensional mask candidate points through each mask in , the three-dimensional mask candidate can be obtained.

[0055] S5. Group the 3D mask candidates based on the communication coverage judgment strategy to obtain groups of 3D mask candidates.

[0056] S51. Calculate the confidence of the 3D mask candidates; S52. Calculate the 3D mask candidate with the highest confidence and the coverage rate between it and other 3D mask candidates that intersect with the 3D mask candidate with the highest confidence. If the coverage rate is greater than the coverage rate threshold, the two 3D mask candidates are divided into the same group of 3D mask candidates. Specifically, in order to make full use of the spatial information embedded in the repeated observations of the same target from different angles, the present invention proposes a 3D mask candidate grouping strategy based on intersection coverage judgment, which divides the 3D mask candidates belonging to different targets into different clusters. The principle of this strategy is that if two 3D mask candidates represent the same target, the intersection point set of them will basically cover the foreground points of the target. On the contrary, if they represent different targets, their intersection point set is empty, or at least they will not cover the two targets at the same time. By projecting the intersection points back to their respective 2D images and calculating the coverage rate, it can be judged whether the two 3D masks correspond to the same target.

[0057] First, for the 3D candidate mask with the highest confidence score

[0058] , iteratively check other candidate masks that intersect with it to see if they meet the coverage rate threshold. If the threshold condition is met, and are considered to represent the same target and are divided into the same group. This operation is repeated for all 3D mask candidates until the grouping of all candidates is completed. The formula for calculating the coverage rate is: where

[0059] represents the dilation operation on the sparse pixels of the point cloud, is the 2D mask obtained by projecting the intersection point cloud of the two 3D candidate masks to be judged onto their respective images, is the corresponding 2D mask, is is the number of pixel points.

[0060] Through the above method, all 3D mask candidates that represent the same target as can be found and form a group of 3D mask candidates , which is defined as follows:

[0061] where is a three - dimensional mask candidate group, is a three - dimensional mask candidate, represents a coverage judgment threshold, which is set to 0.5 in the present invention. For each among all the three - dimensional mask candidates, after performing the above operations, the three - dimensional mask candidates among all the mask candidates can be divided into multiple groups, and each group corresponds to a target in the three - dimensional world.

[0062] S6. Calculate the scores of the points in the three - dimensional mask candidate group, and determine the segmentation threshold through the Otsu threshold segmentation algorithm. The points with scores higher than the segmentation threshold are regarded as foreground points, and the set of foreground points is the target three - dimensional mask.

[0063] Specifically, after finding all the three - dimensional mask candidate groups belonging to a certain target, assign scores to each point in the three - dimensional mask candidate group to measure the possibility of it belonging to the target. This strategy can simultaneously utilize the spatial information of different shapes presented by the same target from different perspectives. By considering the number of observations and the observation angles, higher scores are assigned to the points that are more likely to be foreground points.

[0064] For the target , the set of points in all the three - dimensional mask candidates corresponding to it is denoted as . The point 's score is calculated as follows:

[0065]

[0066] where, represents the number of three - dimensional mask candidates containing the point , is a three - dimensional mask candidate, is a function for calculating the absolute value, is the observation angle closest to orthogonality in multiple perspectives. Assume represents the set of line segments connecting the point and the image exposure points corresponding to each mask candidate, then represents the angle closest to 90° among the angles formed by any two line segments in

[0067] Obviously, the more times the point is observed in different perspective images, the higher the probability of it belonging to the target. At the same time, the closer it is to 90°, the higher the confidence of the point belonging to the target. After assigning scores to each point, the Otsu threshold segmentation algorithm is used to determine the segmentation threshold between foreground points and background points, and the points with scores higher than The points are considered to belong to the target (i.e., the foreground points). The formula for generating the final target 3D mask is as follows:

[0068] By performing the above operations on each group of 3D mask candidates, a set of final 3D masks is generated 。 Figure 6 It is the recognition result of some categories of traffic infrastructure provided by the embodiments of this application.

[0069] Further, after step S6, the following steps are further included: S61, determining the target detection box corresponding to the target 3D mask; S62, cropping the target detection box to obtain a cropped picture; S63, inputting the cropped picture and the preset attribute information group into the multi-modal pre-trained model to obtain the matching degree between the cropped picture and each piece of attribute information in the attribute information group; S64, selecting the piece of attribute information with the highest matching degree as the additional attribute of the corresponding traffic infrastructure.

[0070] Specifically, for each 3D mask, find the target detection box with the highest target detection confidence corresponding to it, crop the image using the target detection bounding box enlarged by 1.5 times, input the cropped picture and a preset group of attribute information into the multi-modal pre-trained model, calculate the matching degree between the picture and each piece of the set attribute information, select the piece of attribute with the highest matching degree as the additional attribute of the traffic infrastructure, and finally obtain the target 3D mask and additional attribute of each traffic infrastructure. Figure 7 It is the recognition result of the attributes of some categories of traffic infrastructure provided by the embodiments of this application.

[0071] Among them, the multi-modal pre-trained model can be a CLIP model (Contrastive Language- Image Pre-Training).

[0072] This application combines point cloud and image data, and uses a multimodal large model to achieve efficient identification of traffic infrastructure. Specifically, first, through an open-set object detection technology based on text and visual cues, objects in street view images are identified; second, using the object detection bounding box as a cue, a SAM segmentation model is used to generate a 2D mask of the object; then, the 2D mask is projected onto the 3D point cloud to obtain 3D mask candidates. Through a 2D-3D spatial projection strategy bound to the point cloud, using multi-view repeated observation information and viewing angles, the 3D mask candidates are grouped, scored, and segmented to generate high-quality 3D masks of the object; finally, a multimodal model based on image-text contrast learning is used to extract multi-dimensional attribute information of the object, including fine semantics, geometric shape, and material features. This application has high adaptability, can identify traffic infrastructure in different urban environments, supports rich category expansion and fine-grained attribute extraction, improves the accuracy of traffic infrastructure identification, and further enhances the decision-making efficiency of infrastructure management and urban planning.

[0073] According to the method described in the above embodiments, this embodiment will be further described from the perspective of an open-set traffic infrastructure three-dimensional digital large model construction device. The open-set traffic infrastructure three-dimensional digital large model construction device can be specifically implemented as an independent entity, or integrated in an electronic device, which can be a terminal, a server, or other devices. Among them, the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a microprocessing box, or other devices.

[0074] Please refer to Figure 8 , Figure 8 Specifically describes the open-set traffic infrastructure three-dimensional digital large model construction device provided by the embodiments of this application, which is applied to an electronic device. The open-set traffic infrastructure three-dimensional digital large model construction device may include: Obtain street view pictures containing traffic infrastructure, point cloud data, and the prompt information corresponding to the street view pictures; Perform object detection on the street view picture based on the prompt information to obtain an object detection frame; Input the object detection frame into the segmentation model to obtain a 2D mask; Based on the geometric relationship between the 2D mask and the point cloud data, project the 2D mask into the three-dimensional space to obtain 3D mask candidates; Group the 3D mask candidates based on the intersection coverage judgment strategy to obtain a group of 3D mask candidates; Calculate the scores of the points in the three-dimensional mask candidate group, and determine the segmentation threshold through the Otsu threshold segmentation algorithm. The points with scores higher than the segmentation threshold are regarded as foreground points, and the set of the foreground points is the target three-dimensional mask.

[0075] In specific implementation, each of the above modules and / or units can be implemented as an independent entity, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of each of the above modules and / or units, please refer to the foregoing method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the foregoing method embodiments, which will not be elaborated herein.

[0076] In addition, an embodiment of the present application further provides an electronic device, which may be a device such as a computer or a tablet computer. The electronic device can implement the steps in any of the embodiments of the method for constructing an open-set traffic infrastructure three-dimensional digital large model provided by the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any of the methods for constructing an open-set traffic infrastructure three-dimensional digital large model provided by the embodiments of the present invention. For details, please refer to the foregoing embodiments, which will not be elaborated herein.

[0077] Figure 9 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown. The electronic device can be used to implement the method for constructing an open-set traffic infrastructure three-dimensional digital large model provided in the above embodiment. The electronic device 500 can be a device such as a terminal or a server. Among them, the terminal may include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.

[0078] The RF circuit 510 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit components for performing these functions. For example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 510 can communicate with various networks such as the Internet, an enterprise intranet, a wireless network or communicate with other devices through a wireless network. The above-mentioned wireless network may include a cellular phone network, a wireless local area network or a metropolitan area network. The above-mentioned wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, and may even include those protocols that have not been developed yet.

[0079] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizes functions such as taking pictures with the front camera, processing the captured images, and switching the display colors of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 520 may further include a memory remotely disposed relative to the processor 580, and these remote memories can be connected to the electronic device 500 through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The input unit 530 can be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function controls. The display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, and these graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541. Optionally, the display panel 541 can be configured in the form of an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode).

[0081] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, and the speaker 561 converts it into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and then converted into audio data. After the audio data is output to the processor 580 for processing, it is sent to another terminal, for example, through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may further include an earphone jack to provide communication between an external earphone and the electronic device 500.

[0082] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), and it provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it does not belong to the essential components of the electronic device 500 and can be omitted completely as needed without changing the essence of the invention.

[0083] The processor 580 is the control center of the electronic device 500, connecting various parts of the entire mobile phone through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 520, and by invoking the data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580 either.

[0084] The electronic device 500 also includes a power source 590 (such as a battery) for supplying power to each component. In some embodiments, the power source can be logically connected to the processor 580 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power source 590 may also include any components such as one or more DC or AC power sources, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0085] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be elaborated here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: An acquisition module, configured to acquire a street view image containing traffic infrastructure, point cloud data, and prompt information corresponding to the street view image; A target detection module, configured to perform target detection on the street view image based on the prompt information to obtain a target detection frame; A two-dimensional mask generation module, configured to input the target detection frame into a segment-anything model to obtain a two-dimensional mask; A three-dimensional mask candidate generation module, configured to project the two-dimensional mask into three-dimensional space based on the geometric relationship between the two-dimensional mask and the point cloud data to obtain a three-dimensional mask candidate; A grouping module, configured to group the three-dimensional mask candidates based on a communication coverage judgment strategy to obtain a three-dimensional mask candidate group; A target three-dimensional mask generation module is used to calculate the scores of the points in the three-dimensional mask candidate group, determine a segmentation threshold through the Otsu threshold segmentation algorithm, take the points with scores higher than the segmentation threshold as foreground points, and set the set of the foreground points as the target three-dimensional mask.

[0086] In specific implementation, each of the above modules can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above modules, reference can be made to the foregoing method embodiments, which will not be elaborated herein.

[0087] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the foregoing embodiments can be completed by instructions, or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. For this purpose, an embodiment of the present invention provides a storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps of any one of the embodiments of the method for constructing an open-set traffic infrastructure three-dimensional digital model provided by the embodiments of the present invention.

[0088] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc, etc.

[0089] Since the instructions stored in the storage medium can execute the steps of any one of the embodiments of the method for constructing an open-set traffic infrastructure three-dimensional digital model provided by the embodiments of the present invention, the beneficial effects achievable by any of the methods for constructing an open-set traffic infrastructure three-dimensional digital model provided by the embodiments of the present invention can be achieved. For details, reference can be made to the foregoing embodiments, which will not be elaborated herein.

[0090] The foregoing has introduced in detail a method, an apparatus, a storage medium, and an electronic device for constructing an open-set traffic infrastructure three-dimensional digital model provided by the embodiments of the present application. Specific examples are used herein to elaborate the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for constructing a large three-dimensional digital model of an open set of transportation infrastructure, characterized in that: include: Obtaining a street view image including traffic infrastructure, point cloud data, and prompt information corresponding to the street view image; Performing target detection on the street view image based on the prompt information to obtain a target detection frame; Input the target detection box into the segmentation model to obtain a two-dimensional mask; Based on the geometric relationship between the two-dimensional mask and the point cloud data, projecting the two-dimensional mask into a three-dimensional space to obtain a three-dimensional mask candidate; Grouping the three-dimensional mask candidates based on a communicative coverage judgment strategy to obtain a three-dimensional mask candidate group; The scores of the points in the three-dimensional mask candidate group are calculated, and the segmentation threshold is determined by the Otsu threshold segmentation algorithm, and the points with scores higher than the segmentation threshold are used as foreground points, and the set of foreground points is used as the target three-dimensional mask.

2. The method for constructing a large three-dimensional digital model of an open set of transportation infrastructure according to claim 1 is characterized in that: The prompt information includes text prompt information and visual prompt information, and the target detection is performed on the street view image based on the prompt information to obtain a target detection frame, including: Inputting the text prompt information and the street view image into an open set object detection model to obtain an object detection frame; The visual cue information and the street view image are input into an open set object detection model to obtain an object detection frame.

3. The method for constructing a large 3D digital model of an open set of transportation infrastructure according to claim 2 is characterized in that: Inputting the text prompt information and the street view image into an open set object detection model to obtain an object detection frame, including: The target detection frame is obtained by the first formula, which is: in, is an open set object detection model. is a panoramic image of the street view, is a textual hint describing the target category, is the target detection box, is the set of bounding boxes of the target, is the corresponding semantic category; The step of inputting the visual cue information and the street view image into an open set object detection model to obtain an object detection frame includes: The target detection frame is obtained by the second formula, which is: in, is another open set object detection model, It is a visual cue. is the target detection box, and They are the detected bounding box and the corresponding semantic category respectively.

4. The method for constructing a large 3D digital model of an open set of transportation infrastructure according to claim 1 is characterized in that: The target detection frame is input into the segmentation model to obtain a two-dimensional mask, including: The two-dimensional mask is obtained by the third formula, which is: in, and Respectively represent the number of target detection bounding boxes obtained using textual prompt information and visual prompt information, Indicates splitting all models, is the input image, is the bounding box set for target detection, is the output 2D mask.

5. The method for constructing a large three-dimensional digital model of an open set of transportation infrastructure according to claim 1 is characterized in that: The projecting the two-dimensional mask into a three-dimensional space based on the geometric relationship between the two-dimensional mask and the point cloud data to obtain a three-dimensional mask candidate includes: According to the external parameter matrix and the internal parameter matrix of the camera, the point cloud data is projected onto the image plane to obtain a plurality of projection points; Taking the projection points falling within each mask range in the two-dimensional mask as three-dimensional mask candidate points; The set of three-dimensional mask candidate points is used as a three-dimensional mask candidate.

6. The method for constructing a large 3D digital model of an open set of transportation infrastructure according to claim 1, characterized in that: The three-dimensional masks are grouped based on the communication coverage judgment strategy to obtain a three-dimensional mask candidate group, including: Calculating the confidence of the three-dimensional mask candidate; Compute the 3D mask candidate with the highest confidence and other 3D mask candidates that intersect with the 3D mask candidate with the highest confidence If the coverage ratio is greater than the coverage ratio threshold, the two three-dimensional mask candidates are divided into the same three-dimensional mask candidate group.

7. The method for constructing a large 3D digital model of an open set of transportation infrastructure according to claim 6 is characterized in that: The calculation formula of the coverage is: in, represents the dilation operation on sparse pixels of the point cloud, It is the two-dimensional mask projected onto the respective images by the intersection point cloud of two three-dimensional mask candidates to be judged. is the corresponding two-dimensional mask, yes The number of pixels; The three-dimensional mask candidate set is defined as: in, is a 3D mask candidate group, Indicates the coverage judgment threshold, is a 3D mask candidate, is the convolution operation.

8. The method for constructing a large 3D digital model of an open set of transportation infrastructure according to claim 1, characterized in that: The calculating the scores of the points in the three-dimensional mask candidate group comprises: The score is calculated using the following formula: in, Indicates that it contains a point The number of 3D mask candidates, is a 3D mask candidate, is a function that calculates the absolute value, It is the closest orthogonal observation angle among multiple perspectives.

9. The method for constructing a large 3D digital model of an open set of transportation infrastructure according to claim 1, characterized in that: After the step of gathering the foreground points into a target three-dimensional mask, the method further comprises: Determine a target detection frame corresponding to the target three-dimensional mask; Cropping the target detection frame to obtain a cropped image; Inputting the cropped image and the preset attribute information group into a multimodal pre-training model to obtain a matching degree between the cropped image and each attribute information in the attribute information group; The piece of attribute information with the highest matching degree is selected as the additional attribute of the corresponding traffic infrastructure setting.

10. An open set of transportation infrastructure three-dimensional digital large model construction device, characterized in that: include: An acquisition module, used to acquire a street view image containing traffic infrastructure, point cloud data, and prompt information corresponding to the street view image; A target detection module, used to perform target detection on the street view image based on the prompt information to obtain a target detection frame; A two-dimensional mask generation module, used for inputting the target detection frame into the segmentation model to obtain a two-dimensional mask; A three-dimensional mask candidate generation module, used for projecting the two-dimensional mask into a three-dimensional space based on the geometric relationship between the two-dimensional mask and the point cloud data to obtain a three-dimensional mask candidate; A grouping module, used for grouping the three-dimensional mask candidates based on a communication coverage judgment strategy to obtain a three-dimensional mask candidate group; The target 3D mask generation module is used to calculate the scores of the points in the 3D mask candidate group, and determine the segmentation threshold by Otsu threshold segmentation algorithm, taking the points with scores higher than the segmentation threshold as foreground points, and taking the set of foreground points as the target 3D mask.

Images