A digital twin three-dimensional scene modeling layout method and system

Abstract

The present invention relates to a digital twin technology, and discloses a method for arranging a digital twin three-dimensional scene modeling, including judging whether the facilities in the object annotation box that appear at least twice in the key frame of the second video data match the pre-stored model, and if so, arranging the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the facilities of the first three-dimensional model and the second video data, and outputting the second three-dimensional model; judging whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, deleting the key frame of at least one pre-stored model that does not conform. The present invention automatically generates a more accurate digital twin model through the second video data and the pre-stored models pre-stored in the database, which is convenient for increasing the clarity of the user's observation of the digital twin model, improving the efficiency of three-dimensional modeling, reducing the modeling time, and reducing the computing pressure on the three-dimensional scene modeling arrangement of the digital twin model.

Description

Technical Field

[0001] The present invention relates to a digital twin technology, and in particular to a method and system for arranging modeling of a digital twin three-dimensional scene. Background Art

[0002] In the digital twin system, the most critical technical indicator is how to efficiently and accurately arrange the three-dimensional scene. Traditional three-dimensional modeling methods are often time-consuming and costly, and it is difficult to meet the modeling needs of large-scale scenes. In addition, since the accuracy of the three-dimensional model generated by the visual algorithm is limited by the collected pixels, it is difficult to generate a high-precision three-dimensional model.

[0003] In order to increase the effect of 3D model generation, the current mainstream method is the multi-frame generation method of the moving process, that is, a visual model is generated through multiple frames. However, this visual model assumes a simple algorithm of multiple interval frames, which is undoubtedly not smart enough, increases computing pressure, and reduces the generation speed of the digital twin model and the quality of the 3D model.

[0004] Therefore, there is a need for a method and system for arranging modeling of a digital twin three-dimensional scene that can reduce computing pressure and improve the quality of three-dimensional modeling. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for arranging modeling of a digital twin three-dimensional scene, which can reduce computing pressure and improve the quality of three-dimensional modeling.

[0006] In a first aspect, the present invention provides a digital twin three-dimensional scene modeling and arrangement method, comprising:

[0007] Using a laser scanner to capture first laser data, and generating a first three-dimensional model according to the first laser data;

[0008] Using a camera to shoot second video data, determining whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match the pre-stored model, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model;

[0009] Determining whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, deleting a key frame of at least one pre-stored model that does not conform;

[0010] Performing texture mapping on the second three-dimensional model through the second video data to generate a third three-dimensional model;

[0011] Output the third 3D model.

[0012] The present invention provides a digital twin three-dimensional scene modeling and arrangement method, wherein a camera is used to shoot second video data, and it is determined whether facilities in an object annotation box that appear at least twice in a key frame of the second video data match a pre-stored model. If they match, the pre-stored model is configured on the first three-dimensional model according to the scaling ratio and placement angle of the facilities of the first three-dimensional model and the second video data, and the second three-dimensional model is output, including:

[0013] S210, using a camera to capture second video data;

[0014] S220, configuring an object annotation frame for each closed image of the second video data;

[0015] S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point; dividing each frame before and after the unit time into interval segments according to the interval point; wherein, if there is an interval point within the unit time, jump to S240, if there is no interval point within the unit time, merge the unit time with the unit time adjacent thereto, and re-execute S230;

[0016] S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, if yes, configuring the two adjacent intervals as a first change group; judging whether the change in the number of annotation boxes of two adjacent intervals in the first change group exceeds a third threshold, if not, selecting a frame of the interval with more object annotation boxes as a key frame; if yes, selecting a frame of the first interval of the first change group as a key frame, or lowering the first threshold and jumping to S230;

[0017] S250, configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the first key frame as the first projection image; configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the other key frames as the second projection image; configuring the pre-stored model that matches the first projection image as the first model, and configuring the angle of capturing the pre-stored model that matches the first projection image as the third angle; according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, adjusting the third angle in the direction from the first acquisition angle to the second acquisition angle to a fourth angle, judging whether the third projection image captured by the first model at the fourth angle matches the second projection image, and if so, scaling the pre-stored model to the third projection image to be consistent with the second projection image, and then scaling the pre-stored model to the first projection image to be consistent with the model image of the pre-stored model captured at the third angle; and configuring the pre-stored model at the same position of the first three-dimensional model as the second projection image, and outputting the second three-dimensional model.

[0018] The present invention provides a digital twin three-dimensional scene modeling and arrangement method, wherein it is determined whether at least two pre-stored models in a second three-dimensional model conform to a relative preset configuration relationship, and if not, a key frame of at least one pre-stored model that does not conform is deleted, including

[0019] S310, pre-storing a preset configuration relationship, wherein the configuration relationship includes a relative position relationship between at least two pre-storing models;

[0020] S320, determine whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models; if not, count whether other groups of pre-stored models other than the group of at least two pre-stored models and including one of the group of at least two pre-stored models conform to the pre-stored relative position relationship, delete all key frames of the generated pre-stored model with the least number of conformities, and jump to S250.

[0021] The present invention provides a digital twin three-dimensional scene modeling and arrangement method, wherein it is determined whether a group of at least two pre-stored models in a second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models, and further includes

[0022] If it matches, delete the object annotation box corresponding to the pre-stored model.

[0023] The present invention provides a method for arranging digital twin three-dimensional scene modeling, wherein in the step of selecting a frame of an interval segment with more object annotation frames as a key frame in S240 and taking a frame of a first interval segment of the first change group as a key frame, a method of taking a frame of a segment as a key frame is:

[0024] It is determined whether the key frame deleted in the above step S320 is located in this interval segment. If so, the interval segment is deleted and step S240 is executed again. If not, the middle frame of the frames arranged before and after the interval segment is used as the key frame.

[0025] In a second aspect, the present invention provides a digital twin three-dimensional scene modeling and arrangement system, comprising:

[0026] A drone, which uses a laser scanner to shoot first laser data and generates a first three-dimensional model according to the first laser data; and uses a camera to shoot second video data;

[0027] A processor is used to determine whether the facilities in the object annotation box that appear at least twice in the key frame of the second video data match the pre-stored model, and if so, configure the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model; determine whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, delete the key frame of at least one pre-stored model that does not conform; perform texture mapping on the second three-dimensional model through the second video data to generate a third three-dimensional model;

[0028] The digital twin output module is used to output the third three-dimensional model.

[0029] The present invention provides a digital twin three-dimensional scene modeling and arrangement system, wherein a camera is used to shoot second video data, and it is determined whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match a pre-stored model. If they match, the pre-stored model is configured on the first three-dimensional model according to the scaling ratio and placement angle of the facilities of the first three-dimensional model and the second video data, and the second three-dimensional model is output, including

[0030] S210, using a camera to capture second video data;

[0031] S220, configuring an object annotation frame for each closed image of the second video data;

[0032] S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point; dividing each frame before and after the unit time into interval segments according to the interval point; wherein, if there is an interval point within the unit time, jump to S240, if there is no interval point within the unit time, merge the unit time with the unit time adjacent thereto, and re-execute S230;

[0033] S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, if yes, configuring the two adjacent intervals as a first change group; judging whether the change in the number of annotation boxes of two adjacent intervals in the first change group exceeds a third threshold, if not, selecting a frame of the interval with more object annotation boxes as a key frame; if yes, selecting a frame of the first interval of the first change group as a key frame, or lowering the first threshold and jumping to S230;

[0034] S250, configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the first key frame as the first projection image; configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the other key frames as the second projection image; configuring the pre-stored model that matches the first projection image as the first model, and configuring the angle of capturing the pre-stored model that matches the first projection image as the third angle; according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, adjusting the third angle in the direction from the first acquisition angle to the second acquisition angle to a fourth angle, judging whether the third projection image captured by the first model at the fourth angle matches the second projection image, and if so, scaling the pre-stored model to the third projection image to be consistent with the second projection image, and then scaling the pre-stored model to the first projection image to be consistent with the model image of the pre-stored model captured at the third angle; and configuring the pre-stored model at the same position of the first three-dimensional model as the second projection image, and outputting the second three-dimensional model.

[0035] The present invention provides a digital twin three-dimensional scene modeling and arrangement system, wherein it is determined whether at least two pre-stored models in a second three-dimensional model conform to a relative preset configuration relationship, and if not, a key frame of at least one pre-stored model that does not conform is deleted, including

[0036] S310, pre-storing a preset configuration relationship, wherein the configuration relationship includes a relative position relationship between at least two pre-storing models;

[0037] S320, determine whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models; if not, count whether other groups of pre-stored models other than the group of at least two pre-stored models and including one of the group of at least two pre-stored models conform to the pre-stored relative position relationship, delete all key frames of the generated pre-stored model with the least number of conformities, and jump to S250.

[0038] The present invention provides a digital twin three-dimensional scene modeling and arrangement system, wherein it is determined whether a group of at least two pre-stored models in a second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models, and further includes

[0039] If it matches, delete the object annotation box corresponding to the pre-stored model.

[0040] The present invention provides a digital twin three-dimensional scene modeling and arrangement system, wherein in the step of selecting a frame of an interval segment with more object annotation frames as a key frame in S240 and taking a frame of a first interval segment of the first change group as a key frame, a method of taking a frame of a segment as a key frame is:

[0041] It is determined whether the key frame deleted in the above step S320 is located in this interval segment. If so, the interval segment is deleted and step S240 is executed again. If not, the middle frame of the frames arranged before and after the interval segment is used as the key frame.

[0042] The difference between the digital twin three-dimensional scene modeling and arrangement method of the present invention and the prior art is that the digital twin three-dimensional scene modeling and arrangement method of the present invention uses the above-mentioned intelligent algorithm to automatically generate a more accurate digital twin model for the facilities that should be generated in the first three-dimensional model originally generated by the laser scanner through the second video data and the pre-stored model pre-stored in the database, which is convenient for increasing the clarity of the user's observation of the digital twin model, improving the three-dimensional modeling efficiency, reducing the modeling time, and reducing the computing pressure on the three-dimensional scene modeling and arrangement of the digital twin model. At the same time, the texture mapping of the second three-dimensional model by the second video data improves the realism of the model.

[0043] The following is a further description of a digital twin three-dimensional scene modeling and layout method of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a method for modeling and arranging a digital twin three-dimensional scene. DETAILED DESCRIPTION

[0045] like Figure 1 As shown, the present invention provides a digital twin three-dimensional scene modeling and layout method, including

[0046] S100, photographing first laser data using a laser scanner, and generating a first three-dimensional model according to the first laser data;

[0047] S200, using a camera to shoot second video data, determining whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match pre-stored models, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model;

[0048] S300, determining whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, if not, deleting a key frame of at least one pre-stored model that does not conform, and jumping to S200, if conforming, jumping to S400;

[0049] S400, performing texture mapping on the second three-dimensional model using the second video data to generate a third three-dimensional model;

[0050] S500: Output the third three-dimensional model.

[0051] The present invention uses the above-mentioned intelligent algorithm to automatically generate a more accurate digital twin model from the facilities that should have been generated in the first three-dimensional model originally generated by the laser scanner through the second video data and the pre-stored model pre-stored in the database, which is convenient for increasing the clarity of the user's observation of the digital twin model, improving the efficiency of three-dimensional modeling, reducing the modeling time, and reducing the computing pressure on the three-dimensional scene modeling layout of the digital twin model. At the same time, the texture mapping of the second three-dimensional model by the second video data improves the realism of the model.

[0052] Among them, the above-mentioned laser scanners and cameras can be configured on drones. When the drones are flying over the top of the industrial park, they can scan the industrial park and simultaneously generate real-time first laser data and second video data, which can be one segment of data or multiple segments of data or real-time live broadcast data.

[0053] When the present invention is implemented, a section of first laser data and second video data can be collected by an unmanned aerial vehicle and sent to a local location for generating a three-dimensional model, or the three-dimensional model can be repeatedly generated by live broadcasting of the first laser data and second video data to ensure the quality and accuracy of the details of the three-dimensional model.

[0054] Among them, texture mapping is performed on the second three-dimensional model through the second video data to generate a third three-dimensional model. It can be understood that the second three-dimensional model may be relatively accurate at the model level, but with the passage of time, the texture on the surface of the three-dimensional model caused by lighting is different. The present invention can perform texture mapping through software such as Unity, Unreal Engine, etc., so as to generate a third three-dimensional model that is closer to the actual color and texture.

[0055] In some embodiments, see Figure 1 , using a camera to shoot second video data, determining whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match the pre-stored model, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the facilities in the first three-dimensional model and the second video data to output the second three-dimensional model, including

[0056] S210, using a camera to capture second video data;

[0057] S220, configuring an object annotation frame for each closed image of the second video data;

[0058] S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point; dividing each frame before and after the unit time into interval segments according to the interval point; wherein, if there is an interval point within the unit time, jump to S240, if there is no interval point within the unit time, merge the unit time with the unit time adjacent thereto, and re-execute S230;

[0059] S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, if yes, configuring the two adjacent intervals as a first change group; judging whether the change in the number of annotation boxes of two adjacent intervals in the first change group exceeds a third threshold, if not, selecting a frame of the interval with more object annotation boxes as a key frame; if yes, selecting a frame of the first interval of the first change group as a key frame, or lowering the first threshold and jumping to S230;

[0060] S250, configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the first key frame as the first projection image; configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the other key frames as the second projection image; configuring the pre-stored model that matches the first projection image as the first model, and configuring the angle of capturing the pre-stored model that matches the first projection image as the third angle; according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, adjusting the third angle in the direction from the first acquisition angle to the second acquisition angle to a fourth angle, judging whether the third projection image captured by the first model at the fourth angle matches the second projection image, and if so, scaling the pre-stored model to the third projection image to be consistent with the second projection image, and then scaling the pre-stored model to the first projection image to be consistent with the model image of the pre-stored model captured at the third angle; and configuring the pre-stored model at the same position of the first three-dimensional model as the second projection image, and outputting the second three-dimensional model.

[0061] The present invention divides the unit time into multiple intervals by the rate of change of adjacent frames; then determines in which interval the key frame of this unit time appears by the amplitude of the change in the number of object annotation boxes in adjacent intervals, thereby obtaining multiple relatively accurate key frames that meet the purpose of shooting; then the matching pre-stored model is configured in the first three-dimensional model after scaling, rotation, and position adjustment to output the second three-dimensional model, thereby directly adding a pre-stored model that does not need to be rendered to replace the facilities that need to be rendered more finely in the original first three-dimensional model. This undoubtedly increases the rendering speed, reduces the rendering and generation pressure, and even improves the rendering quality in a bird's-eye view. After all, many details cannot be captured by looking down, but we use an existing pre-stored model to configure it in a specified position, and even add additional upward-looking details and blind spot details to the second three-dimensional model.

[0062] Among them, S220, configuring an object annotation frame for the closed image of each frame of the second video data, can be understood as: a closed image recognition algorithm similar to the image-free version of the autonomous driving algorithm can be used to configure a corresponding object annotation frame for all closed images. For example, when Tesla is unmanned, it will mark a closed object in the video using an object annotation frame, and as Tesla and the above-mentioned object move relative to each other, the object annotation frame that appears in the video will also move relatively. Then, when the drone flies over the industrial park to build a digital twin model of the industrial park, we can use the above-mentioned object annotation frame to lock the facilities that need to generate a digital twin model or need to be replaced with a pre-stored model, so that the subsequent algorithm can recognize and replace them.

[0063] Among them, S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point, can be understood as: the adjacent change rate of each frame before and after can be obtained by the image comparison algorithm of the prior art, for example, the similarity between the face of an ordinary person and a star is obtained as a quantified number. In the present invention, the adjacent change rate can be used to understand whether the video data has changed too much, or whether there has been a sudden scene change. Once the above-mentioned sudden scene change occurs, an interval point can be generated at this time. Among them, the first threshold value can be 2%, or other set values. For example, the acquisition frame rate of a drone is 24 frames per second. In other words, the image changes by 2% after 0.04 seconds. Then, a large change must have occurred at this time, for example, the drone is subjected to unstable vibrations caused by high-altitude airflow. Then, at this time, the relatively stable picture before should be taken as an interval segment for research.

[0064] The unit time may be 2 to 100 seconds, preferably 5 seconds.

[0065] Among them, each frame before and after the unit time is divided into interval segments according to the interval points. It can be understood that: an interval segment can be better generated through the above interval points. For example, in a video data with a unit time of 5 seconds, there are two interval points, then there should be 3 interval segments.

[0066] Among them, if there is an interval point in the unit time, jump to S240. If there is no interval point in the unit time, the unit time is merged with the adjacent unit time after it, and S230 is re-executed. It can be understood that if there is an interval point in the unit time, then an interval segment must be generated, which can be operated by the subsequent step S240. If there is no interval point, then it is possible that the drone has not moved in the qualified time period. If there are too many key frames with too small movement, it is not conducive to whether the multi-angle acquisition facility is similar to the pre-stored model. Then, this unit time can be merged with the adjacent unit time after it, and S230 is re-executed. That is, there is no interval point in this 5-second unit time, then this 5-second unit time should be merged with the next 5-second unit time, so as to merge a 10-second segment, and re-run S230 with this segment. It should be noted that only the unit time of this segment is different, and other unit times should be the original unit time. That is, 5 seconds are merged into 5 seconds, then only this segment is 10 seconds, and other segments are still 5 seconds.

[0067] Among them, S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, can be understood as: the second threshold can be 40, that is, when the drone collects the second video data at 24 frames per second, the interval time below 40 frames is too short, which may be due to bumpy flight. At this time, the camera diagonal and white balance of the drone will have problems. Therefore, the interval time below 40 frames cannot generate key frames.

[0068] Moreover, the above-mentioned intervals should be adjacent to have reference value.

[0069] If there are, the two adjacent intervals are configured as the first variation group, which can be understood as: the adjacent intervals can be a first variation group. For example, if A, B, and C are three intervals, then the first variation group can be A and B, or B and C. Both can be used as the first variation group for subsequent research.

[0070] Among them, judging whether the change in the number of annotation frames of two adjacent intervals of the first change group exceeds the third threshold can be understood as follows: when studying a first change group, if the change in the number of annotation frames of two adjacent intervals of the first change group is large, it means that the image of the scanned industrial park has undergone a large change, which will cause so many object annotation frames to change at once. The third threshold can be 1 to 100, preferably 3. That is to say, in the same first change group, if the number of object annotation frames of two intervals increases or decreases by more than 3, then it is considered to exceed the third threshold.

[0071] Among them, if it does not exceed, then a frame of the interval with more object annotation frames is selected as the key frame, which can be understood as: if it does not exceed, the change range of the two intervals representing the first change group is not large, then the interval with the most object annotation frames in the first change group can be selected to select the key frame. It should be noted that when the number of object annotation frames in two intervals is the same, one interval can be randomly selected to select the key frame, or the last interval can be selected to select the key frame.

[0072] When selecting key frames, the present invention usually selects an interval segment of more object annotation boxes to generate key frames when the changes are not large, so that multiple interval segments in the same unit time can be denoised, and a relatively continuous interval segment that is better used as a reference for a pre-stored model is selected for subsequent analysis.

[0073] The first one described in the present invention is an early appearing interval segment, and the last one is a late appearing interval segment.

[0074] It should be noted that the interval segment represents a plurality of continuous frames, and the number of object annotation frames counted can be the total number of all object annotation frames collected in all frames of this interval segment, or the average number of each frame. For example, the object annotation frames collected in the first frame are A, B, and C, and the object annotation frames collected from the second frame to the last frame are B, C, and D, then the object annotation frames collected in this interval segment should be 4, namely A, B, C, and D. Or for example, the object annotation frames collected in the first frame are A, B, and C, and the object annotation frames collected from the second frame to the last frame are B, C, and D, then the average number of object annotation frames collected in each frame of this interval segment should be 3. The present invention can select one of the above methods throughout the article to count the number of object annotation frames in the same interval segment.

[0075] Among them, if it exceeds, a frame of the first interval segment of the first change group is taken as a key frame, or the first threshold is lowered and jump to S230. It can be understood that: if it exceeds, it means that the changes in the object annotation boxes of the two interval segments of the first change group are large, so the interval segment that appears earlier should be selected to extract the key frame, so that the key frame is away from the position where the distance change is large and the interval point may appear, so as to increase the clarity and reliability of the key frame.

[0076] Alternatively, if it exceeds, the first threshold should be lowered to reduce the number of interval points, so that more interval segments can be used to generate a setting that is more in line with not exceeding the third threshold, so that more and more suitable interval segments for selecting key frames can be captured, and the key frames can appear with less noise and better clarity, which is more conducive to studying how to configure the pre-stored model to the first three-dimensional model.

[0077] The present invention only selects one of the above methods throughout the text. Preferably, if it exceeds, the first threshold should be lowered and the execution should be repeated starting from S230.

[0078] The first threshold can be lowered by 10% each time until it is lowered three times. If it still cannot exceed the third threshold, the process will end directly.

[0079] Among them, S250, the projection surface screenshot of the facilities in the same object annotation box that appear at least twice in each key frame of the second video data in the first key frame is configured as the first projection image; the projection surface screenshot of the facilities in the same object annotation box that appear at least twice in each key frame of the second video data in the other key frames is configured as the second projection image, which can be understood as: if the facilities in the same object annotation box of all key frames generated by the present invention according to each unit time and each interval segment appear twice, then due to the movement of the drone and the elimination of the redundant stagnant state unit time or interval segment, then the facilities in the same object annotation box in different key frames must be collected at different angles. Although it is the same object, its projection image in the second video data is different, so it is configured as the first projection image and the second projection image. Of course, it can also appear more than twice, such as appearing in three key frames, appearing in four key frames, etc., then all images except the first projection image are second projection images.

[0080] The pre-stored model that matches the first projection image is configured as the first model, and the angle of the captured pre-stored model that matches the first projection image is configured as the third angle. It can be understood that if the first projection image can match the pre-stored model, it must match the projection surface of one of the angles of the pre-stored model. This angle is defined as the third angle for subsequent analysis and processing.

[0081] The image matching judgment method used in the present invention is similar to the current mainstream face recognition unlocking method. That is, the face recognition unlocking method for judging whether the face matches is used to judge whether the second video data is matched with multiple pre-stored models.

[0082] Among them, according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, the third angle is adjusted to the fourth angle along the direction from the first acquisition angle to the second acquisition angle. It can be understood that: the first acquisition angle and the second acquisition angle capture the same facility, and the facility is a three-dimensional object, and it should be compared with the three-dimensional pre-stored model from multiple angles to output a more correct pre-stored model. Then, according to the change from the first acquisition angle to the second acquisition angle, the third angle is adjusted to the fourth angle, so that the second projection image can be compared with the third projection image of the pre-stored model captured at the fourth angle.

[0083] Among them, it is judged whether the third projection image collected by the first model at the fourth angle matches the second projection image. If they match, the pre-stored model is scaled to the third projection image to be consistent with the second projection image, and then the pre-stored model is scaled to the first projection image to be consistent with the model image of the pre-stored model collected at the third angle; and the pre-stored model is configured at the same position of the first three-dimensional model as the second projection image, and the second three-dimensional model is output. It can be understood that if the third projection image matches the fourth projection image, the pre-stored model is scaled according to the ratio of the third projection image to the second projection image, and then the pre-stored model is scaled in the same way as the first projection image is scaled to the second projection image, so that the sizes of the X, Y, and Z axes of the pre-stored model can be adjusted to appropriate sizes, and each scaling does not necessarily scale all the X, Y, and Z axes. For example, in the step of scaling the pre-stored model according to the ratio of the third projection image to the second projection image, the scale may be the scale of the pre-stored image of the XY axis, and then the pre-stored model is scaled in the same way as the first projection image is scaled to the second projection image. The scale may be the scale of the pre-stored image of the YZ axis. After scaling, the pre-stored image is configured in the first three-dimensional model at the same position as the subsequent second projection image, so that a second three-dimensional model with the pre-stored model can be output.

[0084] It should be noted that, before generating the second three-dimensional model, the steps before generating the second three-dimensional model may be repeated, that is, the first three-dimensional model may be configured with a plurality of pre-existing models to generate the second three-dimensional model.

[0085] In some embodiments, see Figure 1 , determining whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, deleting a key frame of at least one pre-stored model that does not conform, including

[0086] S310, pre-storing a preset configuration relationship, wherein the configuration relationship includes a relative position relationship between at least two pre-storing models;

[0087] S320, determine whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models; if not, count whether other groups of pre-stored models other than the group of at least two pre-stored models and including one of the group of at least two pre-stored models conform to the pre-stored relative position relationship, delete all key frames of the generated pre-stored model with the least number of conformities, and jump to S250.

[0088] The present invention pre-stores the relative position relationship between at least two pre-stored models as a criterion for judging whether the generated second three-dimensional model is reasonable, thereby judging the position relationship of the second three-dimensional model through the identified pre-stored models, for example, a person must be located on a building and cannot be located under a building, a bridge must have piers under the bridge, etc., to find abnormal position relationships, thereby determining whether this relationship group is suspicious, and then by comparing the correctness with other pre-stored relative relationships, pre-stored models that are different from most relationships are eliminated, thereby achieving automatic noise reduction of the third three-dimensional model, avoiding the generation of erroneous pre-stored models based on erroneous frames of video data caused by operations such as overexposure, inaccurate focus, and jello deformation of video data caused by bumps on the generated second three-dimensional model, and after finding the erroneous frames, all are deleted, and then jump to S240 for regeneration, that is, there is no such deleted key frame in the video data in step S240.

[0089] Among them, S310, pre-store preset configuration relationship, wherein the configuration relationship includes the relative position relationship between at least two pre-stored models, which can be understood as: the relative position relationship between at least two pre-stored models can be, for example, that a person should stand on the house, rather than under the house, which is obviously unreasonable. We should regard the relative person on the house as a relative position relationship, and judge whether the two pre-stored models are reasonable.

[0090] Among them, if it does not match, then count whether other groups of pre-stored models except the group of at least two pre-stored models, which include one of the group of at least two pre-stored models, meet the pre-stored relative position relationship, and then delete all key frames of the generated pre-stored model with the least number of matches, and jump to S250. It can be understood that: if the present invention encounters contradictory relationships between pre-stored models, it must not meet the pre-stored relative position relationship, but it is not clear which one is wrong, then the relative relationships of other pre-stored models should be compared, and the ones with more errors should be found and all key frames corresponding to the pre-stored models should be deleted, and then jump to S250 to match again.

[0091] In some embodiments, see Figure 1 , determining whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models, and further comprising

[0092] If it matches, delete the object annotation box corresponding to the pre-stored model.

[0093] The present invention deletes the object annotation box of the pre-stored model that has been verified to be correctly generated, so that there is no need to consider the model comparison and generation of this position in the future, thereby reducing the computing power pressure and improving the efficiency of data collection and generation of digital twin models.

[0094] In some embodiments, see Figure 1 In step S240, a frame of an interval segment with more object annotation frames is selected as a key frame, and a frame of a first interval segment of the first variation group is selected as a key frame. The method of selecting a frame of a segment as a key frame is:

[0095] It is determined whether the key frame deleted in the above step S320 is located in this interval segment. If so, the interval segment is deleted and step S240 is executed again. If not, the middle frame of the frames arranged before and after the interval segment is used as the key frame.

[0096] In the present invention, since the above S240 may be a step that is repeatedly executed after the key frame has been deleted, each interval segment may have the problem of key frames being deleted, and each video segment may also have relatively long-term errors. To generate different video segments, the interval segment with the key frame having errors should be deleted, and step S240 should be executed again. The present invention uses the middle frame located in the middle of a segment as the key frame, and can select a point that is relatively farthest from the change point in the interval segment of video data with little change fluctuation as the key frame, thereby ensuring that the generated key frame has good clarity and avoiding the occurrence of error frames.

[0097] like Figure 1 As shown, the present invention provides a digital twin three-dimensional scene modeling and layout system, including

[0098] A drone, which uses a laser scanner to shoot first laser data and generates a first three-dimensional model according to the first laser data; and uses a camera to shoot second video data;

[0099] A processor is used to determine whether the facilities in the object annotation box that appear at least twice in the key frame of the second video data match the pre-stored model, and if so, configure the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model; determine whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, delete the key frame of at least one pre-stored model that does not conform; perform texture mapping on the second three-dimensional model through the second video data to generate a third three-dimensional model;

[0100] The digital twin output module is used to output the third three-dimensional model.

[0101] The present invention uses the above-mentioned intelligent algorithm to automatically generate a more accurate digital twin model from the facilities that should have been generated in the first three-dimensional model originally generated by the laser scanner through the second video data and the pre-stored model pre-stored in the database, which is convenient for increasing the clarity of the user's observation of the digital twin model, improving the efficiency of three-dimensional modeling, reducing the modeling time, and reducing the computing pressure on the three-dimensional scene modeling layout of the digital twin model. At the same time, the texture mapping of the second three-dimensional model by the second video data improves the realism of the model.

[0102] In some embodiments, see Figure 1 , using a camera to shoot second video data, determining whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match the pre-stored model, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the facilities in the first three-dimensional model and the second video data to output the second three-dimensional model, including

[0103] S210, using a camera to capture second video data;

[0104] S220, configuring an object annotation frame for each closed image of the second video data;

[0105] S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point; dividing each frame before and after the unit time into interval segments according to the interval point; wherein, if there is an interval point within the unit time, jump to S240, if there is no interval point within the unit time, merge the unit time with the unit time adjacent thereto, and re-execute S230;

[0106] S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, if yes, configuring the two adjacent intervals as a first change group; judging whether the change in the number of annotation boxes of two adjacent intervals in the first change group exceeds a third threshold, if not, selecting a frame of the interval with more object annotation boxes as a key frame; if yes, selecting a frame of the first interval of the first change group as a key frame, or lowering the first threshold and jumping to S230;

[0107] S250, configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the first key frame as the first projection image; configuring the projection surface screenshot of the facility in the same object annotation box that appears at least twice in each key frame of the second video data in the other key frames as the second projection image; configuring the pre-stored model that matches the first projection image as the first model, and configuring the angle of capturing the pre-stored model that matches the first projection image as the third angle; according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, adjusting the third angle in the direction from the first acquisition angle to the second acquisition angle to a fourth angle, judging whether the third projection image captured by the first model at the fourth angle matches the second projection image, and if so, scaling the pre-stored model to the third projection image to be consistent with the second projection image, and then scaling the pre-stored model to the first projection image to be consistent with the model image of the pre-stored model captured at the third angle; and configuring the pre-stored model at the same position of the first three-dimensional model as the second projection image, and outputting the second three-dimensional model.

[0108] The present invention divides the unit time into multiple intervals by the rate of change of adjacent frames; then determines in which interval the key frame of this unit time appears by the amplitude of the change in the number of object annotation boxes in adjacent intervals, thereby obtaining multiple relatively accurate key frames that meet the purpose of shooting; then the matching pre-stored model is configured in the first three-dimensional model after scaling, rotation, and position adjustment to output the second three-dimensional model, thereby directly adding a pre-stored model that does not need to be rendered to replace the facilities that need to be rendered more finely in the original first three-dimensional model. This undoubtedly increases the rendering speed, reduces the rendering and generation pressure, and even improves the rendering quality in a bird's-eye view. After all, many details cannot be captured by looking down, but we use an existing pre-stored model to configure it in a specified position, and even add additional upward-looking details and blind spot details to the second three-dimensional model.

[0109] In some embodiments, see Figure 1 , determining whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, deleting a key frame of at least one pre-stored model that does not conform, including

[0110] S310, pre-storing a preset configuration relationship, wherein the configuration relationship includes a relative position relationship between at least two pre-storing models;

[0111] S320, determine whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models; if not, count whether other groups of pre-stored models other than the group of at least two pre-stored models and including one of the group of at least two pre-stored models conform to the pre-stored relative position relationship, delete all key frames of the generated pre-stored model with the least number of conformities, and jump to S250.

[0112] The present invention pre-stores the relative position relationship between at least two pre-stored models as a criterion for judging whether the generated second three-dimensional model is reasonable, thereby judging the position relationship of the second three-dimensional model through the identified pre-stored models, for example, a person must be located on a building and cannot be located under a building, a bridge must have piers under the bridge, etc., to find abnormal position relationships, thereby determining whether this relationship group is suspicious, and then by comparing the correctness with other pre-stored relative relationships, pre-stored models that are different from most relationships are eliminated, thereby achieving automatic noise reduction of the third three-dimensional model, avoiding the generation of erroneous pre-stored models based on erroneous frames of video data caused by operations such as overexposure, inaccurate focus, and jello deformation of video data caused by bumps on the generated second three-dimensional model, and after finding the erroneous frames, all are deleted, and then jump to S240 for regeneration, that is, there is no such deleted key frame in the video data in step S240.

[0113] In some embodiments, see Figure 1 , determining whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models, and further comprising

[0114] If it matches, delete the object annotation box corresponding to the pre-stored model.

[0115] The present invention deletes the object annotation box of the pre-stored model that has been verified to be correctly generated, so that there is no need to consider the model comparison and generation of this position in the future, thereby reducing the computing power pressure and improving the efficiency of data collection and generation of digital twin models.

[0116] In some embodiments, see Figure 1 In step S240, a frame of an interval segment with more object annotation frames is selected as a key frame, and a frame of a first interval segment of the first variation group is selected as a key frame. The method of selecting a frame of a segment as a key frame is:

[0117] It is determined whether the key frame deleted in the above step S320 is located in this interval segment. If so, the interval segment is deleted and step S240 is executed again. If not, the middle frame of the frames arranged before and after the interval segment is used as the key frame.

[0118] In the present invention, since the above S240 may be a step that is repeatedly executed after the key frame has been deleted, each interval segment may have the problem of key frames being deleted, and each video segment may also have relatively long-term errors. To generate different video segments, the interval segment with the key frame having errors should be deleted, and step S240 should be executed again. The present invention uses the middle frame located in the middle of a segment as the key frame, and can select a point that is relatively farthest from the change point in the interval segment of video data with little change fluctuation as the key frame, thereby ensuring that the generated key frame has good clarity and avoiding the occurrence of error frames.

[0119] The embodiments described above are merely descriptions of preferred implementation modes of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A digital twin three-dimensional scene modeling and layout method, characterized in that: include Using a laser scanner to capture first laser data, and generating a first three-dimensional model according to the first laser data; Shooting the second video data with a camera, determining whether the facilities in the object annotation box appearing at least twice in the key frame of the second video data match the pre-stored model, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model; Determining whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, deleting a key frame of at least one pre-stored model that does not conform; Performing texture mapping on the second three-dimensional model through the second video data to generate a third three-dimensional model; Output the third 3D model.

2. A digital twin three-dimensional scene modeling and layout method according to claim 1, characterized in that: Using a camera to shoot second video data, determining whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match the pre-stored model, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model, including S210, using a camera to capture second video data; S220, configuring an object annotation frame for a closed image of each frame of the second video data; S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point; dividing each frame before and after the unit time into interval segments according to the interval point; wherein, if there is an interval point within the unit time, jump to S240; if there is no interval point within the unit time, merge the unit time with the unit time adjacent thereto, and re-execute S230; S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, if yes, configuring the two adjacent intervals as a first change group; judging whether the change in the number of annotation boxes of two adjacent intervals in the first change group exceeds a third threshold, if not, selecting a frame of the interval with more object annotation boxes as a key frame; if yes, selecting a frame of the first interval of the first change group as a key frame, or lowering the first threshold and jumping to S230; S250, configuring a projection surface screenshot of a facility in a same object annotation box that appears at least twice in each key frame of the second video data in a key frame where the facility first appears as a first projection image; The projection surface screenshots of the facilities in the same object annotation box that appear at least twice in each key frame of the second video data in the other key frames are configured as the second projection image; the pre-stored model that matches the first projection image is configured as the first model, and the angle of capturing the pre-stored model that matches the first projection image is configured as the third angle; according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, the third angle is adjusted to a fourth angle in the direction from the first acquisition angle to the second acquisition angle, and it is determined whether the third projection image captured by the first model according to the fourth angle matches the second projection image. If they match, the pre-stored model is scaled as follows: the pre-stored model is scaled according to the third projection image to a scaling ratio consistent with the second projection image, and then the pre-stored model is scaled as follows: the pre-stored model is scaled according to the first projection image to a scaling ratio consistent with the model image of the pre-stored model captured at the third angle; and the pre-stored model is configured at the same position of the first three-dimensional model as the second projection image, and the second three-dimensional model is output.

3. A digital twin three-dimensional scene modeling and layout method according to claim 2, characterized in that: Determine whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, delete a key frame of at least one pre-stored model that does not conform, including S310, pre-storing a preset configuration relationship, wherein the configuration relationship includes a relative position relationship between at least two pre-storing models; S320, determine whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models; if not, count whether other groups of pre-stored models other than the group of at least two pre-stored models and including one of the group of at least two pre-stored models conform to the pre-stored relative position relationship, delete all key frames of the generated pre-stored model with the least number of conformities, and jump to S250.

4. A digital twin three-dimensional scene modeling and layout method according to claim 3, characterized in that: Determining whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models, further comprising If it matches, delete the object annotation box corresponding to the pre-stored model.

5. A digital twin three-dimensional scene modeling and layout method according to claim 3, characterized in that: In the step of selecting a frame of the interval with more object annotation frames as a key frame in S240, and selecting a frame of the first interval of the first variation group as a key frame, the method of selecting the key frame is: It is determined whether the key frame deleted in the above step S320 is located in this interval segment. If so, the interval segment is deleted and step S240 is executed again. If not, the middle frame of the frames arranged before and after the interval segment is used as the key frame.

6. A digital twin three-dimensional scene modeling and layout system, characterized by: include A drone, which uses a laser scanner to shoot first laser data and generates a first three-dimensional model according to the first laser data; and uses a camera to shoot second video data; A processor is used to determine whether a facility in an object annotation box that appears at least twice in a key frame of the second video data matches a pre-stored model. If so, the pre-stored model is configured on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facility in the second video data to output the second three-dimensional model; determine whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship; if not, delete the key frame of at least one pre-stored model that does not conform; texture map the second three-dimensional model through the second video data to generate a third three-dimensional model, and output the third three-dimensional model.

7. A digital twin 3D scene modeling and layout system according to claim 6, characterized in that: Using a camera to shoot second video data, determining whether facilities in object annotation boxes that appear at least twice in key frames of the second video data match the pre-stored model, and if so, configuring the pre-stored model on the first three-dimensional model according to the scaling ratio and placement angle of the first three-dimensional model and the facilities in the second video data to output the second three-dimensional model, including S210, using a camera to capture second video data; S220, configuring an object annotation frame for a closed image of each frame of the second video data; S230, obtaining the adjacent change rate of each frame before and after the second video data within the unit time, and taking the position where the adjacent change rate exceeds the first threshold as the interval point; dividing each frame before and after the unit time into interval segments according to the interval point; wherein, if there is an interval point within the unit time, jump to S240; if there is no interval point within the unit time, merge the unit time with the unit time adjacent thereto, and re-execute S230; S240, judging whether there are adjacent intervals in the same unit time and the number of frames exceeds the second threshold, if yes, configuring the two adjacent intervals as a first change group; judging whether the change in the number of annotation boxes of two adjacent intervals in the first change group exceeds a third threshold, if not, selecting a frame of the interval with more object annotation boxes as a key frame; if yes, selecting a frame of the first interval of the first change group as a key frame, or lowering the first threshold and jumping to S230; S250, configuring a projection surface screenshot of a facility in a same object annotation box that appears at least twice in each key frame of the second video data in a key frame where the facility first appears as a first projection image; The projection surface screenshots of the facilities in the same object annotation box that appear at least twice in each key frame of the second video data in the other key frames are configured as the second projection image; the pre-stored model that matches the first projection image is configured as the first model, and the angle of capturing the pre-stored model that matches the first projection image is configured as the third angle; according to the first acquisition angle when the first video captures the first projection image and the second acquisition angle when the second projection image is captured, the third angle is adjusted to a fourth angle in the direction from the first acquisition angle to the second acquisition angle, and it is determined whether the third projection image captured by the first model according to the fourth angle matches the second projection image. If they match, the pre-stored model is scaled as follows: the pre-stored model is scaled according to the third projection image to a scaling ratio consistent with the second projection image, and then the pre-stored model is scaled as follows: the pre-stored model is scaled according to the first projection image to a scaling ratio consistent with the model image of the pre-stored model captured at the third angle; and the pre-stored model is configured at the same position of the first three-dimensional model as the second projection image, and the second three-dimensional model is output.

8. A digital twin 3D scene modeling and layout system according to claim 7, characterized in that: Determine whether at least two pre-stored models in the second three-dimensional model conform to a relative preset configuration relationship, and if not, delete a key frame of at least one pre-stored model that does not conform, including S310, pre-storing a preset configuration relationship, wherein the configuration relationship includes a relative position relationship between at least two pre-storing models; S320, determine whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models; if not, count whether other groups of pre-stored models other than the group of at least two pre-stored models and including one of the group of at least two pre-stored models conform to the pre-stored relative position relationship, delete all key frames of the generated pre-stored model with the least number of conformities, and jump to S250.

9. A digital twin three-dimensional scene modeling and layout system according to claim 8, characterized in that: Determining whether a group of at least two pre-stored models in the second three-dimensional model conforms to the relative position relationship between the at least two pre-stored models, further comprising If it matches, delete the object annotation box corresponding to the pre-stored model.

10. A digital twin three-dimensional scene modeling and layout system according to claim 9, characterized in that: In the step of selecting a frame of the interval with more object annotation frames as a key frame in S240, and selecting a frame of the first interval of the first variation group as a key frame, the method of selecting the key frame is: It is determined whether the key frame deleted in the above step S320 is located in this interval segment. If so, the interval segment is deleted and step S240 is executed again. If not, the middle frame of the frames arranged before and after the interval segment is used as the key frame.

Images