Three-dimensional scene modeling method, cloud system, storage medium and electronic device

Abstract

The embodiment of the invention provides a three-dimensional scene modeling method, a cloud system, a storage medium and an electronic device. The method comprises the following steps: based on a plurality of crowdsourcing data collected and uploaded by a user terminal, performing three-dimensional scene modeling on a to-be-modeled scene through an incremental generation neural radiation field model to obtain a full-field-angle three-dimensional rendering model of the to-be-modeled scene. Through the embodiment of the invention, the problems of difficulty in data acquisition, lack of view angle of a reconstruction main body and low rendering efficiency of the existing three-dimensional reconstruction and rendering method in the related technology can be solved, and the effect of improving the three-dimensional scene modeling efficiency is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a three-dimensional scene modeling method, a cloud system, a storage medium, and an electronic device. Background Art

[0002] Some existing 3D reconstruction and rendering methods have the following problems:

[0003] Difficulty in data acquisition: Existing 3D reconstruction methods require the use of professional equipment, such as laser sensors, depth cameras, or a camera group consisting of multiple RGBD cameras, where RGBD is three primary colors plus image depth; and professional operators are required to collect data to obtain high-precision 3D reconstruction source data. In this way, data acquisition is difficult and costly.

[0004] Poor reconstruction of dynamic scenes: When performing 3D reconstruction of outdoor buildings and street scenes, there are usually dynamic objects in the scene, such as pedestrians, animals, vehicles (such as cars, trucks, non-motor vehicles, etc.), and static objects associated with pedestrians (such as laptops, handbags, suitcases, etc. held by people). It takes a lot of time and computing resources to reconstruct dynamic scenes. At the same time, when dynamic objects block the modeling subject, the 3D reconstruction effect will be even less ideal.

[0005] Lack of perspective of the reconstructed subject: For the 3D modeling of the reconstructed subject, especially high-rise office buildings, cable-stayed bridges, etc., it is difficult to capture images from a bird's-eye view without the help of professional drone acquisition equipment. As a result, there is a lack of data source in the 3D reconstruction training stage, making it difficult to achieve 360-degree panoramic 3D reconstruction and rendering.

[0006] Low rendering efficiency: Most existing 3D rendering tasks are calculated on smart mobile devices. Considering the computing power limitations of mobile devices, they can only provide lower visual resolution and simplified rendering models, which limits the rendering effect and experience. Summary of the invention

[0007] The embodiments of the present invention provide a three-dimensional scene modeling method, a cloud system, a storage medium and an electronic device to at least solve the problems of data acquisition difficulty, lack of reconstruction subject perspective and low rendering efficiency existing in the existing three-dimensional reconstruction and rendering methods in the related art.

[0008] According to one embodiment of the present invention, a three-dimensional scene modeling method is provided, including: based on multiple crowdsourcing data collected and uploaded by a user terminal, three-dimensional scene modeling is performed on the scene to be modeled by incrementally generating a neural radiation field model, and a three-dimensional rendering model of the full field of view of the scene to be modeled is obtained.

[0009] According to another embodiment of the present invention, a cloud system is provided, including: a modeling module, which is used to perform three-dimensional scene modeling on a scene to be modeled by incrementally generating a neural radiation field model based on multiple crowdsourced data collected and uploaded by a user terminal, and obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

[0010] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0011] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0012] Through the above-mentioned embodiments of the present invention, the three-dimensional modeling of the scene is performed through the cloud system, which avoids the limitation of the three-dimensional reconstruction and rendering effect due to insufficient computing power of the mobile terminal, and the crowdsourcing data used by the cloud system can be collected and uploaded by the user terminal, which reduces the difficulty of data acquisition; at the same time, the three-dimensional radiation field distribution of the missing field of view can be generated by incrementally generating a neural radiation field model to obtain a three-dimensional rendering model of the full field of view. Therefore, the problems of data acquisition difficulties, lack of reconstruction subject perspective and low rendering efficiency in the existing three-dimensional reconstruction and rendering methods in the related art can be solved, so as to achieve the effect of improving the efficiency of three-dimensional scene modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a hardware structure block diagram of a computer terminal for running a three-dimensional scene modeling method according to an embodiment of the present invention;

[0014] Figure 2 is a network architecture diagram of a three-dimensional scene modeling system according to an embodiment of the present invention;

[0015] Figure 3 is a flow chart of a three-dimensional scene modeling method according to an embodiment of the present invention;

[0016] Figure 4 is a structural block diagram of a cloud system according to an embodiment of the present invention;

[0017] Figure 5 is a detailed network architecture diagram of a three-dimensional scene modeling system according to an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of a cloud-based distributed image data processing system according to an embodiment of the present invention;

[0019] Figure 7 is a schematic diagram of an end-to-end deep learning model according to an embodiment of the present invention;

[0020] Figure 8 is a flowchart of a 3D reconstruction training phase according to an embodiment of the present invention;

[0021] Fig. 9 is a flow chart of a three-dimensional rendering implementation stage according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0024] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal for running a three-dimensional scene modeling method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor or a programmable logic device) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 for communication functions and an input and output device 108. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0025] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the three-dimensional scene modeling method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0026] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0027] The above three-dimensional scene modeling method of the embodiment of the present invention can also be run on Figure 2 In the network architecture shown in Figure 2 As shown, the network architecture includes: a smart mobile terminal 21, a distributed computing node 22, a distributed storage node 23, and a cloud system 24, wherein the cloud system includes: a cloud distributed image data processing system 241, and a cloud 3D reconstruction system 242.

[0028] Specifically, the smart mobile terminal 21 is used to collect crowdsourcing data for modeling three-dimensional scenes and send the crowdsourcing data to distributed computing nodes; among them, crowdsourcing technology refers to a method of outsourcing work, tasks or projects to a large number of people (ie "crowdsourcing workers") through the Internet to complete them.

[0029] The distributed computing node 22 is used to process the received crowd-sourced data into crowd-sourced data that can be used in the cloud system.

[0030] The cloud system 24 is used to perform 3D reconstruction and 3D rendering based on the received crowdsourced data, obtain a 3D rendering model of the scene to be modeled, and send the 3D rendering model to the smart mobile terminal 21 for display.

[0031] The distributed storage node 23 is used to receive the crowdsourcing data generated by the cloud system 24 during the 3D reconstruction and 3D rendering process and needs to be saved.

[0032] In this embodiment, a three-dimensional scene modeling method running on the above-mentioned computer terminal or network architecture is provided. Figure 3 is a flow chart of a three-dimensional scene modeling method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0033] Step S302, based on the multiple crowdsourced data collected and uploaded by the user terminal, three-dimensional scene modeling is performed on the scene to be modeled by incrementally generating a neural radiation field model to obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

[0034] Among them, the user terminal is equivalent to Figure 2 In the smart mobile terminal, the first cloud system is functionally equivalent to the above-mentioned cloud distributed image data processing system 241, and the second cloud system is functionally equivalent to the above-mentioned cloud 3D reconstruction system 242.

[0035] In this embodiment, the crowdsourcing data includes the following fields: an image sequence field, a location information field, and a phase identification field; wherein the image sequence field is used to store an image sequence, for example: 10 RGB images continuously taken by a user terminal, and the image sequence field is represented by data_frames; the location information field is used to store location information, and the location information refers to the longitude and latitude information when the user terminal takes the image, and the images in the same image sequence share the location information, and the location information field is represented by data_location; the phase identification field is used to store a phase flag, and the phase flag is used to identify the phase used by the crowdsourcing data, including 0 and 1, for example: the phase identification bit of the crowdsourcing data used for the three-dimensional reconstruction phase is 0, or the phase identification bit used for the three-dimensional rendering implementation phase is 1; represented by data_phase.

[0036] In an exemplary embodiment, a piece of crowdsourced data includes: a sequence of 10 consecutive RGB images, corresponding longitude and latitude position information, and a data stage flag.

[0037] In step S302 of the present embodiment, three-dimensional scene modeling is performed on the scene to be modeled based on multiple crowd-sourced data collected and uploaded by the user terminal through the incremental generation of the neural radiation field model in the cloud system, including: the first cloud system performs image data processing on the multiple crowd-sourced data to obtain an optimized first image sequence set and relative pose information of each of the crowd-sourced data; wherein, one of the crowd-sourced data corresponds to a first image sequence in the first image sequence set; the first cloud system updates the information stored in the corresponding fields of the multiple crowd-sourced data based on the first image sequence, the relative pose information and the first position information corresponding to each of the crowd-sourced data, and inputs the updated multiple crowd-sourced data into the second cloud system; the second cloud system performs three-dimensional scene modeling by incrementally generating the neural radiation field model based on the updated multiple crowd-sourced data to obtain a three-dimensional rendering model with a full field of view of the scene to be modeled.

[0038] In one embodiment, the first cloud system divides the area corresponding to the scene to be modeled into multiple modeling sub-areas in advance according to a priori digital maps, and constructs a mapping relationship table between the modeling sub-areas and the distributed computing nodes, and sends the multiple crowdsourced data to the corresponding distributed computing nodes according to the first location information and based on the mapping relationship table; wherein one modeling sub-area corresponds to one first location information; then the first cloud system receives the second image sequence set returned by the distributed computing node, and inputs the second image sequence set into the end-to-end deep learning image processing model for image data processing, wherein each image sequence in the second image sequence set corresponds to one first location information.

[0039] In one embodiment, before the first cloud system performs image data processing on the multiple crowdsourced data uploaded by the user terminal, it will also add the following five fields to the crowdsourced data: data label field, data score field, key frame sequence number field, key frame feature field, and relative posture field; and perform the image data processing on the multiple crowdsourced data to which the fields have been added.

[0040] In one embodiment, the image data processing includes: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, extra-feature matching, loop detection and posture backend optimization.

[0041] In one embodiment, the first cloud system processes image data of multiple crowdsourced data uploaded by a user terminal to obtain an optimized first image sequence set, including: for any second image sequence in the second image sequence set, the first cloud system extracts masks of dynamic objects in each image in the second image sequence through the image semantic segmentation, and maps the masks back to the feature map corresponding to the image; wherein the dynamic objects include: pedestrians, animals, vehicles, and movable still objects related to the pedestrians; based on the contextual relationship of the mask in the feature map and the feature relationship between the previous and next frames of the second image sequence, the image inside the mask is background-completed through the image generation to obtain an optimized image sequence; through the above-mentioned steps of optimizing the image sequence, all the second image sequences in the second image sequence set are optimized to obtain the first image sequence set.

[0042] In an exemplary embodiment, before extracting the masks of dynamic objects in each image in the image sequence, the method further includes: scoring the image quality of the second image sequence through the image quality judgment to obtain a first score of the second image sequence, and updating the first score to the data score field of the crowdsourcing data corresponding to the second image sequence; and unifying the resolution and size of the scored images through the image denoising and the image super-resolution sharing weights.

[0043] In one embodiment, the first cloud system processes image data of a plurality of crowdsourced data uploaded by a user terminal to obtain relative pose information of each of the crowdsourced data, including: for any first image sequence in the first image sequence set, extracting descriptor information of each image in the first image sequence through the descriptor feature extraction and the intra-feature matching, and setting the image with the most common view descriptors as a key frame of the first image sequence; extracting fine-grained features of the key frame through the fine-grained feature extraction to obtain deep features, wherein the deep features include fine-grained representational features and fine-grained semantic features; performing feature splicing on the first position information corresponding to the first image sequence and the deep features through the extra-feature matching to obtain the key frame features of the first image sequence; performing feature matching, loop detection and graph optimization on the deep features in the key frame features of the first image sequence through the extra-feature matching, the loop detection and the back-end pose optimization, and obtaining the relative pose information of the crowdsourced data corresponding to the first image sequence based on the corresponding relationship between each of the first image sequences and the plurality of crowdsourced data; obtaining the relative pose information of other crowdsourced data through the above-mentioned step of obtaining relative pose information.

[0044] Step S304, the first cloud system updates the plurality of crowd-sourced data accordingly based on the first image sequence, the relative posture information, and the first position information corresponding to each of the crowd-sourced data, and inputs the updated plurality of crowd-sourced data into the second cloud system;

[0045] Before step S304 of the present embodiment, the method further includes: for any of the crowdsourced data, judging the stage of the crowdsourced data according to the stage identification field of the crowdsourced data; for the first crowdsourced data in the three-dimensional reconstruction training stage, storing the first crowdsourced data having a first score greater than or equal to a first preset threshold in a distributed database; for the second crowdsourced data in the three-dimensional rendering implementation stage, performing similarity matching between the key frame features of the second crowdsourced data and the corresponding key frame features stored in the distributed database, and storing the second crowdsourced data having a similarity less than a second preset threshold in the distributed database.

[0046] Step S306: The second cloud system performs three-dimensional scene modeling by incrementally generating a neural radiation field model based on the updated plurality of crowd-sourced data, and obtains a three-dimensional rendering model of the full field of view of the scene to be modeled.

[0047] Before step S306 of the present embodiment, the method further includes: the second cloud system obtains a plurality of the first crowd-sourced data from a distributed database, and sorts the plurality of the first crowd-sourced data in descending order based on the size of the first scores of the first crowd-sourced data; inputs the first N first crowd-sourced data, the relative posture information of the first N first crowd-sourced data, and the first position information corresponding to the first N first crowd-sourced data into the pre-designed incremental generation of neural radiation field model to obtain the trained incremental generation of neural radiation field model, wherein N is an integer greater than zero; adjusts the weights and biases of the neural network in the trained incremental generation of neural radiation field model to converge the incremental generation of neural radiation field model; when the incremental generation of neural radiation field model completes convergence, sorts the plurality of the first crowd-sourced data in descending order based on the size of the weighted scores of the first N first crowd-sourced data, and selects the first M first crowd-sourced data to perform fine-tuning training on the incremental generation of neural radiation field model, wherein M is an integer greater than zero and less than N.

[0048] In an exemplary embodiment, before sorting the plurality of first crowd-sourced data in descending order based on the sizes of the final scores of the first N first crowd-sourced data, the method further includes: scoring the first N first crowd-sourced data to obtain second scores of the first N first crowd-sourced data; and performing weighted calculation on the first scores and the second scores of the first N first crowd-sourced data to obtain weighted scores of the N first crowd-sourced data.

[0049] Through the above steps, the three-dimensional reconstruction and rendering of the scene are performed through the cloud system, which avoids the limitation of the three-dimensional reconstruction and rendering effect due to insufficient computing power of the mobile terminal. The crowdsourced data used by the cloud system can be collected and uploaded by the user terminal, which reduces the difficulty of data acquisition; at the same time, the three-dimensional radiation field distribution of the missing field of view can be generated by incrementally generating a neural radiation field model to obtain a three-dimensional rendering model of the full field of view. Therefore, the problems of data acquisition difficulties, lack of reconstruction subject perspective, and low rendering efficiency in the existing three-dimensional reconstruction and rendering methods in the relevant technology can be solved, and the efficiency of three-dimensional scene modeling can be improved.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0051] In the present embodiment, a three-dimensional scene modeling system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0052] Figure 4 is a structural block diagram of a cloud system according to an embodiment of the present invention. Figure 4 As shown, the cloud system includes:

[0053] The modeling module 10 is used to perform three-dimensional scene modeling on the scene to be modeled by incrementally generating a neural radiation field model based on multiple crowdsourced data collected and uploaded by the user terminal, and obtain a three-dimensional rendering model of the full field of view of the scene to be modeled. It should be noted that the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned modules are all located in the same processor; or the above-mentioned modules are located in different processors in any combination.

[0054] To facilitate the understanding of the technical solution provided by the present invention, a detailed description will be given below in conjunction with embodiments of specific scenarios.

[0055] Figure 5 is a detailed network architecture diagram of a three-dimensional scene modeling system according to an embodiment of the present invention, such as Figure 5 As shown, the system includes: a smart mobile terminal 21, a distributed computing node 22, a distributed storage node 23, a cloud system 24, a cloud distributed image data processing system 241, and a cloud three-dimensional reconstruction system 242. The cloud distributed image data processing system 241 also includes a data distribution module 61, an image processing module 62 and a data storage module 63. The cloud three-dimensional reconstruction system 242 includes incremental generation of neural radiation fields.

[0056] Among them, the cloud-based distributed image data processing system 241 and the cloud-based three-dimensional reconstruction system 242 of the cloud-based system 24 both include an end-to-end deep learning model, so the cloud-based system 24 has powerful computing resource support.

[0057] The cloud-based distributed image processing system is a cloud-based distributed image processing system for crowdsourced data. It is deployed in the cloud and combines end-to-end deep learning models, distributed computing technology, and distributed storage technology. It is a distributed image processing system.

[0058] In this embodiment, the smart mobile terminal is used to collect data and upload crowdsourced data, including:

[0059] The RGB camera equipped with the smart mobile terminal is used as the data acquisition medium, and data is collected in a crowdsourcing manner. Three types of information are uploaded based on the collected data: image sequence, longitude and latitude position information at the time of shooting (obtained through the smart mobile terminal), and data stage flag. These three types of information together constitute a crowdsourcing data.

[0060] Here, an image sequence refers to a plurality of RGB images taken continuously, which is represented by the field data_frames; for example, a set of image sequences is 10 RGB images taken continuously.

[0061] The longitude and latitude information at the time of shooting is shared by this group of image sequences. For example, 10 RGB images share the same location information, which is represented by the field data_location;

[0062] The data phase flag is used to indicate the specific phase of the data effect, and is represented by the field data_phase, where data_phase is 0 for the 3D reconstruction training phase, and data_phase is 1 for the 3D rendering implementation phase.

[0063] Specifically, the specific stage in which the crowdsourced data is used is distinguished according to the stage flag of the uploaded crowdsourced data, for example: data_phase 0 indicates the 3D reconstruction training stage, and data_phase 1 indicates the 3D rendering implementation stage. In the 3D reconstruction training stage, the crowdsourced data is uploaded to the cloud by the smart mobile terminal as the source data for 3D reconstruction. After successful upload, the life cycle of the smart mobile terminal data collection task ends; in the 3D rendering implementation stage, the crowdsourced data is uploaded to the cloud by the smart mobile terminal as the query data for 3D rendering, and waits for system matching and query. If there is a corresponding 3D reconstruction model, it will be sent from the cloud to the smart mobile terminal and displayed on the smart mobile terminal.

[0064] After the smart mobile terminal collects the data, it uploads a sequence of 10 continuous RGB images, the corresponding longitude and latitude information at the time of shooting, and the data stage flag as a crowdsourcing data to the cloud system, namely the cloud-based distributed image data processing system and the cloud-based 3D reconstruction system.

[0065] Figure 6 is a schematic diagram of a cloud-based distributed image data processing system according to an embodiment of the present invention. Figure 6 As shown, before the cloud system 24 is put into operation, the cloud distributed image data processing system is provided with a preprocessing process, that is, the cloud distributed image data processing system will pre-select a suitable modeling area division method according to the digital map of the prior data as a template according to the actual application situation, divide the entire modeling area into multiple modeling sub-areas, and construct a mapping table between the modeling sub-areas and the distributed computing nodes, wherein a modeling sub-area corresponds to an independent distributed computing node one-to-one, and the mapping table needs to be used in both the 3D reconstruction training stage and the 3D rendering implementation stage.

[0066] In one embodiment, there are two ways to divide the modeling area:

[0067] 1) Based on the prior digital map, the entire modeling area is divided into multiple modeling sub-areas with uniform areas using streets as dividing lines;

[0068] 2) Based on the prior digital map, the modeling area is divided into multiple sub-areas with uneven areas, taking the landmark building as the center of the modeling sub-area.

[0069] The cloud-based distributed image data processing system is also used to add new fields to the uploaded crowd-sourced data after receiving it, find the modeling sub-area to which the longitude and latitude information of the crowd-sourced data belongs, and then send the crowd-sourced data to the corresponding distributed computing node based on the previously constructed mapping table.

[0070] Among them, the five fields that the cloud-based distributed image data processing system supplements to the crowdsourcing data are: data_id, data_score, data_keyframe_id, data_keyframe_feature, and data_frame_pose.

[0071] 1) data_id field, which is used to encapsulate the unique code of crowdsourcing data. The data_id of each crowdsourcing data is different. The cloud-based distributed image data processing system generates a unique code based on the upload time, geographic location information, and device model information of the crowdsourcing data, and updates it to the data_id field.

[0072] 2) The data_score field is used to encapsulate the image sequence score of the crowdsourced data. The score range of the image sequence of the crowdsourced data is [0, 100].

[0073] The image sequence score of crowdsourced data is a floating point number used for image sequence evaluation and screening. The data_score is initially scored by the cloud-based distributed image data processing system. Subsequently, the data_score is re-weighted and calculated by the incremental radiation field model system to obtain the final score. The calculation of the data_score in the incremental radiation field model system is calculated by the following formula (1):

[0074] data_score=a*quality_score+b*nerf_score formula (1)

[0075] Among them, quality_score is calculated by the image processing module 62 of the cloud-based distributed image data processing system, and its initial value is 0; nerf_score is calculated by the incremental generation of neural radiation field model of the incremental generation of radiation field model system, and its initial value is 0; a and b are weight hyperparameters, in the cloud-based distributed image data processing system, a=0.9, b=0; in the incremental generation of neural radiation field model of the incremental generation of radiation field model system, a=0.4, b=0.6.

[0076] 3) The data_keyframe_id field is used to encapsulate the crowdsourcing data key frame number. The crowdsourcing data key frame number is represented by an integer in [0, 9], which means the most representative image number among the 10 RGB images in an image sequence. The corresponding image can be found in data_frames through data_keyframe_id.

[0077] 4) The data_keyframe_feature field is used to encapsulate the keyframe deep learning features, which represent the deep learning features of the keyframe image. The unified dimension is 130 and is calculated by the cloud-based distributed image data processing system.

[0078] 5) data_frame_pose field, used to encapsulate the relative pose information of the crowdsourced data. The relative pose information of the crowdsourced data is represented by the displacement parameter T and the rotation parameter Q. Where T = (x t ,y t ,z t ), represents a three-dimensional translation vector, Q = (w q ,x q ,y q ,z q ) represents a rotation vector in the form of a quaternion. T and Q can represent the relationship between the relative displacement and rotation direction of each image in an image sequence. Each image in the image sequence has a set of T and Q values.

[0079] The relative pose corresponding to an image sequence refers to the relative position and orientation relationship between two images, which is usually composed of a rotation vector and a translation vector, which describes the transformation from one camera coordinate system to another. In the fields of computer vision and robotics, it is often necessary to determine the relative pose between a set of images in order to align, register, or establish a spatial relationship between the images; and in the 3D reconstruction training stage, it is necessary to determine the relative pose between two cameras in order to calculate the corresponding points of the points in the scene in the fields of view of the two cameras.

[0080] Estimating the relative pose corresponding to an image is an important problem in the field of computer vision and robotics, and there are many methods to achieve it, including feature matching, structured light, deep learning, etc. In each embodiment of the present invention, relative pose based on feature matching of deep learning is used.

[0081] After the fields are supplemented, the fields included in the crowdsourcing data format are shown in Table 1, that is, a complete crowdsourcing data has 8 attributes, which are represented by 8 fields.

[0082] Table 1 Crowdsourcing data format

[0083] Field Name Field meaning Initialization phase data_frames 10 RGB images Smart mobile terminal data_location Longitude and latitude information Smart mobile terminal data_phase Data phase flag Smart mobile terminal data_id Crowdsourcing data unique code Cloud data_score Crowdsourced data evaluation score Cloud data_keyframe_id Key frame number Cloud data_keyframe_feature Keyframe Deep Learning Features Cloud data_frame_pose Relative poses of 10 RGB images Cloud

[0084] The distribution module 61 is used to supplement 5 fields to the uploaded crowd-sourcing data, and then divide the uploaded crowd-sourcing data into corresponding modeling areas based on the longitude and latitude position information in the crowd-sourcing data, and then send the crowd-sourcing data to the corresponding distributed computing node 22 through the mapping relationship table between the modeling area and the distributed computing node.

[0085] The distributed computing node 22 is used to receive a plurality of crowd-sourced data in the same area but at different specific locations, and extract the image sequence corresponding to the data_frames field of the crowd-sourced data and the longitude and latitude position information represented by data_location, and input the image sequence set composed of the plurality of crowd-sourced data image sequences and the corresponding position information into the end-to-end deep learning image processing model of the image processing module 62;

[0086] The image processing module 62 is mainly composed of an end-to-end deep learning image processing model. All processing procedures of the image processing module 62 are executed on the GPU. The end-to-end deep learning image processing model has one CPU-GPU I / O when inputting data and one GPU / CPU I / O when outputting data. There are two I / O operations in the whole process. Other image data processing is performed on the GPU throughout the whole process. Therefore, the image processing efficiency is extremely high. The schematic diagram of the end-to-end deep learning model architecture is shown in FIG. Figure 7 shown.

[0087] The end-to-end deep learning image processing model in the image processing module 62 takes image sequence and longitude and latitude position information as input, and takes optimized image sequence, image sequence score, key frame number, key frame deep learning features and relative pose of image sequence as output. Based on the powerful feature extraction capability of deep convolutional neural network and the nonlinear mapping capability brought by activation function, combined with the downsampling, upsampling and feature fusion methods of convolutional neural network, it can realize the image data processing capabilities that can only be achieved by traditional multi-model series connection: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, extra-feature matching, loop detection and pose back-end optimization.

[0088] Specifically, in the image quality judgment process, the image quality judgment component in the cloud distributed image data processing system 241 includes a softmax component, which uses the feature extraction and feature abstraction capabilities of the convolutional neural network to judge the quality of the input image, and scores the image sequence, recording the score as quality_score; and updating the quality_score to the attribute data_score field of the crowdsourcing data;

[0089] For example, the input image sequence is scored to obtain quality_score. The higher the quality_score, the higher the quality of the original image and the greater the contribution to the end-to-end deep learning model. On the contrary, if the quality_score is lower than the threshold (for example, 0.3), during model training, during the forward reasoning of the end-to-end deep learning model, the convolutional neural network feature weight corresponding to the quality_score is reset to 0, that is, it does not participate in the subsequent feature extraction and function implementation, thereby improving the forward reasoning efficiency of the end-to-end deep learning model, that is, the model training efficiency.

[0090] As a normalized output component, softmax is a commonly used indicator in multi-classification problems, which is used to represent the probability distribution of a sample belonging to each category. In deep learning, softmax is usually used as the activation function of the output layer, which can map the output of the network to a probability distribution. In the embodiment of the present invention, the softmax output value is a floating point number with a value range of [0,100]. The higher the softmax score, the higher the image quality, and the greater the contribution to the subsequent cloud-based 3D reconstruction system.

[0091] In this embodiment, unlike a conventional deep learning image processing system, the image processing module 62 is an end-to-end deep learning model. Compared with a conventional pipeline architecture deep learning model, it can significantly reduce the CPU-GPU exchange operation of data and greatly reduce the data I / O throughput of cloud computing nodes, especially when processing massive crowdsourcing data, which will greatly improve the image data processing efficiency. Through the cloud distributed computing nodes, optimized image sequences, image sequence scores, key frame numbers, key frame deep learning features, and relative pose information of images in the image sequence can be obtained.

[0092] By using an end-to-end deep learning algorithm for image data processing, the accuracy of reconstruction and rendering is effectively supported: In this embodiment, an end-to-end deep learning algorithm is used. Except for the two I / O operations of input / output, the entire process is calculated on the GPU, realizing the functions of image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, feature internal matching, fine-grained feature extraction, feature external matching, loop detection and posture backend optimization. Compared with traditional computer vision algorithms, it has higher accuracy and robustness, can handle more complex scenes and data, and improves the accuracy of reconstruction and rendering; compared with the pipeline processing of deep learning models, it greatly reduces I / O operations and improves the real-time performance of reconstruction and rendering.

[0093] The data storage module 63 is used to store the image sequence (crowdsourcing data) whose score of the data_score field is greater than or equal to 0.3 to the distributed storage node 23 after calculating the image quality score quality_score at the softmax layer and updating the data_score field of the crowdsourcing data. For the image sequence (crowdsourcing data) whose score is less than 0.3, it is not stored, and the corresponding quality_score score value is updated to the data_score field.

[0094] In one embodiment, image denoising and image super-resolution share weights for unified processing of crowdsourced data. Since the devices for crowdsourcing uploads are not the same, the resolution and size of the uploaded images are also inconsistent, and the data needs to be processed uniformly, and the image sequences are unified to the same resolution and size through image denoising and image super-resolution.

[0095] In one embodiment, image semantic segmentation and image generation are two independent sub-branches, which share the previous feature extraction and have a strict sequential relationship. That is, the image semantic segmentation component is first used to extract the mask of the dynamic object in the image, and then the mask is mapped back to the feature map corresponding to the image. The image generation component uses the contextual relationship of the mask in the feature map and the feature relationship between the previous and next frames of the image sequence to regenerate the image inside the mask and complete it with background information unrelated to the dynamic object.

[0096] The mask is the outline of the dynamic object, the area within the outline is the main body of the dynamic object, and the area outside the outline is the background that is irrelevant to the dynamic object.

[0097] In this implementation, dynamic objects can be divided into four categories: (1) pedestrians, i.e., pedestrians sitting, standing, or moving in the image; (2) animals, i.e., cats, dogs, birds, etc. in the image; (3) vehicles, i.e., cars, trucks, buses, non-motor vehicles, etc.; (4) pedestrian-related movable static objects, such as laptops, handbags, suitcases, and long-handled umbrellas held by people. The image semantic segmentation component will perform pixel-level classification on these four types of dynamic objects and obtain corresponding masks.

[0098] There are two conditions for determining the dynamic object of type (4), and one of them is sufficient:

[0099] a. The pedestrian and the movable static object share the same mask edge. At this time, the pedestrian and the movable static object happen to be in contact, that is, the pedestrian mask and the movable static object mask share part of the same mask edge;

[0100] b. The intersection over union (IoU) of the pedestrian segmentation area and the movable static object segmentation area is greater than 0.1. As shown in formula (2) and formula (3), there is an interaction relationship between the pedestrian and the movable static object. That is, the ratio of the intersection area and the union area of ​​the pedestrian mask and the movable static object mask is greater than 0.1.

[0101] IoU mask =occur_maskunion_mask formula (2)

[0102] IoU mask >0.1 formula (3)

[0103] Among them, occur_mask represents the intersection area of ​​the masks of pedestrians and movable static objects, and union_mask represents the union area of ​​the masks of pedestrians and movable static objects.

[0104] The end-to-end deep learning image processing model, based on image semantic segmentation and image generation components, will filter dynamic objects in the output image and complete the clean background. The image semantic segmentation and image generation components jointly implement the dynamic object filtering and background completion functions of the image. The mask is the outline of the dynamic object, the main body of the dynamic object is inside the outline, and the background outside the outline is unrelated to the dynamic object; thus, the optimized image of the image sequence set is obtained, and the optimized image is copied to the data_frames field, overwriting the original uploaded image sequence.

[0105] In one embodiment, descriptor feature extraction and intra-feature matching are used to extract the descriptor information of the image, and the corresponding image with the most common view descriptors is matched and set as the key frame of the image sequence, and the key frame id of the corresponding image sequence is updated, that is, the data_keyframe_id field is updated.

[0106] In one embodiment, the fine-grained feature extraction is to further extract fine-grained features from key frame features, which are composed of 64-dimensional fine-grained representation features and 64-dimensional fine-grained semantic features, for a total of 128-dimensional deep features.

[0107] In one embodiment, when matching outside of features, key frame feature splicing is required, and the feature composition is: 128-dimensional fine-grained features, and 2-dimensional location information (longitude and latitude information), and finally a 130-dimensional key frame feature is obtained, and the corresponding key frame deep learning feature attributes are updated, that is, the data_keyframe_feature field is updated.

[0108] In one embodiment, feature external matching, loop detection and back-end pose optimization perform feature matching, loop detection and graph optimization on fine-grained features of key frames selected from different image sequences to obtain pose information of the image sequence and record it in the data_frame_pose field.

[0109] Different from the conventional deep learning image processing system, the image processing module 62 of this embodiment is an end-to-end architecture deep learning model. Compared with the conventional pipeline architecture deep learning model, it can significantly reduce the CPU-GPU exchange operation of data and greatly reduce the data I / O throughput of cloud computing nodes, especially when processing massive crowdsourcing data, which will greatly improve the image data processing efficiency. Through the cloud distributed computing nodes, it is possible to obtain optimized image sequences, image sequence scores, key frame numbers, key frame deep learning features, and relative pose information of images in the image sequence.

[0110] The data storage module 63 is further used to determine the phase of the crowd-sourced data by firstly judging the phase of the crowd-sourced data through the data_phase field in the crowd-sourced data when storing the crowd-sourced data;

[0111] When the crowdsourced data is in the 3D reconstruction training stage, the quality_score score of the image sequence after passing through the image processing module is obtained through the data_score field in the crowdsourced data, wherein crowdsourced data with a quality_score score lower than 0.3 is not stored, and crowdsourced data with a quality_score score not less than 0.3 is stored in the distributed database.

[0112] When the crowdsourcing data is in the three-dimensional rendering implementation stage, the 130-dimensional key frame features recorded in the data_keyframe_feature field of the uploaded crowdsourcing data are matched with the data_keyframe_feature field features of the data stored in the distributed database (i.e., the distributed computing storage node 23) for similarity. Here, L2 similarity, i.e., Euclidean distance, is used for measurement. If the similarity is greater than 0.6, this crowdsourcing data is not stored, otherwise it is stored.

[0113] In this embodiment, the data storage module 63 is also used to update the quality_score value to the data_score field accordingly.

[0114] In addition, for stored data, the image sequence optimized by the image processing module 62 will be copied to the data_frames field, covering the original unoptimized image sequence image, and the data_id and data_location fields remain unchanged; the key frame sequence number is recorded in the data_keyframe_id field; the key frame deep learning features are recorded in the data_keyframe_feature field; and the relative pose information of the image sequence is recorded in the data_frame_pose field.

[0115] In the cloud-based three-dimensional reconstruction system, the optimized image sequence, posture information, and longitude and latitude position information collected and uploaded by the cloud-based distributed image data processing system 241 are used to train the incremental generation of the neural radiation field model. The incremental generation of the neural radiation field model is also an end-to-end deep learning model that can predict the radiation field distribution at the camera location based on the calculated posture information and modeling data, and can generate the three-dimensional radiation field distribution of the missing field of view. The model shares the same feature extraction network, and is accompanied by three branches: depth estimation, neural radiation field prediction, and posture regression. The deep learning features of the three branches are then spliced ​​and input into the multi-field of view radiation field generation module, and the neural radiation field of the full field of view is generated in combination with the position information.

[0116] In the cloud-based 3D reconstruction system, the architecture of the end-to-end deep learning model includes the following five components:

[0117] 1) Backbone network, used for basic feature extraction;

[0118] 2) Neural radiance field component, which fits the basic neural radiance field based on deep learning features extracted from the backbone network;

[0119] 3) Depth estimation component, which estimates the distance of the camera medium from the reconstructed subject from the image based on the deep learning features extracted by the backbone network;

[0120] 4) The pose regression component regresses the pose of feature points in the image based on the deep learning features extracted by the backbone network and the pose data of the image sequence output by the cloud-based distributed image data processing system;

[0121] 5) Multi-viewing angle neural radiation field generation: based on generative neural networks and combining the outputs of neural radiation field components, depth estimation components, and pose regression components, a three-dimensional neural radiation field with missing viewing angles is generated.

[0122] The cloud-based three-dimensional reconstruction system of the embodiment of the present invention incrementally generates neural radiation fields. Different from the traditional neural radiation field method, the incremental generation of neural radiation fields in the embodiment of the present invention not only has the function of three-dimensional reconstruction of the neural radiation field model, but can also generate field of view angles that cannot be captured by training data, so that the three-dimensional reconstructed model can be displayed with a full 360-degree field of view.

[0123] Then, the score quality_score in the cloud-based distributed image data processing system is combined for weighted calculation to update the weighted score data_score of the crowdsourcing data.

[0124] The incremental training implemented by the incremental generation of neural radiation field model is to sort the data_score field scores from large to small in the distributed database, select the 100 crowdsourced data with the highest scores to train the incremental generation of neural radiation field, and after the training, use the softmax component to score the weights of the 100 training data in the model to obtain the nerf_score, which is then combined with the score quality_score in the cloud-based distributed image data processing system for weighted calculation to update the weighted score data_score of the crowdsourced data.

[0125] The embodiment of the present invention also provides a three-dimensional scene modeling method, which performs position division, image data processing, feature extraction and matching on the data uploaded by crowdsourcing based on distributed computing and deep learning methods, thereby realizing three-dimensional reconstruction of the modeling object. The method combines crowdsourcing technology, cloud computing technology, computer vision and deep learning technologies, has the advantages of high efficiency, real-time and low cost, and can be widely used in industry, urban landscape display, transportation planning, cultural heritage protection, entertainment and other fields.

[0126] In the embodiment of the present invention, the cloud-based distributed image data processing system 241 is a pre-process of the cloud-based 3D reconstruction system 242 , and the cloud-based distributed image data processing system 241 provides training data guarantee for the cloud-based 3D reconstruction system 242 .

[0127] In this embodiment, the full parameterization of the 3D reconstruction training phase and the sparse parameterization of the 3D rendering implementation phase are achieved by the re-parameterization technology. The incremental generation radiation field model system has efficient and high-quality image data processing capabilities and can completely and accurately complete the reconstruction and real-time rendering tasks of large-scale 3D scenes.

[0128] In one embodiment, the three-dimensional scene modeling method is implemented in two stages: the three-dimensional reconstruction training stage and the three-dimensional rendering implementation stage. The three-dimensional reconstruction training stage is in the first stage and the three-dimensional rendering implementation stage is in the latter stage. The three-dimensional modeling of the modeling subject needs to be completed in the three-dimensional reconstruction training stage before the rendering object can be viewed in the three-dimensional rendering implementation stage. In the three-dimensional reconstruction training stage, the smart mobile terminal is responsible for data collection and crowdsourcing upload, and the cloud is responsible for data area division, distributed computing node distribution, image sequence optimization, and incremental generation of neural radiation field model training; in the three-dimensional rendering implementation stage, the smart mobile terminal is responsible for data collection, upload, and receiving and displaying the rendering model from the distributed cloud. The cloud is responsible for querying, feature matching, posture calculation, three-dimensional rendering, and rendering model distribution of the data uploaded by the smart mobile terminal.

[0129] The smart mobile terminal data acquisition, cloud-based distributed image data processing system, and cloud-based 3D reconstruction system in the present invention follow a sequential relationship in the entire process. First, enter the smart mobile terminal data acquisition process and upload the collected crowdsourced data to the cloud; then the cloud-based distributed image data processing system processes the original data uploaded by the crowdsourced data to obtain optimized image sequences and posture information; finally, through the cloud-based 3D reconstruction system, in the 3D reconstruction training phase, the training incrementally generates the neural radiation field model, and in the 3D rendering implementation phase, generates the corresponding rendering model and completes the distribution to the smart mobile terminal.

[0130] 1) 3D reconstruction training stage.

[0131] During the 3D reconstruction training phase, the smart mobile terminal is responsible for data collection and crowdsourcing data uploading, while the cloud is responsible for data area division, distributed computing node distribution, image sequence optimization, and incremental generation of neural radiation field model training.

[0132] Figure 8 is a flow chart of a 3D reconstruction training phase according to an embodiment of the present invention. Figure 8 As shown, the process includes the following steps:

[0133] Step S801, the smart mobile terminal uploads the collected crowdsourced data to the cloud distributed image data processing system, wherein each piece of crowdsourced data includes an image sequence of multiple continuous RGB images, corresponding longitude and latitude position information, and an identification bit of the crowdsourced data.

[0134] In an exemplary embodiment, one or more smart mobile terminals collect and upload crowdsourced data, and each smart mobile terminal uploads one or more pieces of crowdsourced data.

[0135] Step S802 , determining the data_phase field of the uploaded crowd-sourced data to determine which phase the crowd-sourced data is in, for example: data_phase 0 indicates the 3D reconstruction training phase, and data_phase 1 indicates the 3D rendering implementation phase.

[0136] Step S803, adding the following five fields to the uploaded crowdsourcing data: data_id, data_score, data_keyframe_id, data_keyframe_feature, data_frame_pose.

[0137] Step S804, extracting the data_location field of the crowdsourcing data, referring to the mapping table between the modeling sub-areas and the distributed computing nodes, and sending the crowdsourcing data to the corresponding distributed computing nodes.

[0138] Step S805: The end-to-end deep learning graphics processing model receives the image sequence set and corresponding position information sent by the distributed computing node, performs image data processing on the image sequence set, and obtains an optimized image sequence set and relative pose information corresponding to the optimized image sequence set.

[0139] Among them, the image data processing flow includes: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, external-feature matching, loop detection and posture backend optimization.

[0140] Step S806, when the crowdsourced data belongs to the 3D reconstruction training stage, the score stored in the data_score field of the crowdsourced data is determined, and the crowdsourced data with a score not less than a preset threshold is stored in the distributed database.

[0141] The score in step S806 is equivalent to the first score in the above-mentioned embodiment.

[0142] During the 3D reconstruction training phase, incremental iterative training is performed based on the data_score of the training data. As more and more high-quality crowdsourcing data is available, the model becomes more accurate and complete. The end-to-end model uses the full weight parameter mode for model fitting.

[0143] In an exemplary embodiment, the preset threshold is set to 0.3.

[0144] Step S807, designing an incremental generation neural radiation field model.

[0145] The incrementally generated neural radiation field model is a model that can predict the radiation field distribution at the camera's location with a full field of view.

[0146] Step S808, using the image sequence set optimized in step S805 as a training set, the image data obtained by the cloud-based distributed image data processing system, the relative posture information, and the longitude and latitude position information collected and uploaded by the smart mobile terminal are input into the designed incremental generation of the neural radiation field model, after the deep learning features of the three branches of neural radiation field distribution, depth estimation and posture regression in the designed incremental generation of the neural radiation field model are spliced, and then the neural radiation field model is output through the multi-viewing angle neural radiation field generation network.

[0147] Step S809 continuously adjusts the weights and biases of the neural network through a back-propagation algorithm to minimize the error between the generated neural radiation field and the actual radiation field.

[0148] Step S810, after the model converges, the score information will be multiplied with the score of the image sequence set when judging the image quality after softmax according to the parameter weight information to obtain the final score information;

[0149] Step S811, during incremental training, the image sequence set will be sorted according to the score information, and only the top 10 image sequences will be selected to participate in fine-tuning training.

[0150] Among them, step S804 corresponds to the data distribution module 61 in the above embodiment, step S805 corresponds to the image processing module 62 in the above embodiment, and step S806 corresponds to the data storage module 63 in the above embodiment.

[0151] During the 3D reconstruction training phase, the end-to-end deep learning model system applies the full feature weights of the deep neural network for model forward reasoning and gradient back propagation, ensuring that the trained 3D model has accurate architecture, clear texture, and natural lighting.

[0152] 2) Three-dimensional rendering implementation stage.

[0153] In the 3D rendering implementation stage, the smart mobile terminal is responsible for data collection, crowdsourcing data uploading, and receiving and displaying rendering models from the distributed cloud. The cloud is responsible for querying, feature matching, pose calculation, 3D rendering, and sending rendering models for the crowdsourcing data uploaded by the smart mobile terminal.

[0154] Fig. 9 is a flow chart of a three-dimensional rendering implementation stage according to an embodiment of the present invention. Fig. 9 As shown, the process includes the following steps:

[0155] Step S901, the smart mobile terminal uploads the collected crowdsourced data to the cloud distributed image data processing system, wherein each piece of crowdsourced data includes an image sequence of multiple continuous RGB images, corresponding longitude and latitude position information, and an identification bit of the crowdsourced data.

[0156] Step S902 , determine the data_phase field of the uploaded crowd-sourced data to determine which stage the crowd-sourced data is in. For example, data_phase 0 indicates the 3D reconstruction training stage, and data_phase 1 indicates the 3D rendering implementation stage.

[0157] Step S903, add the following five fields to the uploaded crowdsourcing data: data_id, data_score, data_keyframe_id, data_keyframe_feature, data_frame_pose.

[0158] Step S904, extracting the data_location field of the crowd-sourced data, referring to the mapping table between the modeling sub-areas and the distributed computing nodes, and sending the crowd-sourced data to the corresponding distributed computing nodes.

[0159] Step S905: The end-to-end deep learning graphics processing model receives the image sequence set and corresponding position information sent by the distributed computing node, performs image data processing on the image sequence set, and obtains an optimized image sequence set and relative pose information corresponding to the optimized image sequence set.

[0160] Among them, the image data processing flow includes: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, external-feature matching, loop detection and posture backend optimization.

[0161] In the 3D rendering implementation stage, the smart mobile terminal captures the RGB image and longitude and latitude information of the desired rendered object and uploads it to the cloud, requesting to send the 3D rendering model. The cloud-based distributed image data processing system extracts features from the image to obtain the 130-dimensional feature vector data_keyframe_feature and relative pose data_frame_pose of the key frame in the image sequence. The corresponding incremental generation neural radiation field model is then called for forward reasoning, and the rendering model of the full field of view is output and sent to the smart terminal, enabling real-time rendering of the modeled object by the smart mobile device. The specific steps include:

[0162] Step S906, when the crowdsourcing data belongs to the 3D rendering implementation stage, the 130-dimensional key frame features of the data_keyframe_feature record of the crowdsourcing data uploaded this time are matched with the data_keyframe_feature field features of the data stored in the distributed database, and the crowdsourcing data with a similarity greater than a preset threshold is stored.

[0163] In one embodiment, if the similarity between all key frames and the uploaded RGB image is below 0.6, it means that no suitable 3D rendering model is found, and the user can be guided to perform crowdsourcing data collection through a smart mobile terminal.

[0164] In an exemplary embodiment, L2 similarity, that is, Euclidean distance, is used for measurement. If the similarity is greater than 0.6, the crowd-sourced data is not stored; otherwise, the crowd-sourced data is stored.

[0165] In the 3D rendering implementation stage, the end-to-end model uses a sparse weight parameter model for model forward reasoning, which can achieve efficient and fast calculations while better handling complex light propagation situations. It can also adaptively adapt to different light sources and material properties, complete missing perspectives, and improve the efficiency and accuracy of reconstruction and rendering.

[0166] Step S907, obtaining crowdsourcing data from distributed data.

[0167] Step S908, obtaining a rendering model: using the optimized image sequence, relative pose, and longitude and latitude position information uploaded by crowdsourcing as input, using the corresponding incrementally generated neural radiation field model for inference to obtain a 3D rendering model with a full field of view;

[0168] Step S909, rendering and presentation: the three-dimensional rendering model obtained by cloud-based reasoning is sent to the requesting user's smart mobile device for viewing.

[0169] Among them, step S904 corresponds to the data distribution module 61 in the above embodiment, step S905 corresponds to the image processing module 62 in the above embodiment, and step S906 corresponds to the data storage module 63 in the above embodiment.

[0170] In the 3D rendering implementation stage, the model of the incrementally generated radiation field model system only needs to use a sparse weight parameter mode for model forward reasoning, which can achieve efficient and fast calculations while better handling complex light propagation situations, ensuring the real-time performance of 3D rendering.

[0171] Through the above steps, the data collection for 3D reconstruction and rendering using smart mobile devices is portable and flexible: traditional 3D reconstruction and rendering require the use of high-end equipment and complex software, and often require the data to be processed to be transferred to a computer for processing. In the embodiment of the present invention, the RGB camera of the smart mobile device is used for data collection, and the data is uploaded to the cloud for processing, which avoids the high requirements for device performance and makes data processing more flexible and convenient.

[0172] The above-mentioned embodiments of the present invention can be applied to smart mobile terminals and cloud-based distributed computing and storage nodes.

[0173] The smart mobile terminal is one of the important application media of the embodiment of the present invention, and is the entrance and exit for data uploading and rendering model receiving. The smart mobile terminal here can be a smart phone, tablet computer, portable computer, or other mobile device with a camera, computing power, and the ability to connect to the Internet. These devices have a wide range of application scenarios, are popular devices, and are very portable. Their advantage is that they can realize the crowdsourcing collection, uploading, and presentation of three-dimensional rendering models of the data required for three-dimensional reconstruction in a simple and easy-to-use way, providing a more efficient and convenient operating experience, and greatly reducing the professional threshold for three-dimensional modeling data collection.

[0174] Another application medium is distributed cloud computing and storage resources. The cloud system of the embodiment of the present invention is divided into a cloud distributed image data processing system and a cloud distributed image data processing system. The two end-to-end deep learning models contained therein require powerful computing resource support, and cloud computing resources play a vital role in the reconstruction and rendering process. By taking advantage of cloud computing, the efficient use and sharing of computing resources can be fully utilized, computing efficiency can be improved, and costs can be reduced. At the same time, the data storage and management functions of the cloud can also facilitate large-scale data management and sharing.

[0175] Therefore, the application environment of the present invention is intelligent mobile devices and cloud computing and storage resources. Such an application environment can give full play to the advantages of the present invention and realize efficient, convenient and high-quality 3D reconstruction and real-time rendering functions.

[0176] The method of the present invention is further described below with a specific scenario embodiment, which is a three-dimensional modeling of urban traffic street scenes. Specifically, a specific area of ​​the urban traffic street scene is three-dimensionally modeled and rendered for map navigation reference, display of urban landscape, traffic planning, and production of urban area digital sand table.

[0177] Step S1, preparation.

[0178] Before starting 3D modeling, some preparations need to be made. First, the target area for modeling needs to be determined, such as a specific area in a city, such as a commercial center or a tourist attraction. Second, a call is made to citizens (users) and the minimum configuration of mobile devices for participating in crowdsourced data collection is provided. These devices should be able to take high-quality photos and videos. Finally, a crowdsourced data collection and 3D rendering app is released, and simple and convenient operation instructions are provided so that users can quickly master the skills of taking photos or videos and uploading data to the cloud.

[0179] Step S2: capturing a multi-view image sequence.

[0180] In the target area, open the app and use your smart mobile device to take 10 photos in a row. These photos should cover the entire target area and capture as many objects as possible, such as buildings, streets, vehicles, pedestrians, etc., and try to ensure correct exposure and clear images.

[0181] Step S3, upload data to the cloud.

[0182] Through the app, the photos taken are uploaded to the cloud, and the location information is obtained through the app, which constitutes a crowdsourced data. This data will be used for subsequent feature extraction and 3D modeling. Since the crowdsourcing method is used to upload data, multiple people upload their own photos to cover a wider area and include modeling object information under different lighting conditions, which improves the quality and reliability of the data.

[0183] Step S4: cloud image data processing.

[0184] In the cloud, the image data uploaded by the crowdsourcing is divided by location area in combination with the distributed node mapping table constructed by the cloud preprocessing process. Several image sequences in the same location area will constitute an image sequence set, and the image sequence set will be sent to the corresponding distributed computing node. Then the cloud-based distributed image processing module and data storage module will process it. The end-to-end deep learning image processing model of the image processing module will perform image data processing, feature extraction, matching and pose calculation on the image sequence set, and finally obtain an optimized image sequence set and relative pose information. The data storage module will determine whether the crowdsourcing data needs to be stored based on the data_score obtained by the image processing module.

[0185] Step S5, neural radiation field model.

[0186] In the same distributed computing node, after the crowdsourced data is processed by the cloud image data, the optimized image sequence set, relative pose information, and longitude and latitude position information of the crowdsourced data will be used as input and processed by the cloud-based 3D reconstruction system. In the 3D reconstruction training stage, the cloud-based 3D reconstruction system trains the incremental generation of the neural radiation field model and uses the full amount of parameters for training. The model should be able to map the image data to the 3D world coordinate system, generate a 3D neural radiation field model, and automatically complete the neural radiation field information of the missing field of view. During the training process, incremental training can be achieved based on the scores of the image sequence, and the accuracy of the neural radiation field model can be continuously evolved. In the 3D rendering implementation stage, the cloud-based 3D reconstruction system uses the trained 3D neural radiation field model and sparse parameters for reasoning to improve the efficiency of 3D rendering model generation.

[0187] Step S6: Rendering the street view model

[0188] Through the above steps, the street view data has been successfully converted into a three-dimensional model. In the three-dimensional rendering implementation stage, the three-dimensional rendering model sent from the cloud can be received on a smart mobile device and viewed on the terminal screen for users to interact and browse.

[0189] When rendering, you can set different rendering parameters according to user needs, such as scene lighting, shadows, materials, etc., to meet the needs of different users. At the same time, you can also combine AR technology to integrate street view models into the real environment to provide a more immersive experience.

[0190] Through the above-mentioned embodiments of the present invention, the incremental generation of neural radiation field models is used for three-dimensional modeling and real-time rendering, which improves the rendering efficiency and quality, and realizes full-field-angle rendering of the modeled object: In the embodiments of the present invention, the neural radiation field model is trained using optimized image data, and deployed in the cloud to realize real-time rendering of the scene. Compared with traditional ray tracing-based rendering algorithms, the model in the embodiments of the present invention has higher rendering speed and rendering quality, and can provide more realistic rendering effects while ensuring real-time performance. In addition, the neural radiation field model of the present invention can realize incremental training, and the model will continue to evolve and become more and more accurate as high-quality crowdsourcing data is uploaded.

[0191] Based on the three-dimensional scene modeling method and cloud system proposed in the embodiment of the present invention, the three-dimensional scene modeling can have significant advantages such as low cost, high precision, high efficiency, and good usability. It adopts the cloud distributed computing method, can process a large amount of image data at the same time, and achieve compatibility with different devices, improves the convenience and universality of the system, can effectively solve the problems existing in traditional methods, and has broad application prospects and market value. It can be widely used in augmented reality, cultural heritage protection, digital cities, etc.

[0192] The smart mobile terminal used in the embodiment of the present invention is a popular device, which is very portable and has a wide range of application scenarios. Its advantage is that it can realize the crowdsourcing collection and uploading of data required for three-dimensional reconstruction and the presentation of three-dimensional rendering models in a simple and easy-to-use way, providing a more efficient and convenient operation experience, and greatly reducing the professional threshold for three-dimensional modeling data collection. Distributed cloud computing and storage nodes, cloud computing nodes play a vital role in the reconstruction and rendering process. In the embodiment of the present invention, the advantages of cloud computing are utilized to give full play to the efficient use and sharing of computing resources, improve computing efficiency and reduce costs. At the same time, the data storage and management functions of the cloud can also facilitate large-scale data management and sharing.

[0193] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0194] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0195] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0196] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0197] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0198] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0199] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional scene modeling method, characterized in that: Applied to cloud systems, including: Based on multiple crowdsourced data collected and uploaded by the user terminal, three-dimensional scene modeling is performed on the scene to be modeled by incrementally generating a neural radiation field model, so as to obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

2. The method according to claim 1, characterized in that Based on multiple crowdsourced data collected and uploaded by user terminals, a neural radiation field model is incrementally generated in the cloud system to perform three-dimensional scene modeling on the scene to be modeled, including: The first cloud system performs image data processing on the plurality of crowdsourced data to obtain an optimized first image sequence set and relative position information of each of the crowdsourced data; wherein one of the crowdsourced data corresponds to a first image sequence in the first image sequence set; The first cloud system updates the information stored in the corresponding fields of the plurality of crowd-sourced data based on the first image sequence, the relative posture information, and the first position information corresponding to each of the crowd-sourced data, and inputs the updated plurality of crowd-sourced data into the second cloud system; The second cloud system performs three-dimensional scene modeling by incrementally generating a neural radiation field model based on the updated plurality of crowd-sourced data, thereby obtaining a three-dimensional rendering model of the scene to be modeled with a full field of view.

3. The method according to claim 2, characterized in that The first cloud system performs image data processing on the plurality of crowdsourced data, including: According to the first location information and based on the mapping relationship table between the modeling sub-areas and the distributed computing nodes, the plurality of crowdsourced data are sent to the corresponding distributed computing nodes; wherein the modeling sub-areas are obtained by dividing the area corresponding to the scene to be modeled by the first cloud system according to the prior digital map, and one modeling sub-area corresponds to one first location information; The first cloud system receives a second image sequence set returned by the distributed computing node, and inputs the second image sequence set into an end-to-end deep learning image processing model to perform the image data processing, wherein each image sequence in the second image sequence set corresponds to one of the first position information.

4. The method according to claim 3, characterized in that The first cloud system processes the plurality of crowdsourced data to obtain an optimized first image sequence set, including: For any second image sequence in the second image sequence set, the first cloud system extracts the mask of the dynamic object in each image in the second image sequence through the image semantic segmentation, and maps the mask back to the feature map corresponding to the image; wherein the dynamic object includes: pedestrians, animals, vehicles, and movable still objects related to the pedestrians; Based on the contextual relationship of the mask in the feature map and the feature relationship between the previous and next frames of the second image sequence, background completion is performed on the image inside the mask through the image generation to obtain an optimized image sequence; Through the above step of optimizing the image sequence, all the second image sequences in the second image sequence set are optimized to obtain the first image sequence set.

5. The method according to claim 2, characterized in that: The first cloud system performs image data processing on the plurality of crowdsourced data to obtain relative position information of each of the crowdsourced data, including: For any of the first image sequences, extracting descriptor information of each image in the first image sequence through the descriptor feature extraction and the intra-feature matching, and setting the image with the most common view descriptors as a key frame of the first image sequence; The fine-grained feature extraction is to extract fine-grained features from the key frame to obtain deep features, wherein the deep features include fine-grained representation features and fine-grained semantic features; Through the feature external matching, the first position information corresponding to the first image sequence is feature spliced ​​with the depth feature to obtain the key frame feature of the first image sequence; Through the feature external matching, the loop closure detection and the back-end pose optimization, feature matching, loop closure detection and graph optimization are performed on the deep features in the key frame features of the first image sequence, and based on the corresponding relationship between each of the first image sequences and the plurality of crowdsourced data, the relative pose information of the crowdsourced data corresponding to the first image sequence is obtained; Through the above step of obtaining relative posture information, the relative posture information of other crowdsourcing data is obtained.

6. The method according to claim 2, characterized in that Before inputting the updated plurality of crowd-sourced data into the second cloud system, the method further includes: For any of the crowd-sourced data, judging the stage of the crowd-sourced data according to the stage identification field of the crowd-sourced data; For the first crowdsourced data in the 3D reconstruction training stage, storing the first crowdsourced data having a first score greater than or equal to a first preset threshold into a distributed database; wherein the first score is obtained by the cloud system scoring the image quality of the image sequence corresponding to the first crowdsourced data through an image quality judgment component; For the second crowd-sourced data in the three-dimensional rendering implementation stage, the key frame features of the second crowd-sourced data are matched with the corresponding key frame features stored in the distributed database for similarity, and the second crowd-sourced data with a similarity less than a second preset threshold is stored in the distributed database.

7. The method according to claim 6, characterized in that Before performing three-dimensional scene modeling by incrementally generating a neural radiation field model, the method further includes: The second cloud system obtains a plurality of the first crowd-sourced data from a distributed database, and sorts the plurality of the first crowd-sourced data in descending order based on the first scores of the first crowd-sourced data; Inputting the first N first crowdsourced data, the relative posture information of the first N first crowdsourced data, and the first position information corresponding to the first N first crowdsourced data into the pre-designed incremental generation of neural radiation field model to obtain the trained incremental generation of neural radiation field model, wherein N is an integer greater than zero; Adjusting the weights and biases of the neural network in the trained incrementally generated neural radiation field model to converge the incrementally generated neural radiation field model; When the incremental generation of the neural radiation field model completes convergence, based on the weighted scores of the first N first crowd-sourced data, the multiple first crowd-sourced data are sorted in order from large to small, and the first M first crowd-sourced data are selected to perform fine-tuning training on the incremental generation of the neural radiation field model, where M is an integer greater than zero and less than N.

8. A cloud system, characterized in that: include: The modeling module is used to perform three-dimensional scene modeling on the scene to be modeled based on multiple crowdsourced data collected and uploaded by the user terminal by incrementally generating a neural radiation field model, so as to obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images