Three-dimensional scene modeling method, cloud system, storage medium, and electronic device

Abstract

Embodiments of the present disclosure provide a three-dimensional scene modeling method, a cloud system, a storage medium, and an electronic device. The method comprises: on the basis of multiple pieces of crowdsourcing data collected and uploaded by a user terminal, performing three-dimensional scene modeling on a scene to be modeled by means of an incremental generative neural radiance field model, to obtain a three-dimensional rendering model of a full field of view of said scene.

Description

Three-dimensional scene modeling method, cloud system, storage medium and electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on Chinese patent application CN202310803208.6, filed on June 30, 2023, entitled “Three-dimensional scene modeling method, cloud system, storage medium and electronic device”, and claims the priority of the patent application, and all the contents disclosed therein are incorporated into this disclosure by reference. Technical Field

[0003] The embodiments of the present disclosure relate to the field of communications, and more specifically, to a three-dimensional scene modeling method, a cloud system, a storage medium, and an electronic device. Background Art

[0004] Some existing 3D reconstruction and rendering methods have the following problems:

[0005] Difficulty in data acquisition: Existing 3D reconstruction methods require the use of specialized equipment, such as laser sensors, depth cameras, or a camera array consisting of multiple RGBD cameras (RGBD represents the three primary colors plus image depth). Furthermore, data acquisition by specialized operators is required to obtain high-precision 3D reconstruction source data. This makes data acquisition difficult and costly.

[0006] Poor dynamic scene reconstruction: When performing 3D reconstruction of outdoor buildings and street scenes, there are usually dynamic objects in the scene, such as pedestrians, animals, vehicles (such as cars, trucks, non-motorized vehicles, etc.), and static objects associated with pedestrians (such as handheld laptops, handbags, suitcases, etc.). Dynamic scene reconstruction requires a lot of time and computing resources. At the same time, when dynamic objects block the modeling subject, the 3D reconstruction effect will be even less ideal.

[0007] Lack of perspective for the reconstructed subject: For the 3D modeling of the reconstructed subject, especially high-rise office buildings and cable-stayed bridges, it is difficult to capture images from a bird's-eye view without the help of professional drone acquisition equipment. This leads to a lack of data source in the 3D reconstruction training stage, making it difficult to achieve 360-degree panoramic 3D reconstruction and rendering.

[0008] Low rendering efficiency: Existing 3D rendering tasks are mostly calculated on smart mobile devices. Considering the computing power limitations of mobile devices, they can often only provide lower visual resolution and simplified rendering models, which limits the rendering effect and experience.

[0009] Summary of the Invention

[0010] Embodiments of the present disclosure provide a three-dimensional scene modeling method, a cloud system, a storage medium, and an electronic device.

[0011] According to one embodiment of the present disclosure, a three-dimensional scene modeling method is provided, including: based on multiple crowdsourced data collected and uploaded by a user terminal, three-dimensional scene modeling is performed on the scene to be modeled by incrementally generating a neural radiation field model, and a three-dimensional rendering model of the scene to be modeled with a full field of view angle is obtained.

[0012] According to another embodiment of the present disclosure, a cloud system is provided, including: a modeling module, which is configured to perform three-dimensional scene modeling on the scene to be modeled based on multiple crowdsourced data collected and uploaded by a user terminal by incrementally generating a neural radiation field model, and obtain a three-dimensional rendering model of the scene to be modeled with a full field of view angle.

[0013] According to another embodiment of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when running.

[0014] According to another embodiment of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a hardware structure block diagram of a computer terminal for running a three-dimensional scene modeling method according to an embodiment of the present disclosure;

[0016] FIG2 is a network architecture diagram of a three-dimensional scene modeling system according to an embodiment of the present disclosure;

[0017] FIG3 is a flow chart of a three-dimensional scene modeling method according to an embodiment of the present disclosure;

[0018] FIG4 is a structural block diagram of a cloud system according to an embodiment of the present disclosure;

[0019] FIG5 is a detailed network architecture diagram of a three-dimensional scene modeling system according to an embodiment of the present disclosure;

[0020] FIG6 is a schematic diagram of a cloud-based distributed image data processing system according to an embodiment of the present disclosure;

[0021] FIG7 is a schematic diagram of an end-to-end deep learning model according to an embodiment of the present disclosure;

[0022] FIG8 is a flow chart of a 3D reconstruction training phase according to an embodiment of the present disclosure;

[0023] FIG9 is a flowchart of a three-dimensional rendering implementation stage according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and in conjunction with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal running a three-dimensional scene modeling method in an embodiment of the present disclosure. As shown in Figure 1, the computer terminal may include one or more (only one is shown in Figure 1) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor or a programmable logic device) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include more or fewer components than those shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the three-dimensional scene modeling method in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] The above-mentioned three-dimensional scene modeling method of the embodiment of the present disclosure can also run on the network architecture shown in Figure 2. As shown in Figure 2, the network architecture includes: a smart mobile terminal 21, a distributed computing node 22, a cloud system 23, and a distributed storage node 24, wherein the cloud system 23 includes: a cloud-based distributed image data processing system 241 and a cloud-based three-dimensional reconstruction system 242.

[0030] Specifically, the smart mobile terminal 21 is configured to collect crowdsourced data for modeling a three-dimensional scene and send the crowdsourced data to a distributed computing node; wherein, crowdsourcing technology refers to a method of outsourcing work, tasks or projects to a large number of people (i.e., "crowdsourcing workers") through the Internet for completion.

[0031] The distributed computing node 22 is configured to process the received crowdsourced data into crowdsourced data that can be used in the cloud system 23 .

[0032] The cloud system 23 is configured to perform 3D reconstruction and 3D rendering based on the received crowdsourced data, obtain a 3D rendering model of the scene to be modeled, and send the 3D rendering model to the smart mobile terminal 21 for display.

[0033] The distributed storage node 24 is configured to receive crowdsourcing data generated by the cloud system 23 during the 3D reconstruction and 3D rendering process and to be stored.

[0034] In this embodiment, a three-dimensional scene modeling method running on the above-mentioned computer terminal or network architecture is provided. FIG3 is a flow chart of the three-dimensional scene modeling method according to an embodiment of the present disclosure. As shown in FIG3 , the flow chart includes the following steps:

[0035] Step S302 , based on the multiple crowdsourced data collected and uploaded by the user terminal, three-dimensional scene modeling is performed on the scene to be modeled by incrementally generating a neural radiation field model to obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

[0036] Among them, the user terminal is functionally equivalent to the smart mobile terminal in Figure 2, the first cloud system is functionally equivalent to the above-mentioned cloud distributed image data processing system 241, and the second cloud system is functionally equivalent to the above-mentioned cloud 3D reconstruction system 242.

[0037] In this embodiment, the crowdsourcing data includes the following fields: an image sequence field, a location information field, and a stage identification field; wherein the image sequence field is used to store an image sequence, for example: 10 RGB images continuously taken by a user terminal, and the image sequence field is represented by data_frames; the location information field is used to store location information, which refers to the latitude and longitude information when the user terminal takes the image, and the images in the same image sequence share the location information, and the location information field is represented by data_location; the stage identification field is used to store a stage flag, and the stage flag is used to identify the stage used by the crowdsourcing data, including 0 and 1, for example: the stage identification bit of the crowdsourcing data used for the three-dimensional reconstruction stage is 0, or the stage identification bit used for the three-dimensional rendering implementation stage is 1; represented by data_phase.

[0038] In an exemplary embodiment, a piece of crowdsourced data includes: a sequence of 10 consecutive RGB images, corresponding latitude and longitude position information, and a data stage flag.

[0039] In step S302 of this embodiment, three-dimensional scene modeling is performed on the scene to be modeled based on multiple crowd-sourced data collected and uploaded by the user terminal through the incremental generation of a neural radiation field model in the cloud system, including: the first cloud system performs image data processing on the multiple crowd-sourced data to obtain an optimized first image sequence set and relative pose information of each of the crowd-sourced data; wherein, one of the crowd-sourced data corresponds to a first image sequence in the first image sequence set; the first cloud system updates the information stored in the corresponding fields of the multiple crowd-sourced data based on the first image sequence, the relative pose information and the first position information corresponding to each of the crowd-sourced data, and inputs the updated multiple crowd-sourced data into the second cloud system; the second cloud system performs three-dimensional scene modeling by incrementally generating a neural radiation field model based on the updated multiple crowd-sourced data to obtain a three-dimensional rendering model with a full field of view of the scene to be modeled.

[0040] In one embodiment, the first cloud system divides the area corresponding to the scene to be modeled into multiple modeling sub-areas in advance based on a prior digital map, and constructs a mapping relationship table between the modeling sub-areas and the distributed computing nodes. According to the first location information and based on the mapping relationship table, the multiple crowdsourced data are sent to the corresponding distributed computing nodes; wherein, one modeling sub-area corresponds to one first location information; then the first cloud system receives the second image sequence set returned by the distributed computing node, and inputs the second image sequence set into an end-to-end deep learning image processing model for image data processing, wherein each image sequence in the second image sequence set corresponds to one first location information.

[0041] In one embodiment, before the first cloud system performs image data processing on the multiple crowdsourced data uploaded by the user terminal, it will also add the following five fields to the crowdsourced data: data label field, data score field, key frame sequence number field, key frame feature field, and relative posture field; and perform the image data processing on the multiple crowdsourced data after the fields are added.

[0042] In one embodiment, the image data processing includes: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, extra-feature matching, loop detection and posture backend optimization.

[0043] In one embodiment, a first cloud-based system processes image data of a plurality of crowdsourced data uploaded by a user terminal to obtain an optimized first image sequence set, including: for any second image sequence in the second image sequence set, the first cloud-based system extracts masks of dynamic objects in each image in the second image sequence through the image semantic segmentation, and maps the masks back to the feature map corresponding to the image; wherein the dynamic objects include: pedestrians, animals, vehicles, and movable still objects related to the pedestrians; based on the contextual relationship of the mask in the feature map and the feature relationship between the previous and next frames of the second image sequence, the background of the image inside the mask is completed through the image generation to obtain an optimized image sequence; through the above-mentioned steps of optimizing the image sequence, all the second image sequences in the second image sequence set are optimized to obtain the first image sequence set.

[0044] In an exemplary embodiment, before extracting the masks of dynamic objects in each image in the image sequence, the method further includes: scoring the image quality of the second image sequence through the image quality judgment to obtain a first score of the second image sequence, and updating the first score to the data score field of the crowdsourcing data corresponding to the second image sequence; and unifying the resolution and size of the scored images through the image denoising and the image super-resolution sharing weights.

[0045] In one embodiment, a first cloud system processes image data of a plurality of crowdsourced data uploaded by a user terminal to obtain relative pose information of each of the crowdsourced data, including: for any first image sequence in the first image sequence set, extracting descriptor information of each image in the first image sequence through the descriptor feature extraction and the intra-feature matching, and setting the image with the most common view descriptors as a key frame of the first image sequence; performing fine-grained feature extraction on the key frame to obtain depth features through the fine-grained feature extraction, wherein the depth features include fine-grained representational features and fine-grained semantic features; performing feature splicing on first position information corresponding to the first image sequence and the depth features through the extra-feature matching to obtain key frame features of the first image sequence; performing feature matching, loop closure detection, and graph optimization on the depth features in the key frame features of the first image sequence through the extra-feature matching, the loop closure detection, and the back-end pose optimization, and obtaining the relative pose information of the crowdsourced data corresponding to the first image sequence based on the correspondence between each of the first image sequence and the plurality of crowdsourced data; and obtaining the relative pose information of other crowdsourced data through the above-mentioned steps of obtaining relative pose information.

[0046] In one embodiment, the first cloud system updates the plurality of crowd-sourced data accordingly based on the first image sequence, the relative pose information, and the first position information corresponding to each of the crowd-sourced data, and inputs the updated plurality of crowd-sourced data into the second cloud system;

[0047] In one embodiment, before inputting the updated multiple crowdsourced data into the second cloud system, the method further includes: for any of the crowdsourced data, determining the stage of the crowdsourced data based on the stage identification field of the crowdsourced data; for the first crowdsourced data in the three-dimensional reconstruction training stage, storing the first crowdsourced data having a first score greater than or equal to a first preset threshold in a distributed database; for the second crowdsourced data in the three-dimensional rendering implementation stage, performing similarity matching between the key frame features of the second crowdsourced data and the corresponding key frame features stored in the distributed database, and storing the second crowdsourced data having a similarity less than a second preset threshold in the distributed database.

[0048] In one embodiment, the second cloud system performs three-dimensional scene modeling by incrementally generating a neural radiation field model based on the updated plurality of crowdsourced data, thereby obtaining a three-dimensional rendering model of the scene to be modeled with a full field of view.

[0049] In one embodiment, before performing three-dimensional scene modeling through the incremental generation of a neural radiation field model, the method further includes: the second cloud system obtains a plurality of the first crowd-sourced data from a distributed database, and sorts the plurality of the first crowd-sourced data in descending order based on the size of the first scores of the first crowd-sourced data; inputs the first N first crowd-sourced data, the relative posture information of the first N first crowd-sourced data, and the first position information corresponding to the first N first crowd-sourced data into the pre-designed incremental generation of a neural radiation field model to obtain the trained incremental generation of a neural radiation field model, wherein N is an integer greater than zero; adjusts the weights and biases of the neural network in the trained incremental generation of a neural radiation field model to converge the incremental generation of a neural radiation field model; when the incremental generation of a neural radiation field model completes convergence, sorts the plurality of the first crowd-sourced data in descending order based on the size of the weighted scores of the first N first crowd-sourced data, and selects the first M first crowd-sourced data to perform fine-tuning training on the incremental generation of a neural radiation field model, wherein M is an integer greater than zero and less than N.

[0050] In an exemplary embodiment, before sorting the plurality of first crowd-sourced data in descending order based on the sizes of the final scores of the first N first crowd-sourced data, the method further includes: scoring the first N first crowd-sourced data to obtain second scores of the first N first crowd-sourced data; and performing weighted calculation on the first scores and the second scores of the first N first crowd-sourced data to obtain weighted scores of the N first crowd-sourced data.

[0051] Through the above steps, the three-dimensional reconstruction and rendering of the scene are performed through the cloud system, avoiding the limitation of the three-dimensional reconstruction and rendering effect due to insufficient computing power of the mobile terminal. The crowdsourced data used by the cloud system can be collected and uploaded by the user terminal, which reduces the difficulty of data acquisition. At the same time, through the incremental generation of the neural radiation field model, the three-dimensional radiation field distribution of the missing field of view angle can be generated to obtain a three-dimensional rendering model with a full field of view angle. Therefore, it can solve the problems of data acquisition difficulties, missing reconstruction subject perspectives, and low rendering efficiency in existing three-dimensional reconstruction and rendering methods in related technologies, and achieve the effect of improving the efficiency of three-dimensional scene modeling.

[0052] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0053] In this embodiment, a three-dimensional scene modeling system is also provided. The system is configured to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0054] FIG4 is a structural block diagram of a cloud system according to an embodiment of the present disclosure. As shown in FIG4 , the cloud system includes:

[0055] The modeling module 10 is configured to perform three-dimensional scene modeling on the scene to be modeled based on multiple crowdsourced data collected and uploaded by the user terminal by incrementally generating a neural radiation field model, and obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

[0056] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0057] To facilitate understanding of the technical solutions provided by the present disclosure, embodiments of specific scenarios will be described in detail below.

[0058] Figure 5 is a detailed network architecture diagram of the three-dimensional scene modeling system according to an embodiment of the present disclosure. As shown in Figure 5, in addition to including: a smart mobile terminal 21, a distributed computing node 22, a cloud system 23, a distributed storage node 24, a cloud distributed image data processing system 241, and a cloud three-dimensional reconstruction system 242, the cloud distributed image data processing system 241 also includes a data distribution module 61, an image processing module 62 and a data storage module 63, and the cloud three-dimensional reconstruction system 242 includes incremental generation of neural radiation fields.

[0059] Among them, the cloud-based distributed image data processing system 241 and the cloud-based three-dimensional reconstruction system 242 of the cloud-based system 23 both include an end-to-end deep learning model, so the cloud-based system 23 has powerful computing resource support.

[0060] The cloud-based distributed image processing system is a cloud-based distributed image processing system for crowdsourced data. It is deployed in the cloud and combines end-to-end deep learning models, distributed computing technology, and distributed storage technology. It is a distributed image processing system.

[0061] In this embodiment, the smart mobile terminal is configured to collect data and upload crowdsourced data, including:

[0062] The RGB camera equipped with a smart mobile terminal is used as the data acquisition medium and crowdsourcing is used to collect data. Based on the collected data, three types of information are uploaded: image sequence, latitude and longitude position information at the time of shooting (obtained through the smart mobile terminal), and data stage flag. These three types of information together constitute a crowdsourcing data.

[0063] Here, an image sequence refers to multiple RGB images taken continuously, which is represented by the field data_frames; for example, a set of image sequences is 10 RGB images taken continuously.

[0064] The latitude and longitude information at the time of shooting is shared by this set of image sequences. For example, 10 RGB images share the same location information, which is represented by the field data_location;

[0065] The data phase flag is used to indicate the specific phase of the data effect, represented by the field data_phase, where data_phase is 0 for the 3D reconstruction training phase, and data_phase is 1 for the 3D rendering implementation phase.

[0066] In one embodiment, the specific phase of the uploaded crowdsourced data is distinguished based on its phase flag. For example, a data_phase of 0 indicates the 3D reconstruction training phase, while a data_phase of 1 indicates the 3D rendering implementation phase. During the 3D reconstruction training phase, the crowdsourced data serves as the source data for 3D reconstruction and is uploaded to the cloud by the smart mobile terminal. Upon successful upload, the lifecycle of the smart mobile terminal's data collection task ends. During the 3D rendering implementation phase, the crowdsourced data serves as query data for 3D rendering and is uploaded to the cloud by the smart mobile terminal, awaiting system matching and query. If a corresponding 3D reconstruction model is found, it is sent from the cloud to the smart mobile terminal for display on the smart mobile terminal.

[0067] After the smart mobile terminal collects the data, it uploads a sequence of 10 continuous RGB images, the corresponding latitude and longitude information at the time of shooting, and the data stage flag as a crowdsourced data to the cloud system, namely the cloud-based distributed image data processing system and the cloud-based 3D reconstruction system.

[0068] Figure 6 is a schematic diagram of a cloud-based distributed image data processing system according to an embodiment of the present disclosure. As shown in Figure 6, before the cloud-based system 23 is put into operation, the cloud-based distributed image data processing system is provided with a preprocessing process, that is, the cloud-based distributed image data processing system will pre-select a suitable modeling area division method based on a digital map of prior data as a template according to actual application conditions, divide the entire modeling area into multiple modeling sub-areas, and construct a mapping table between modeling sub-areas and distributed computing nodes, wherein a modeling sub-area corresponds one-to-one to an independent distributed computing node. This mapping table needs to be used in both the 3D reconstruction training stage and the 3D rendering implementation stage.

[0069] In one embodiment, there are two ways to divide the modeling area:

[0070] 1) Based on the prior digital map, the entire modeling area is divided into multiple modeling sub-areas with uniform areas using streets as dividing lines;

[0071] 2) Based on the prior digital map, the modeling area is divided into multiple sub-areas with uneven areas, taking the landmark building as the center of the modeling sub-area.

[0072] The cloud-based distributed image data processing system is also configured to, after receiving the uploaded crowd-sourced data, add new fields to the uploaded crowd-sourced data, find the modeling sub-area to which the longitude and latitude information of the crowd-sourced data belongs, and then send the crowd-sourced data to the corresponding distributed computing node based on the previously constructed mapping table.

[0073] Among them, the five fields supplemented by the cloud-based distributed image data processing system for crowdsourcing data are: data_id, data_score, data_keyframe_id, data_keyframe_feature, and data_frame_pose.

[0074] 1) The data_id field is used to encapsulate the unique code of the crowdsourced data. The data_id of each piece of crowdsourced data is different. The cloud-based distributed image data processing system generates a unique code based on the upload time, geographic location information, and device model information of the crowdsourced data, and updates it to the data_id field.

[0075] 2) The data_score field is used to encapsulate the image sequence score of the crowdsourced data. The score range of the image sequence of the crowdsourced data is [0, 100].

[0076] The image sequence score of crowdsourced data is a floating-point number used for image sequence evaluation and screening. The data_score is initially scored by the cloud-based distributed image data processing system. Subsequently, the data_score is re-weighted and calculated by the incremental radiation field model system to obtain the final score. The data_score in the incremental radiation field model system is calculated using the following formula (1): data_score = a*quality_score + b*nerf_score Formula (1)

[0077] Among them, quality_score is calculated by the image processing module 62 of the cloud-based distributed image data processing system, and its initial value is 0; nerf_score is calculated by the incremental generation of neural radiation field model of the incremental generation of radiation field model system, and its initial value is 0; a and b are weight hyperparameters. In the cloud-based distributed image data processing system, a=0.9, b=0; in the incremental generation of neural radiation field model of the incremental generation of radiation field model system, a=0.4, b=0.6.

[0078] 3) The data_keyframe_id field is used to encapsulate the crowdsourced data keyframe number. The crowdsourced data keyframe number is represented by an integer in [0, 9], which represents the most representative image number among the 10 RGB images in an image sequence. The corresponding image can be found in data_frames through data_keyframe_id.

[0079] 4) The data_keyframe_feature field is used to encapsulate the keyframe deep learning features, which represent the deep learning features of the keyframe image. The unified dimension is 130 and is calculated by the cloud-based distributed image data processing system.

[0080] 5) data_frame_pose field, used to encapsulate the relative pose information of the crowdsourced data image. The relative pose information of the crowdsourced data image is represented by the displacement parameter T and the rotation parameter Q. Where T = (x t ,y t ,z t ), represents a three-dimensional translation vector, Q=(w q ,x q ,y q ,z q ) represents a rotation vector in quaternion form. T and Q can represent the relative displacement and rotation relationship between images in an image sequence. Each image in the image sequence has a set of T and Q values.

[0081] The relative pose of an image sequence refers to the relative position and orientation between two images. It typically consists of a rotation vector and a translation vector, which describe the transformation from one camera coordinate system to another. In computer vision and robotics, determining the relative pose of a set of images is often necessary to align, register, or establish a spatial relationship between them. During 3D reconstruction training, determining the relative pose of two cameras is crucial in order to calculate corresponding points in the scene's field of view.

[0082] Estimating the relative pose of images is an important problem in computer vision and robotics. There are many methods to achieve this, including feature matching, structured light, deep learning, etc. The embodiments of this disclosure use relative pose based on deep learning feature matching.

[0083] After the fields are supplemented, the fields included in the crowdsourcing data format are shown in Table 1. That is, a complete crowdsourcing data has 8 attributes, which are represented by 8 fields.

[0084] Table 1 Crowdsourcing data format

[0085] The distribution module 61 is configured to supplement the uploaded crowdsourcing data with five fields, and then divide the uploaded crowdsourcing data into corresponding modeling areas based on the latitude and longitude position information in the crowdsourcing data, and then send the crowdsourcing data to the corresponding distributed computing node 22 through the mapping relationship table between the modeling area and the distributed computing node.

[0086] The distributed computing node 22 is configured to receive multiple crowdsourced data from the same area but at different specific locations, extract the image sequence corresponding to the data_frames field of the crowdsourced data and the longitude and latitude location information represented by data_location, and input the image sequence set consisting of the multiple crowdsourced data image sequences and the corresponding location information into the end-to-end deep learning image processing model of the image processing module 62;

[0087] Image processing module 62 primarily comprises an end-to-end deep learning image processing model, with all processing steps executed on the GPU. This end-to-end deep learning image processing model performs one CPU-GPU I / O operation for data input and one GPU-CPU I / O operation for data output, for a total of two I / O operations. All other image data processing is performed entirely on the GPU, resulting in extremely high image processing efficiency. A schematic diagram of the end-to-end deep learning model architecture is shown in Figure 7.

[0088] The end-to-end deep learning image processing model in the image processing module 62 takes image sequence and longitude and latitude position information as input, and outputs optimized image sequence, image sequence score, key frame number, key frame deep learning feature and relative pose of the image sequence. Based on the powerful feature extraction capability of the deep convolutional neural network and the nonlinear mapping capability brought by the activation function, combined with the downsampling, upsampling and feature fusion methods of the convolutional neural network, it can realize the image data processing capabilities that can only be achieved by traditional multi-model series connection: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, feature intra-matching, fine-grained feature extraction, feature extra-matching, loop detection and pose back-end optimization.

[0089] In one embodiment, in the image quality judgment process, the image quality judgment component in the cloud-distributed image data processing system 241 includes a softmax component. The softmax component uses the feature extraction and feature abstraction capabilities of the convolutional neural network to judge the quality of the input image, and scores the image sequence, recording the score as quality_score; and updates the quality_score to the attribute data_score field of the crowdsourced data.

[0090] For example, the input image sequence is scored to obtain a quality_score. A higher quality_score value indicates higher quality of the original image and a greater contribution to the end-to-end deep learning model. Conversely, if the quality_score value is below a threshold (e.g., 0.3), during model training, during the forward inference process of the end-to-end deep learning model, the convolutional neural network feature weights corresponding to that quality_score value are reset to 0, meaning they do not participate in subsequent feature extraction and function implementation. This improves the forward inference efficiency of the end-to-end deep learning model, and therefore the model training efficiency.

[0091] As a normalized output component, softmax is a commonly used indicator in multi-classification problems, used to represent the probability distribution of a sample belonging to each category. In deep learning, softmax is often used as the activation function of the output layer, which can map the output of the network to a probability distribution. In the embodiment of the present disclosure, the softmax output value is a floating point number with a value range of [0,100]. The higher the softmax score, the higher the image quality and the greater the contribution to the subsequent cloud-based 3D reconstruction system.

[0092] In this embodiment, unlike conventional deep learning image processing systems, image processing module 62 utilizes an end-to-end deep learning model. Compared to conventional pipeline-based deep learning models, this significantly reduces CPU-GPU data exchange operations and significantly reduces the data I / O throughput of cloud computing nodes. This significantly improves image data processing efficiency, particularly when processing massive amounts of crowdsourced data. Through distributed cloud computing nodes, optimized image sequences, image sequence scores, keyframe numbers, keyframe deep learning features, and relative pose information for images within the sequence can be obtained.

[0093] By employing an end-to-end deep learning algorithm for image data processing, the accuracy of reconstruction and rendering is effectively supported: This embodiment uses an end-to-end deep learning algorithm. Except for the input / output I / O operations, all calculations are performed on the GPU. This implements image quality assessment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, extra-feature matching, loop detection, and back-end pose optimization. Compared to traditional computer vision algorithms, this algorithm has higher accuracy and robustness, can handle more complex scenes and data, and improves the accuracy of reconstruction and rendering. Compared to the pipeline processing of deep learning models, it greatly reduces I / O operations and improves the real-time performance of reconstruction and rendering.

[0094] The data storage module 63 is configured to store image sequences (crowdsourced data) whose scores in the data_score field are greater than or equal to 0.3 to the distributed storage node 24 after calculating the image quality score quality_score in the softmax layer and updating the data_score field of the crowdsourced data. Image sequences (crowdsourced data) whose scores are less than 0.3 are not stored, and the corresponding quality_score score values ​​are updated to the data_score field.

[0095] In one embodiment, image denoising and image super-resolution share weights to unify the processing of crowdsourced data. Since the devices used for crowdsourcing uploads are different, the resolution and size of the uploaded images are also inconsistent. Therefore, the data needs to be processed uniformly, and the image sequences are unified to the same resolution and size through image denoising and image super-resolution.

[0096] In one embodiment, image semantic segmentation and image generation are two independent sub-branches, which share the previous feature extraction and have a strict sequential relationship. That is, the image semantic segmentation component is first used to extract the mask of the dynamic object in the image, and then the mask is mapped back to the feature map corresponding to the image. The image generation component uses the contextual relationship of the mask in the feature map and the feature relationship between the previous and next frames of the image sequence to regenerate the image inside the mask and complete it with background information unrelated to the dynamic object.

[0097] The mask is the outline of the dynamic object, the area within the outline is the main body of the dynamic object, and the area outside the outline is the background that has nothing to do with the dynamic object.

[0098] In this embodiment, dynamic objects can be divided into four categories: (1) pedestrians, i.e., pedestrians sitting, standing, or moving in the image; (2) animals, i.e., cats, dogs, birds, etc. in the image; (3) vehicles, i.e., cars, trucks, buses, non-motorized vehicles, etc.; and (4) pedestrian-related movable static objects, such as laptops, handbags, suitcases, and long-handled umbrellas held by people. The image semantic segmentation component performs pixel-level classification on these four categories of dynamic objects and generates corresponding masks.

[0099] There are two conditions for determining dynamic objects in category (4), and only one of them is required:

[0100] a. The pedestrian and the movable static object share the same mask edge. In this case, the pedestrian and the movable static object happen to be in contact, that is, the pedestrian mask and the movable static object mask share part of the same mask edge;

[0101] b. The intersection-over-union (IoU) of the pedestrian segmentation area and the movable static object segmentation area is greater than 0.1, as shown in formula (2) and formula (3). At this time, there is an interactive relationship between pedestrians and movable static objects, that is, the ratio of the intersection area and the union area of ​​the pedestrian mask and the movable static object mask is greater than 0.1. IoU mask =occur_maskunion_mask Formula (2) IoU mask >0.1 Formula (3)

[0102] Among them, occur_mask represents the intersection area of ​​the masks of pedestrians and movable static objects, and union_mask represents the union area of ​​the masks of pedestrians and movable static objects.

[0103] This end-to-end deep learning image processing model, based on image semantic segmentation and image generation components, filters out dynamic objects and completes clean backgrounds in the output image. The image semantic segmentation and image generation components jointly implement dynamic object filtering and background completion. The mask represents the outline of the dynamic object, with the main body of the dynamic object inside the outline and the background outside the outline unrelated to the dynamic object. This generates an optimized image for the image sequence set and copies it to the data_frames field, overwriting the originally uploaded image sequence.

[0104] In one embodiment, descriptor feature extraction and intra-feature matching are used to extract the descriptor information of the image, and match the corresponding image with the most common view descriptors, set it as the key frame of the image sequence, and update the key frame id of the corresponding image sequence, that is, update the data_keyframe_id field.

[0105] In one embodiment, fine-grained feature extraction is to further extract fine-grained features from key frame features, which are composed of 64-dimensional fine-grained representation features and 64-dimensional fine-grained semantic features, for a total of 128-dimensional deep features.

[0106] In one embodiment, when matching outside of features, key frame feature splicing is required. The feature composition consists of: 128-dimensional fine-grained features and 2-dimensional location information (latitude and longitude information), and finally a 130-dimensional key frame feature is obtained. The corresponding key frame deep learning feature attributes are updated, that is, the data_keyframe_feature field is updated.

[0107] In one embodiment, feature out-matching, loop detection, and back-end pose optimization perform feature matching, loop detection, and graph optimization on fine-grained features of key frames selected from different image sequences to obtain pose information of the image sequence and record it in the data_frame_pose field.

[0108] Unlike conventional deep learning image processing systems, the image processing module 62 of this embodiment utilizes an end-to-end deep learning model. Compared to conventional pipeline-based deep learning models, this significantly reduces CPU-GPU data exchange operations and significantly reduces the data I / O throughput of cloud computing nodes. This significantly improves image data processing efficiency, particularly when processing massive amounts of crowdsourced data. Through distributed cloud computing nodes, optimized image sequences, image sequence scores, keyframe numbers, keyframe deep learning features, and relative pose information for images within the sequence can be obtained.

[0109] The data storage module 63 is further configured to first determine the phase of the crowdsourced data by using the data_phase field in the crowdsourced data when storing the crowdsourced data;

[0110] When the crowdsourced data is in the 3D reconstruction training stage, the quality_score score of the image sequence after passing through the image processing module is obtained through the data_score field in the crowdsourced data. Among them, the crowdsourced data with a quality_score score lower than 0.3 is not stored, and the crowdsourced data with a quality_score score not less than 0.3 is stored in the distributed database.

[0111] When the crowdsourcing data is in the three-dimensional rendering implementation stage, the 130-dimensional key frame features recorded in the data_keyframe_feature field of the uploaded crowdsourcing data are matched with the data_keyframe_feature field features of the data stored in the distributed database (i.e., the distributed computing storage node 23) for similarity. Here, L2 similarity, i.e., Euclidean distance, is used for measurement. If the similarity is greater than 0.6, this crowdsourcing data is not stored; otherwise, this crowdsourcing data is stored.

[0112] In this embodiment, the data storage module 63 is further configured to update the quality_score value to the data_score field accordingly.

[0113] In addition, for stored data, the image sequence optimized by the image processing module 62 will be copied to the data_frames field, overwriting the original unoptimized image sequence image, and the data_id and data_location fields remain unchanged; the key frame sequence number is recorded in the data_keyframe_id field; the key frame deep learning features are recorded in the data_keyframe_feature field; and the relative pose information of the image sequence is recorded in the data_frame_pose field.

[0114] In the cloud-based three-dimensional reconstruction system, the optimized image sequence, posture information, and longitude and latitude position information collected and uploaded by the cloud-based distributed image data processing system 241 are used to train the incremental generation of neural radiation field models. The incremental generation of neural radiation field models is also an end-to-end deep learning model that can predict the radiation field distribution at the camera's location based on the calculated posture information and modeling data, and can generate three-dimensional radiation field distributions for missing field angles. The model shares the same feature extraction network and is accompanied by three branches: depth estimation, neural radiation field prediction, and posture regression. The deep learning features of the three branches are then spliced ​​and input into the multi-field angle radiation field generation module, and the neural radiation field for the full field angle is generated in combination with the position information.

[0115] In the cloud-based 3D reconstruction system, the end-to-end deep learning model architecture includes the following five components:

[0116] 1) Backbone network, used for basic feature extraction;

[0117] 2) Neural Radiance Field Component, which fits the basic neural radiance field based on deep learning features extracted from the backbone network;

[0118] 3) Depth estimation component, which estimates the distance of the camera medium from the reconstructed subject from the image based on the deep learning features extracted by the backbone network;

[0119] 4) Pose regression component, based on the deep learning features extracted by the backbone network and the image sequence pose data output by the cloud-based distributed image data processing system, regresses the pose of feature points in the image;

[0120] 5) Multi-viewing angle neural radiation field generation, based on generative neural networks, and combining the outputs of the neural radiation field component, depth estimation component, and pose regression component, to generate a three-dimensional neural radiation field with missing view angles.

[0121] The cloud-based three-dimensional reconstruction system of the embodiment of the present disclosure is an incremental generation of neural radiation fields. Unlike traditional neural radiation field methods, the incremental generation of neural radiation fields in the embodiment of the present disclosure not only has the function of three-dimensional reconstruction of neural radiation field models, but can also generate field of view angles that cannot be captured by training data, so that the three-dimensional reconstructed model can be displayed with a full 360-degree field of view.

[0122] Then, the weighted calculation is performed based on the score quality_score in the cloud-based distributed image data processing system to update the weighted score data_score of the crowdsourcing data.

[0123] The incremental training implemented by the incremental generation of neural radiation field model is to sort the data_score field scores from large to small in the distributed database, select the 100 crowdsourced data with the highest scores to train the incremental generation of neural radiation field. After the training, the softmax component is used to score the weights of the 100 training data in the model to obtain nerf_score, and then combine it with the score quality_score in the cloud-based distributed image data processing system for weighted calculation to update the weighted score data_score of the crowdsourced data.

[0124] The disclosed embodiments also provide a 3D scene modeling method that uses distributed computing and deep learning methods to perform location segmentation, image data processing, feature extraction, and matching on crowdsourced uploaded data, thereby achieving 3D reconstruction of the modeled object. This method, which combines crowdsourcing, cloud computing, computer vision, and deep learning technologies, boasts high efficiency, real-time performance, and low cost, and is widely applicable to industries, urban landscape display, transportation planning, cultural heritage preservation, entertainment, and other fields.

[0125] In the embodiment of the present disclosure, the cloud-based distributed image data processing system 241 is a pre-process of the cloud-based 3D reconstruction system 242 , and the cloud-based distributed image data processing system 241 provides training data guarantee for the cloud-based 3D reconstruction system 242 .

[0126] In this embodiment, full parameterization during the 3D reconstruction training phase and sparse parameterization during the 3D rendering implementation phase are achieved using reparameterization techniques. This incremental radiation field model generation system boasts efficient and high-quality image data processing capabilities, enabling complete and accurate reconstruction and real-time rendering of large-scale 3D scenes.

[0127] In one embodiment, the three-dimensional scene modeling method is implemented in two stages: a three-dimensional reconstruction training stage and a three-dimensional rendering implementation stage. The three-dimensional reconstruction training stage is implemented first, and the three-dimensional rendering implementation stage is implemented later. The three-dimensional modeling of the modeling subject needs to be completed in the three-dimensional reconstruction training stage before the rendered object can be viewed in the three-dimensional rendering implementation stage. In the three-dimensional reconstruction training stage, the smart mobile terminal is responsible for data collection and crowdsourcing upload, while the cloud is responsible for data area division, distributed computing node distribution, image sequence optimization, and incremental generation of neural radiation field model training. In the three-dimensional rendering implementation stage, the smart mobile terminal is responsible for data collection and upload, as well as receiving and displaying the rendering model from the distributed cloud. The cloud is responsible for querying, feature matching, pose calculation, three-dimensional rendering, and issuing the rendering model for the data uploaded by the smart mobile terminal.

[0128] The smart mobile terminal data acquisition, cloud-based distributed image data processing system, and cloud-based 3D reconstruction system described in this disclosure follow a sequential relationship throughout the entire process. First, the smart mobile terminal data acquisition process begins, uploading crowdsourced data to the cloud. The cloud-based distributed image data processing system then processes the raw data uploaded by the crowdsourced system to generate optimized image sequences and pose information. Finally, the cloud-based 3D reconstruction system incrementally generates a neural radiation field model during the 3D reconstruction training phase. During the 3D rendering implementation phase, the corresponding rendering model is generated and distributed to the smart mobile terminal.

[0129] 1) 3D reconstruction training stage.

[0130] During the 3D reconstruction training phase, smart mobile terminals are responsible for data collection and crowdsourcing data uploading, while the cloud is responsible for data area division, distributed computing node distribution, image sequence optimization, and incremental generation of neural radiation field model training.

[0131] FIG8 is a flow chart of the 3D reconstruction training phase according to an embodiment of the present disclosure. As shown in FIG8 , the process includes the following steps:

[0132] In step S801, the smart mobile terminal uploads the collected crowdsourced data to the cloud-based distributed image data processing system. Each piece of crowdsourced data includes an image sequence of multiple continuous RGB images, corresponding latitude and longitude location information, and an identifier of the crowdsourced data.

[0133] In an exemplary embodiment, one or more smart mobile terminals collect and upload crowdsourced data, and each smart mobile terminal uploads one or more pieces of crowdsourced data.

[0134] Step S802 , determining the data_phase field of the uploaded crowd-sourced data to determine which phase the crowd-sourced data is in. For example, data_phase 0 indicates the 3D reconstruction training phase, and data_phase 1 indicates the 3D rendering implementation phase.

[0135] Step S803 , adding the following five fields to the uploaded crowdsourcing data: data_id, data_score, data_keyframe_id, data_keyframe_feature, and data_frame_pose.

[0136] Step S804 , extracting the data_location field of the crowdsourced data, referring to the mapping table between the modeling sub-areas and the distributed computing nodes, and sending the crowdsourced data to the corresponding distributed computing nodes.

[0137] Step S805: The end-to-end deep learning graphics processing model receives the image sequence set and corresponding position information sent by the distributed computing node, performs image data processing on the image sequence set, and obtains an optimized image sequence set and relative pose information corresponding to the optimized image sequence set.

[0138] Among them, the image data processing process includes: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, extra-feature matching, loop detection and posture backend optimization.

[0139] Step S806 , when the crowdsourced data belongs to the 3D reconstruction training stage, the score stored in the data_score field of the crowdsourced data is determined, and the crowdsourced data with a score not less than a preset threshold is stored in the distributed database.

[0140] The score in step S806 is equivalent to the first score in the above embodiment.

[0141] During the 3D reconstruction training phase, incremental iterative training is performed based on the data_score of the training data. As more high-quality crowdsourced data becomes available, the model becomes more accurate and complete. The end-to-end model uses a full weight parameter model for model fitting.

[0142] In an exemplary embodiment, the preset threshold is set to 0.3.

[0143] Step S807: Design an incremental generation neural radiation field model.

[0144] The incremental generative neural radiation field model is a model that can predict the radiation field distribution at the camera's location with full field of view.

[0145] Step S808, using the image sequence set optimized in step S805 as a training set, the image data, relative posture information, and longitude and latitude position information collected and uploaded by the cloud-based distributed image data processing system are input into the designed incremental generation of a neural radiation field model. After the deep learning features of the three branches of neural radiation field distribution, depth estimation, and posture regression in the designed incremental generation of the neural radiation field model are spliced, the neural radiation field model is output through a multi-field-of-view neural radiation field generation network.

[0146] Step S809 continuously adjusts the weights and biases of the neural network through the back propagation algorithm to minimize the error between the generated neural radiation field and the actual radiation field.

[0147] Step S810: After the model converges, the parameter weight information is used, and after softmax, the score information is multiplied by the score of the image sequence set during image quality judgment to obtain the final score information;

[0148] In step S811 , during incremental training, the image sequence set is sorted according to its score information, and only the top 10 image sequences are selected to participate in fine-tuning training.

[0149] Among them, step S804 corresponds to the data distribution module 61 in the above embodiment, step S805 corresponds to the image processing module 62 in the above embodiment, and step S806 corresponds to the data storage module 63 in the above embodiment.

[0150] During the 3D reconstruction training phase, the end-to-end deep learning model system applies the full feature weights of the deep neural network for model forward reasoning and gradient backpropagation, ensuring that the trained 3D model architecture is accurate, the texture is clear, and the lighting is natural.

[0151] 2) 3D rendering implementation stage.

[0152] In the 3D rendering implementation stage, the smart mobile terminal is responsible for data collection, crowdsourcing data uploading, and receiving and displaying the rendering model from the distributed cloud. The cloud is responsible for querying, feature matching, pose calculation, 3D rendering, and sending the rendering model for the crowdsourcing data uploaded by the smart mobile terminal.

[0153] FIG9 is a flow chart of a three-dimensional rendering implementation stage according to an embodiment of the present disclosure. As shown in FIG9 , the flow includes the following steps:

[0154] In step S901, the smart mobile terminal uploads the collected crowdsourced data to the cloud-based distributed image data processing system, wherein each piece of crowdsourced data includes an image sequence of multiple continuous RGB images, corresponding latitude and longitude location information, and an identifier of the crowdsourced data.

[0155] Step S902 , determining the data_phase field of the uploaded crowdsourcing data to determine which stage the crowdsourcing data is in. For example, data_phase 0 indicates the 3D reconstruction training stage, and data_phase 1 indicates the 3D rendering implementation stage.

[0156] Step S903 , add the following five fields to the uploaded crowdsourcing data: data_id, data_score, data_keyframe_id, data_keyframe_feature, and data_frame_pose.

[0157] Step S904 , extracting the data_location field of the crowdsourced data, referring to the mapping table between the modeling sub-areas and the distributed computing nodes, and sending the crowdsourced data to the corresponding distributed computing nodes.

[0158] Step S905: The end-to-end deep learning graphics processing model receives the image sequence set and corresponding position information sent by the distributed computing node, performs image data processing on the image sequence set, and obtains an optimized image sequence set and relative pose information corresponding to the optimized image sequence set.

[0159] Among them, the image data processing process includes: image quality judgment, image denoising, image super-resolution, image semantic segmentation, image generation, descriptor feature extraction, intra-feature matching, fine-grained feature extraction, extra-feature matching, loop detection and posture backend optimization.

[0160] During the 3D rendering implementation phase, the smart mobile terminal captures the RGB image and latitude and longitude information of the desired rendered object and uploads it to the cloud, requesting the 3D rendering model. The cloud-based distributed image data processing system extracts features from the image to obtain the 130-dimensional feature vector data_keyframe_feature and relative pose data_frame_pose of the key frames in the image sequence. The corresponding incremental generation neural radiation field model is then called for forward reasoning, outputting a full-field-of-view rendering model and sending it to the smart terminal, enabling real-time rendering of the modeled object on the smart mobile device. The specific steps include:

[0161] Step S906: When the crowdsourced data belongs to the 3D rendering implementation stage, similarity matching is performed between the 130-dimensional keyframe features of the data_keyframe_feature record of the uploaded crowdsourced data and the data_keyframe_feature field features of the data stored in the distributed database, and crowdsourced data with a similarity greater than a preset threshold is stored.

[0162] In one embodiment, if the similarity between all key frames and the uploaded RGB image is below 0.6, it means that no suitable 3D rendering model has been found, and the user can be guided to collect crowdsourced data through a smart mobile terminal.

[0163] In an exemplary embodiment, L2 similarity, that is, Euclidean distance, is used for measurement. If the similarity is greater than 0.6, the crowd-sourced data is not stored; otherwise, the crowd-sourced data is stored.

[0164] During the 3D rendering implementation stage, the end-to-end model uses a sparse weight parameter model for forward model reasoning, which can achieve efficient and fast calculations while better handling complex light propagation situations. It can also adaptively adapt to different light sources and material properties, complete missing perspectives, and improve the efficiency and accuracy of reconstruction and rendering.

[0165] Step S907: Obtain crowdsourced data from distributed data.

[0166] Step S908, obtaining a rendering model: using the optimized image sequence, relative pose, and longitude and latitude position information uploaded by crowdsourcing as input, and using the corresponding incrementally generated neural radiation field model for inference, a 3D rendering model with a full field of view is obtained;

[0167] Step S909, rendering and presentation: the 3D rendering model obtained by cloud-based reasoning is sent to the requesting user's smart mobile device for viewing.

[0168] Among them, step S904 corresponds to the data distribution module 61 in the above embodiment, step S905 corresponds to the image processing module 62 in the above embodiment, and step S906 corresponds to the data storage module 63 in the above embodiment.

[0169] During the 3D rendering implementation stage, the model of the incrementally generated radiation field model system only needs to use a sparse weight parameter mode for model forward reasoning, which can achieve efficient and fast calculations while better handling complex light propagation situations, ensuring the real-time performance of 3D rendering.

[0170] Through the above steps, data collection for 3D reconstruction and rendering using a smart mobile device is portable and flexible. Traditional 3D reconstruction and rendering require high-end equipment and complex software, and often require transferring the data to a computer for processing. In contrast, the disclosed embodiment uses the RGB camera of a smart mobile device for data collection and uploads the data to the cloud for processing, eliminating the need for high device performance and making data processing more flexible and convenient.

[0171] The above-mentioned embodiments of the present disclosure can be applied to smart mobile terminals and cloud-based distributed computing and storage nodes.

[0172] The smart mobile terminal is one of the important application media of the embodiments of the present disclosure and is the entry and exit for data uploading and rendering model reception. The smart mobile terminal here can be a mobile device such as a smartphone, tablet computer, portable computer, etc. that has a camera, computing power, and can connect to the Internet. These devices have a wide range of application scenarios and are popular devices that are very portable. Their advantage is that they can realize the crowdsourcing collection and uploading of data required for 3D reconstruction and the presentation of 3D rendering models in a simple and easy-to-use manner, providing a more efficient and convenient operating experience and significantly reducing the professional threshold for 3D modeling data collection.

[0173] Another application medium is distributed cloud computing and storage resources. The cloud system of the disclosed embodiment is divided into a cloud-based distributed image data processing system and a cloud-based distributed image data processing system. The two end-to-end deep learning models included therein require powerful computing resources, and cloud computing resources play a vital role in the reconstruction and rendering processes. Leveraging the advantages of cloud computing, we can fully utilize and share computing resources, improve computing efficiency, and reduce costs. At the same time, the cloud's data storage and management functions also facilitate large-scale data management and sharing.

[0174] Therefore, the application environment of the present disclosure is smart mobile devices and cloud computing and storage resources. Such an application environment can give full play to the advantages of the present disclosure and realize efficient, convenient, high-quality 3D reconstruction and real-time rendering functions.

[0175] The following further illustrates the above-mentioned method using a specific scenario example, namely, 3D modeling of urban traffic street scenes. Specifically, 3D modeling and rendering of specific areas of urban traffic street scenes are performed for map navigation reference, urban landscape display, traffic planning, and the creation of digital sand tables for urban areas.

[0176] Step S1, preparation.

[0177] Before starting 3D modeling, some preparation is necessary. First, you need to determine the target area for modeling, such as a specific area of ​​a city, such as a commercial center or tourist attraction. Second, you need to call on citizens (users) and provide the minimum mobile device configuration required to participate in crowdsourced data collection. These devices should be able to capture high-quality photos and videos. Finally, you need to release a crowdsourced data collection and 3D rendering app and provide simple and convenient operation instructions so that users can quickly master the skills of taking photos or videos and uploading data to the cloud.

[0178] Step S2: capturing a multi-view image sequence.

[0179] Open the app and use your smart mobile device to take 10 consecutive photos in the target area. These photos should cover the entire target area and capture as many objects as possible, such as buildings, streets, vehicles, and pedestrians. Try to ensure correct exposure and clear images.

[0180] Step S3: upload data to the cloud.

[0181] By uploading photos to the cloud through the app and obtaining location information, crowdsourced data is generated. This data is used for subsequent feature extraction and 3D modeling. The use of crowdsourcing to upload data allows multiple people to upload their own photos, covering a wider area and including information about modeled objects under different lighting conditions, improving data quality and reliability.

[0182] Step S4: cloud image data processing.

[0183] In the cloud, the crowdsourced image data is divided by location region using a distributed node mapping table constructed using the cloud preprocessing process. Image sequences within the same location region form an image sequence set, which is then sent to the corresponding distributed computing node. This is then processed by the cloud-based distributed image processing module and data storage module. The image processing module's end-to-end deep learning image processing model performs image data processing, feature extraction, matching, and pose calculation on the image sequence set, ultimately generating an optimized image sequence set and relative pose information. The data storage module determines whether to store the crowdsourced data based on the data_score obtained by the image processing module.

[0184] Step S5: neural radiation field model.

[0185] In the same distributed computing node, after the crowdsourced data has been processed by the cloud image data, the optimized image sequence set, relative pose information, and longitude and latitude position information of the crowdsourced data will be used as input for processing by the cloud-based 3D reconstruction system. During the 3D reconstruction training phase, the cloud-based 3D reconstruction system trains an incremental neural radiation field model using full parameters for training. The model should be able to map image data into a 3D world coordinate system, generate a 3D neural radiation field model, and automatically complete the neural radiation field information for missing field of view angles. During the training process, incremental training can be achieved based on the scores of the image sequence, and the accuracy of the neural radiation field model can be continuously evolved. During the 3D rendering implementation phase, the cloud-based 3D reconstruction system uses the trained 3D neural radiation field model and sparse parameters for inference to improve the efficiency of 3D rendering model generation.

[0186] Step S6: Rendering the street view model

[0187] Through the above steps, the street view data has been successfully converted into a 3D model. In the 3D rendering implementation stage, the 3D rendering model sent from the cloud can be received on a smart mobile device and viewed on the terminal screen for user interaction and browsing.

[0188] During rendering, you can set different rendering parameters according to user needs, such as scene lighting, shadows, materials, etc., to meet the needs of different users. At the same time, you can also combine AR technology to integrate street view models into the real environment to provide a more immersive experience.

[0189] Through the above-mentioned embodiments of the present disclosure, the incremental generation of neural radiation field models is used for three-dimensional modeling and real-time rendering, which improves the rendering efficiency and quality, and realizes full-field-angle rendering of the modeled object: In the embodiments of the present disclosure, the neural radiation field model is trained using optimized image data and deployed in the cloud to realize real-time rendering of the scene. Compared with the traditional ray tracing-based rendering algorithm, the model in the embodiment of the present disclosure has higher rendering speed and rendering quality, and can provide more realistic rendering effects while ensuring real-time performance. In addition, the neural radiation field model of the present disclosure can realize incremental training, and the model will continue to evolve and become more and more accurate as high-quality crowdsourcing data is uploaded.

[0190] The 3D scene modeling method and cloud-based system proposed in the embodiments of this disclosure offer significant advantages, including low cost, high precision, high efficiency, and ease of use. By utilizing cloud-based distributed computing, it can simultaneously process large amounts of image data and achieve compatibility with diverse devices, improving the system's convenience and universality. This method effectively addresses the challenges inherent in traditional methods and possesses broad application prospects and market value. It can be widely applied in areas such as augmented reality, cultural heritage preservation, and digital cities.

[0191] The smart mobile terminal used in the embodiments of the present disclosure is a popular device, which is very portable and has a wide range of application scenarios. Its advantage is that it can realize the crowdsourcing collection and uploading of data required for three-dimensional reconstruction and the presentation of three-dimensional rendering models in a simple and easy-to-use way, providing a more efficient and convenient operation experience, and greatly reducing the professional threshold for three-dimensional modeling data collection. Distributed cloud computing and storage nodes, cloud computing nodes play a vital role in the reconstruction and rendering process. In the embodiments of the present disclosure, the advantages of cloud computing are utilized to give full play to the efficient use and sharing of computing resources, improve computing efficiency and reduce costs. At the same time, the data storage and management functions of the cloud can also facilitate large-scale data management and sharing.

[0192] An embodiment of the present disclosure further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0193] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0194] An embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0195] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0196] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0197] Obviously, those skilled in the art should understand that the modules or steps of the present disclosure described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any particular combination of hardware and software.

[0198] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations of the present disclosure are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A three-dimensional scene modeling method, applied to a cloud system, comprising: Based on multiple crowdsourced data collected and uploaded by the user terminal, three-dimensional scene modeling is performed on the scene to be modeled by incrementally generating a neural radiation field model, so as to obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

2. The method according to claim 1, wherein: Based on multiple crowdsourced data collected and uploaded by user terminals, a neural radiation field model is incrementally generated in the cloud system to perform three-dimensional scene modeling on the scene to be modeled, including: The first cloud system performs image data processing on the plurality of crowdsourced data to obtain an optimized first image sequence set and relative position information of each of the crowdsourced data; wherein one of the crowdsourced data corresponds to a first image sequence in the first image sequence set; The first cloud system updates the information stored in the corresponding fields of the plurality of crowd-sourced data based on the first image sequence, the relative posture information, and the first position information corresponding to each of the crowd-sourced data, and inputs the updated plurality of crowd-sourced data into the second cloud system; The second cloud system performs three-dimensional scene modeling by incrementally generating a neural radiation field model based on the updated plurality of crowd-sourced data, thereby obtaining a three-dimensional rendering model of the scene to be modeled with a full field of view.

3. The method according to claim 2, wherein: The first cloud system performs image data processing on the plurality of crowdsourced data, including: According to the first location information and based on the mapping relationship table between the modeling sub-areas and the distributed computing nodes, the plurality of crowdsourced data are sent to the corresponding distributed computing nodes; wherein the modeling sub-areas are obtained by dividing the area corresponding to the scene to be modeled by the first cloud system according to the prior digital map, and one modeling sub-area corresponds to one first location information; The first cloud system receives a second image sequence set returned by the distributed computing node, and inputs the second image sequence set into an end-to-end deep learning image processing model to perform the image data processing, wherein each image sequence in the second image sequence set corresponds to one of the first position information.

4. The method according to claim 3, wherein: The first cloud system processes the plurality of crowdsourced data to obtain an optimized first image sequence set, including: For any second image sequence in the second image sequence set, the first cloud system extracts the mask of the dynamic object in each image in the second image sequence through the image semantic segmentation, and maps the mask back to the feature map corresponding to the image; wherein the dynamic object includes: pedestrians, animals, vehicles, and movable still objects related to the pedestrians; Based on the contextual relationship of the mask in the feature map and the feature relationship between the previous and next frames of the second image sequence, background completion is performed on the image inside the mask through the image generation to obtain an optimized image sequence; Through the above step of optimizing the image sequence, all the second image sequences in the second image sequence set are optimized to obtain the first image sequence set.

5. The method according to claim 2, wherein: The first cloud system performs image data processing on the plurality of crowdsourced data to obtain relative position information of each of the crowdsourced data, including: For any of the first image sequences, extracting descriptor information of each image in the first image sequence through the descriptor feature extraction and the intra-feature matching, and setting the image with the most common view descriptors as a key frame of the first image sequence; The fine-grained feature extraction is to extract fine-grained features from the key frame to obtain deep features, wherein: The deep features include fine-grained representation features and fine-grained semantic features; Through the feature external matching, the first position information corresponding to the first image sequence is feature spliced ​​with the depth feature to obtain the key frame feature of the first image sequence; Through the feature external matching, the loop closure detection and the back-end pose optimization, feature matching, loop closure detection and graph optimization are performed on the deep features in the key frame features of the first image sequence, and based on the corresponding relationship between each of the first image sequences and the plurality of crowdsourced data, the relative pose information of the crowdsourced data corresponding to the first image sequence is obtained; Through the above step of obtaining relative posture information, the relative posture information of other crowdsourcing data is obtained.

6. The method according to claim 2, wherein: Before inputting the updated plurality of crowd-sourced data into the second cloud system, the method further includes: For any of the crowd-sourced data, judging the stage of the crowd-sourced data according to the stage identification field of the crowd-sourced data; For the first crowdsourced data in the 3D reconstruction training stage, storing the first crowdsourced data having a first score greater than or equal to a first preset threshold into a distributed database; wherein the first score is obtained by the cloud system scoring the image quality of the image sequence corresponding to the first crowdsourced data through an image quality judgment component; For the second crowd-sourced data in the three-dimensional rendering implementation stage, the key frame features of the second crowd-sourced data are matched with the corresponding key frame features stored in the distributed database for similarity, and the second crowd-sourced data with a similarity less than a second preset threshold is stored in the distributed database.

7. The method according to claim 6, wherein: Before performing three-dimensional scene modeling by incrementally generating a neural radiation field model, the method further includes: The second cloud system obtains a plurality of the first crowd-sourced data from a distributed database, and sorts the plurality of the first crowd-sourced data in descending order based on the first scores of the first crowd-sourced data; Inputting the first N first crowdsourced data, the relative posture information of the first N first crowdsourced data, and the first position information corresponding to the first N first crowdsourced data into the pre-designed incremental generation of neural radiation field model to obtain the trained incremental generation of neural radiation field model, wherein N is an integer greater than zero; Adjusting the weights and biases of the neural network in the trained incrementally generated neural radiation field model so that the incrementally generated neural radiation field model converges; When the incremental generation of the neural radiation field model completes convergence, based on the weighted scores of the first N first crowd-sourced data, the multiple first crowd-sourced data are sorted in order from large to small, and the first M first crowd-sourced data are selected to perform fine-tuning training on the incremental generation of the neural radiation field model, where M is an integer greater than zero and less than N.

8. A cloud system comprising: The modeling module is configured to perform three-dimensional scene modeling on the scene to be modeled based on multiple crowdsourced data collected and uploaded by the user terminal by incrementally generating a neural radiation field model, and obtain a three-dimensional rendering model of the scene to be modeled with a full field of view.

9. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images