Image processing method and device and related equipment

Abstract

The embodiment of the invention discloses an image processing method and device and related equipment, and belongs to the technical field of computer vision. In the embodiment of the invention, a visual object geometric model and a subsequent object NeRF model capable of being implicitly rendered are formed at the same time through a sample data set. A user edits display attributes such as display postures of the to-be-simulated object through the visual object geometric model, so that high editability of visual simulation is realized. And meanwhile, when the visual simulation result is rendered, the image can be rendered directly through the object NeRF model, so that the high trueness of the visual simulation result is realized, and the resource consumption in the rendering process is low.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to an image processing method, apparatus, and related equipment. Background Art

[0002] Image processing in visual simulation technology can be understood as: generating virtual scene images close to real scenes through sample data such as collected sample images. For example, in the field of autonomous driving, visual simulation technology can be used to generate virtual scene images used to simulate the environment around autonomous driving vehicles, thereby providing more data for the autonomous driving algorithms of autonomous driving vehicles.

[0003] In the related art, in order to edit the foreground in the visual simulation process, the foreground in the image is explicitly rendered using computer graphics (CG) technology. However, the process of displaying and rendering the foreground using CG technology requires manual operation by professional manpower and tool software, and requires a lot of time and hardware resources to improve the realism of the foreground. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus and related equipment, which can not only achieve high editability of the visual simulation process but also reduce resource consumption while ensuring high fidelity of the visual simulation results. The technical solution is as follows:

[0005] In a first aspect, an image processing method is provided. In the method, a sample data set is obtained, the sample data set includes a plurality of first sample data, the plurality of first sample data are used to indicate the imaging information of a first object from multiple perspectives; a first object neural radiation field NeRF model and a first object geometric model are determined based on the plurality of first sample data, the first object NeRF model is used to predict an arbitrary perspective image of the first object, and the first object geometric model is a three-dimensional model of the first object; the first object geometric model is displayed on a front-end interactive interface; in response to an editing operation of a user on the front-end interactive interface, the display properties and rendering perspective pose of the first object geometric model are determined, the display properties including the display pose; at least one first object image is determined based on the display properties of the first object geometric model, the rendering perspective pose, and the first object NeRF model.

[0006] In the embodiment of the present application, a visualized object geometry model and an object NeRF model that can be implicitly rendered later are simultaneously formed through a sample data set. The user edits the display properties of the object to be simulated, such as the display posture, through the visualized object geometry model, thereby achieving high editability of the visual simulation. At the same time, when rendering the visual simulation results, the image can be directly rendered through the object NeRF model, thereby achieving high fidelity of the visual simulation results and low resource consumption in the rendering process.

[0007] Based on the method provided in the first aspect, in a possible implementation manner, the implementation process of determining at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model can be: based on the display properties of the first object geometric model and the rendering perspective pose, determine the first rendering pose in a first coordinate system, the first coordinate system is the object coordinate system of the first object geometric model, and the rendering perspective pose is the pose in the world coordinate system; based on the first rendering pose and the NeRF model of the first object, determine at least one first object image.

[0008] The viewing pose required by the NeRF model to render the image is the viewing pose in the object coordinate system of the geometric model to be rendered, while the viewing pose edited by the user is usually the viewing pose in the world coordinate system. Therefore, after obtaining the rendering viewing pose edited by the user, the rendering viewing pose needs to be converted into the first rendering pose in the first coordinate system, that is, converted into the first rendering pose aligned with the NeRF model of the first object.

[0009] Based on the method provided in the first aspect, in a possible implementation, the rendering perspective pose includes multiple perspective poses, and the first rendering pose includes multiple rendering poses. In this scenario, the implementation process of determining at least one first object image based on the first rendering pose and the NeRF model of the first object may be: determining a first rendering trajectory based on the multiple rendering poses; determining multiple first object images based on the first rendering trajectory and the NeRF model of the first object.

[0010] Through the above method, the user can only edit several key viewpoint poses, and a rendering trajectory can be formed based on the several viewpoint poses edited by the user.

[0011] Based on the method provided in the first aspect, in a possible implementation manner, the implementation process of determining the first rendering trajectory based on the first rendering pose may be: determining the first rendering trajectory through an interpolation algorithm based on multiple rendering poses.

[0012] In the embodiment of the present application, more rendering poses can be obtained through an interpolation algorithm to form a rendering trajectory.

[0013] Based on the method provided in the first aspect, in a possible implementation, the sample data set also includes a plurality of second sample data, and the plurality of second sample data are used to indicate the imaging information of the second object from multiple perspectives. In this scenario, in the method, the second object NeRF model and the second object geometric model can also be determined based on the plurality of second sample data, the second object NeRF model is used to predict the arbitrary perspective image of the second object, and the second object geometric model is the three-dimensional model of the second object; the second object geometric model is displayed on the front-end interactive interface; the display properties of the second object are determined in response to the user's editing operation on the front-end interactive interface; at least one second object image is determined based on the display properties of the second object, the rendering perspective posture and the NeRF model of the second object; at least one first object image and at least one second object image are fused to obtain panoramic simulation data.

[0014] In the embodiment of the present application, the above method can also be used to perform visual simulation on a scene including multiple objects.

[0015] Based on the method provided in the first aspect, in a possible implementation, at least one first object image and at least one second object image correspond one-to-one to the same rendering perspective pose in at least one rendering perspective pose. In this scenario, the implementation process of fusing at least one first object image and at least one second object image to obtain panoramic simulation data can be: obtaining a first object image and a second object image corresponding to the first object image from at least one first object image and at least one second object image, the first object image includes a first object red, green, and blue (RGB) image and a first object depth image, and the second object image includes a second object RGB image and a second object depth image; based on the first object depth image and the second object depth image, the first object RGB image and the second object RGB image are fused into a panoramic RGB image.

[0016] Based on the first object depth image and the second object depth image, the occlusion relationship between the first object and the second object at the same viewpoint can be determined, and then based on the occlusion relationship, it is determined whether the pixel value displayed at the viewpoint in the panoramic RGB image is the first object RGB image or the second object scene RGB image, thereby improving the realism of the visual simulation result.

[0017] Based on the method provided in the first aspect, in a possible implementation, the first object is a foreground object and the second object is a background object. In this scenario, user-configured object type information may also be received, the object type information being used to indicate that the first object is a foreground object and the second object is a background object.

[0018] In the embodiment of the present application, the user may edit which objects are foreground objects and which objects are background objects.

[0019] Based on the method provided in the first aspect, in a possible implementation, the display attribute also includes a display size. In this scenario, in response to the user's editing operation on the front-end interactive interface, the implementation process of determining the display attribute of the first object geometric model may be: in response to the user's movement and / or rotation operation on the first object geometric model, determining the display posture of the first object geometric model; in response to the user's scaling operation on the first object geometric model, determining the display size of the first object geometric model.

[0020] In the embodiment of the present application, the user can move and / or rotate the first object geometric model on the front-end interactive interface, thereby editing the display posture of the first object geometric model. The user can also scale the first object geometric model on the front-end interactive interface, thereby editing the display size of the first object geometric model.

[0021] Based on the method provided in the first aspect, in a possible implementation manner, in response to the user's editing operation on the front-end interactive interface, the implementation process of determining the rendering perspective posture can be: displaying a virtual rendering camera on the front-end interactive interface; and determining the rendering perspective posture in response to the user's movement and / or rotation operation of the virtual rendering camera.

[0022] In an embodiment of the present application, the user can move and / or rotate the virtual rendering camera on the front-end interactive interface, thereby editing the rendering perspective posture.

[0023] Based on the method provided in the first aspect, in a possible implementation manner, the first object geometric model includes a first object mesh model.

[0024] In the embodiment of the present application, the user can edit the visual simulation requirements based on the visualized first object mesh model.

[0025] In a second aspect, an image processing device is provided, which has the function of implementing the image processing method in the first aspect. The image processing device includes at least one module, and the at least one module is used to implement the image processing method provided in the first aspect.

[0026] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the image processing method provided in the first aspect above.

[0027] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the image processing method provided in the first aspect.

[0028] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the image processing method provided in the first aspect.

[0029] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of an image processing system provided in an embodiment of the present application;

[0031] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;

[0032] Figure 3 is a schematic diagram of a front-end interactive interface provided in an embodiment of the present application;

[0033] Figure 4 is a schematic diagram of another front-end interactive interface provided in an embodiment of the present application;

[0034] Figure 5 is a schematic diagram of another front-end interactive interface provided in an embodiment of the present application;

[0035] Figure 6 is another image processing flow diagram provided in an embodiment of the present application;

[0036] Figure 7 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0037] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0038] Fig. 9 It is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0039] Fig.10 It is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application more clear, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] To facilitate subsequent understanding, the application scenarios of the embodiments of the present application are first explained below.

[0042] Visual simulation technology has always been one of the key research technologies in the fields of computer vision and computer graphics. Highly realistic visual simulation technology can generate sufficient data to train visual perception systems with better capabilities. In recent years, as applications such as autonomous driving and robots have gradually moved from laboratories to practical use, the requirements for visual simulation technology have become increasingly higher. Not only is visual simulation technology required to have a higher degree of realism, but it is also required that the generation of these data does not consume too much time, manpower, equipment and other costs.

[0043] Several visual simulation technologies are explained below by way of example.

[0044] (1) CG-based visual simulation technology.

[0045] CG-based visual simulation technology mainly includes three steps:

[0046] First: Preparation and production of 3D materials. Since models generated directly through 3D reconstruction technology are often difficult to use directly, in CG-based visual simulation technology, 3D materials often require professional modelers to use professional software to manually create or modify them, and then specially produce corresponding materials to achieve better visual effects.

[0047] Second: Combination editing of 3D materials. The 3D materials created in the first step are usually single and need to be combined in a 3D scene. This step usually requires the participation of professional modelers, who use professional software to adjust the position of the 3D materials, scene lighting, etc.

[0048] Third: 3D rendering. For example, 3D rendering can be performed by using physically-based rendering (PBR) technology.

[0049] For example, many autonomous driving simulation platforms use CG-based visual simulation technology to simulate vehicle driving scenarios.

[0050] Currently, CG-based visual simulation technology has gradually been able to produce relatively realistic simulation effects, but overall there is still a certain gap from photo-realistic effects. In addition, the more realistic the rendering, the greater the consumption of hardware and time, which is inconsistent with the demand for cost control during simulation.

[0051] Based on the above content, we can see that the advantages of CG-based visual simulation technology are mature technology and strong editability. However, the problem is that it is difficult to meet the requirements of high realism and low resource consumption at the same time.

[0052] (2) Visual simulation technology based on NeRF model.

[0053] Different from the CG-based visual simulation technology that expresses 3D models in the form of explicit mesh models, the NeRF model-based visual simulation technology implicitly expresses the 3D model in the form of a neural network. Therefore, the images generated by the NeRF model-based visual simulation technology have photo-level realism.

[0054] In addition, the visual simulation technology based on the NeRF model does not require manual operation by professional modelers. It can directly obtain high-fidelity rendering results through end-to-end execution of the algorithm, which can greatly save labor costs. Moreover, the NeRF model itself does not require the participation of professional software or tools. It can be completely executed in the background, and only requires equipment that can complete neural network reasoning. However, the visual simulation technology based on CG sometimes requires better graphics card support in order to obtain higher performance, such as graphics cards that support ray tracing. Therefore, the visual simulation technology based on the NeRF model can greatly save hardware costs.

[0055] However, the visual simulation technology based on the NeRF model also has its own problems, namely, it does not have good editability, which means it is difficult to generate new scenarios through editing based on the existing NeRF model.

[0056] Based on this, an embodiment of the present application provides an image processing method, which can not only achieve high editability of the visual simulation process but also reduce resource consumption while ensuring high authenticity of the visual simulation results.

[0057] Among them, the image processing method provided in the embodiment of the present application can be applied to a cloud scenario, and the cloud scenario can be an autonomous driving scenario, etc. In this scenario, the image processing method is executed by a cloud platform, that is, the cloud platform provides cloud services to users, and the cloud service is used to implement the image processing method provided in the embodiment of the present application. Optionally, the image processing method provided in the embodiment of the present application can also be executed by a terminal device. The embodiment of the present application does not limit the application scenario of the image processing method.

[0058] The image processing system involved in the embodiment of the present application is explained below.

[0059] Figure 1 Schematic diagram of an image processing system provided in an embodiment of the present application. Figure 1 As shown, the system includes a data acquisition unit 10 and a visual simulation device 20. The visual simulation device includes a front-end interaction unit 201, an asset storage unit 202 and a calculation unit 203.

[0060] The data acquisition unit 10 and the visual simulation device 20 may be connected to each other via wired or wireless means for communication, and the various units in the visual simulation device 20 may also be connected to each other via wired or wireless means for communication.

[0061] The data acquisition unit 10 is used to collect sample data for subsequent visual simulation through the sample data. The sample data is used to indicate at least one object in a specific target scene, and the at least one object may include a foreground object and a background object. For example, the sample data can be a sample image collected for the target scene, or a sample point cloud collected for the target scene. The front-end interaction unit 201 is used for human-computer interaction to realize functions such as uploading sample data, visual editing, and displaying visual simulation results. The asset storage unit 202 is used to store sample data collected by the data acquisition unit 10 and information such as the trained NeRF model. The computing unit 203 is used to implement algorithm calculations to realize the image processing method provided in the embodiment of the present application.

[0062] The data collection unit 10 may be any device capable of collecting sample data. For example, in an autonomous driving scenario, the data collection unit 10 may be a camera and radar on an autonomous driving vehicle, or may be other sample data collection devices independent of the autonomous driving vehicle, such as a mobile phone, a road sampling vehicle, and a scanner.

[0063] The visual simulation device 20 can be a cloud platform. In this scenario, the visual simulation device 20 can be implemented by one or more computing devices. Among them, the front-end interaction unit 201, the asset storage unit 202 and the computing unit 203 can be integrated on the same computing device, or deployed on different computing devices, which is not limited in the embodiments of the present application.

[0064] The image processing method provided in the embodiment of the present application is explained below.

[0065] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application. The method can be applied to Figure 1 The visual simulation device shown in FIG. Figure 2 As shown, the method includes the following steps.

[0066] Step 201: Acquire a sample data set, where the sample data set includes a plurality of first sample data, and the plurality of first sample data are used to indicate imaging information of a first object at a plurality of viewing angles.

[0067] When performing visual simulation on the first object, it is necessary to first obtain a sample data set so as to render the first object at a desired viewing angle based on the sample data in the sample data set.

[0068] The first object may be any object in a scene. For example, the first object is a foreground object. In this scenario, the method provided by the embodiment of the present application can realize visual simulation of the foreground object in a scene.

[0069] In the embodiment of the present application, visual simulation can also be performed on multiple objects in a scene at the same time. In this scenario, the sample data set also includes multiple second sample data, and the multiple second sample data are used to indicate multiple viewing angle imaging information of the second object.

[0070] The first object and the second object may be any two objects in a scene. For example, the first object is a foreground object, and the second object is a background object. In this scenario, the method provided by the embodiment of the present application can simultaneously realize the visual simulation of the foreground object and the background object in a scene.

[0071] The following explanation is given by taking the first object as a foreground image and the second object as a background object as an example.

[0072] For example, in an autonomous driving scenario, foreground objects may include dynamic vehicles around the autonomous driving vehicle, and background objects may include the road where the autonomous driving vehicle is located, static buildings around the road, and the like.

[0073] In some embodiments, the sample data set includes a first sample data set and a second sample data set, the first sample data set includes a plurality of first sample data, and the second sample data set includes a plurality of second sample data.

[0074] In an embodiment of the present application, the first sample data for the foreground object and the second sample data for the background object can come from different data sources, so that the user can collect first sample data with a wider perspective and second sample data with a wider perspective based on needs to improve the realism of subsequent visual simulation results.

[0075] For example, in an autonomous driving scenario, first sample data of other vehicles on the road around the autonomous driving vehicle can be collected through a mobile phone or other camera device, and second sample data of the roads around the autonomous driving vehicle can be collected through vehicle-mounted sensors such as radars or cameras.

[0076] Subsequent users can Figure 1 The front-end interaction unit uploads the first sample data set and the second sample data set to the visual simulation device to implement the image processing method provided in the embodiment of the present application through the visual simulation device.

[0077] When the user uploads the first sample data set and the second sample data set to the visual simulation device through the front-end interaction unit, the user needs to indicate that the first sample data set is a sample data set for foreground objects and the second sample data set is a sample data set for background objects.

[0078] In some embodiments, the visual simulation device receives object category information configured by a user, where the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

[0079] Figure 3 Schematic diagram of an interface for user configuration object types provided in an embodiment of the present application. Figure 3 As shown, the front-end interaction unit provides a front-end interaction interface, and the front-end interaction interface displays a foreground object file import option and a background object file import option.

[0080] After the user clicks the foreground object file import option, the first sample data set is uploaded to the visual simulation device, and the visual simulation device subsequently confirms that the current user sets the first object corresponding to the first sample data set as the foreground object. Correspondingly, after the user clicks the background object file import option, the second sample data set is uploaded to the visual simulation device, and the visual simulation device subsequently confirms that the current user sets the second object corresponding to the second sample data set as the background object.

[0081] The above example is that the first sample data and the second sample data come from different data sources. Optionally, in other embodiments, the sample data in the sample data set may be collected by the same sample data collection device without distinguishing between the first sample data and the second sample data.

[0082] For example, in an autonomous driving scenario, sample data of the environment around the autonomous driving vehicle (including background such as roads and foreground such as other dynamic vehicles) can be collected by on-board sensors such as radar or cameras. In this scenario, the foreground objects and background objects can be separated from the sample data by image recognition algorithms to facilitate the subsequent training of different NeRF models.

[0083] In addition, the sample data in the sample data set may include a sample image or a sample point cloud, and the pose of the device that collects the sample data. By way of example, the sample data set includes a plurality of sample images and a camera pose corresponding to each sample image. The camera pose corresponding to the sample image can be understood as the pose of the camera that collects the sample image. By way of another example, the sample data set includes a plurality of point cloud data and a radar pose corresponding to each point cloud data. The radar pose corresponding to the point cloud data can be understood as the pose of the radar that collects the point cloud data.

[0084] Step 202: Determine a first object NeRF model and a first object geometric model based on a plurality of first sample data, wherein the first object NeRF model is used to predict an arbitrary viewing angle image of the first object, and the first object geometric model is a three-dimensional model of the first object.

[0085] In an embodiment of the present application, in order to simultaneously achieve high editability of visual simulation, high fidelity of visual simulation results, and low resource consumption of the visual simulation process, a visualized object geometry model and an object NeRF model that can be implicitly rendered later can be simultaneously formed through a sample data set. The user edits the display properties of the object to be simulated, such as the display posture, through the visualized object geometry model, thereby achieving high editability of visual simulation. At the same time, when rendering the visual simulation results, the image can be rendered directly through the object NeRF model, thereby achieving high fidelity of the visual simulation results and low resource consumption in the rendering process.

[0086] Optionally, in a scenario where the sample data set in step 201 also includes multiple second sample data, a second object NeRF model and a second object geometric model can also be determined based on the multiple second sample data, the second object NeRF model is used to predict any perspective image of the second object, and the second object geometric model is a three-dimensional model of the second object.

[0087] The following is an example of determining the NeRF model and the geometric model of the first object, the NeRF model and the geometric model of the second object based on the sample data set. The implementation methods in other scenarios can refer to the following content, and no examples will be given one by one in the following.

[0088] For ease of understanding, the principle of the NeRF model is explained first. For any object to be simulated, sample images of the object from different perspectives are collected, and the camera pose of each sample image is determined. The sample images and the corresponding camera pose are sent to the NeRF model for training. The trained NeRF model can then input the two-dimensional image of the object from any perspective.

[0089] Based on this, in some embodiments, the implementation process of determining the first object NeRF model and the second object NeRF model based on the sample data set can be: training a first initialized NeRF model based on multiple first sample data in the sample data set to obtain the first object NeRF model, and training a second initialized NeRF model based on multiple second sample data in the sample data set to obtain the second object NeRF model.

[0090] After obtaining the first object NeRF model, a first mesh model can be constructed based on the first object NeRF model, and the first mesh model is used as the first object geometric model. After obtaining the second object NeRF model, a second mesh model can be constructed based on the second object NeRF model, and the second mesh model is used as the second object geometric model.

[0091] In other words, in the embodiment of the present application, the first object geometric model may include the first object mesh model, and the second object geometric model may include the second object mesh model. In this way, subsequent users can edit the visual simulation requirements based on the visualized first object mesh model and the second object mesh model.

[0092] For example, in a scene where the first object is a foreground object and the second object is a background object, the first object geometric model includes a foreground object mesh model, and the second object geometric model includes a background object mesh model.

[0093] For example, the first object NeRF model may be a neural implicit surfaces (NeuS) model. In this scenario, after the first object NeRF model is obtained, the foreground mesh model may be constructed by using a signed distance function (SDF) in the first object NeRF model.

[0094] As another example, the second object NeRF model may be a NeRF in the wild (NeRF-W) model. In this scenario, after obtaining the second object NeRF model, the output result of the second object NeRF model may be processed by, for example, a marching cubes algorithm to construct a background grid model.

[0095] It should be noted that the embodiments of the present application do not limit the types of the NeRF model of the first object and the NeRF model of the second object, nor do they limit the algorithms for constructing the geometric model of the first object and the second object. Any NeRF model and algorithm for constructing a geometric model can be applied in the embodiments of the present application.

[0096] Optionally, in a scenario where the sample data in the sample data set is sample point cloud data, in addition to obtaining the first object geometric model and the second object geometric model by constructing the first object mesh model and the second object mesh model, the cloud platform can also directly construct the first object point cloud model and the second object point cloud model through the sample point cloud data, and use the first object point cloud model as the first object geometric model and the second object point cloud model as the second object geometric model.

[0097] In this scenario, optionally, the cloud platform can also determine the first object geometric model by constructing a first object point cloud model, and determine the second object geometric model by constructing a second object mesh model.

[0098] In addition, before determining the first object NeRF model and the first object geometric model based on multiple first sample data, the first object sample data can also be processed to eliminate other interfering objects except the first object in the first object sample data. This can not only reduce the amount of calculation required for training the first object NeRF model, but also improve the realism of subsequent visual simulation results.

[0099] For example, in a scenario where the first object is a foreground object and the first sample data is a foreground sample image, the first sample data may be processed by processing the foreground sample image through a semantic segmentation algorithm to identify the foreground of the foreground sample image, thereby generating a foreground mask.

[0100] For example, in an autonomous driving scenario, for a foreground sample image, a semantic segmentation algorithm can be used to identify other dynamic vehicles and other foregrounds around the autonomous driving vehicle in the foreground sample image, thereby generating a mask of the other dynamic vehicles and other foregrounds.

[0101] Accordingly, before determining the second object NeRF model and the second object geometric model based on multiple second sample data, the second sample data can also be processed to eliminate other interfering objects in the second sample data except the second object. This can not only reduce the amount of calculation required for training the second object NeRF model, but also improve the realism of subsequent visual simulation results.

[0102] For example, in a scenario where the second object is a background object and the second sample data is a background sample image, the second sample data may be processed by processing the background sample image using a semantic segmentation algorithm to identify the background of the background sample image, thereby generating a background mask.

[0103] For example, in an autonomous driving scenario, for background sample images, a semantic segmentation algorithm can be used to identify the background such as the road surrounding the autonomous driving vehicle in the background sample image, thereby generating a mask of the background such as the road.

[0104] Optionally, if the first sample data and the second sample data are not distinguished in the sample data set, the sample data can be directly identified to obtain the foreground mask and the background mask of the sample data, and then the first object NeRF model and the first object geometric model are determined through the foreground mask, and the second object NeRF model and the second object geometric model are determined based on the background mask.

[0105] Step 203: Displaying the first object geometric model on the front-end interactive interface.

[0106] In an embodiment of the present application, in order to improve the editability of the visual simulation process, a front-end interactive interface is provided to the user, and the first object geometric model constructed in step 202 is displayed on the front-end interactive interface, so that the user can use the front-end interactive interface to edit the first object at a certain angle to be simulated, that is, to edit the display properties of the first object geometric model.

[0107] Optionally, in the scene where the second object geometric model is determined in step 202, the second object geometric model can also be displayed on the front-end interactive interface, so that the user can also use the front-end interactive interface to edit the angle of the second object to be simulated, that is, edit the display properties of the second object geometric model.

[0108] For example, in a scene where the first object is a foreground object and the second object is a background object, the first object geometric model and the second object geometric model are simultaneously displayed on the front-end interactive interface, and the user can edit the position of the foreground object on the background object on the front-end interactive interface with the help of the first object geometric model and the second object geometric model, thereby realizing the combination of the foreground object and the background object. The specific implementation method will be described in detail in the subsequent step 204.

[0109] Step 204: In response to the user's editing operation on the front-end interactive interface, the display attributes and rendering perspective posture of the first object geometric model are determined, and the display attributes include the display posture.

[0110] In some embodiments, step 204 may be implemented by: determining a display pose of the first object geometric model in response to a user's movement and / or rotation operation on the first object geometric model.

[0111] Optionally, in some other embodiments, the display attribute also includes a display size. In this scenario, step 204 may be implemented by: determining the display position of the first object geometric model in response to the user's movement and / or rotation operation on the first object geometric model; and determining the display size of the first object geometric model in response to the user's scaling operation on the first object geometric model.

[0112] In addition, in the scenario where the front-end interactive interface also displays the second object geometric model in step 203, the display properties of the first object geometric model, the display properties of the second object geometric model and the rendering perspective pose can be determined in response to the user's editing operation on the front-end interactive interface.

[0113] The following explanation is given by taking the simultaneous determination of the display attributes of the first object geometric model, the display attributes of the second object geometric model, and the rendering perspective pose as an example.

[0114] Among them, in response to the user's editing operation on the front-end interactive interface, the display properties of the first object geometric model and the display properties of the second object geometric model can be determined as follows: in response to the user's movement and / or rotation operations on the first object geometric model and / or the second object geometric model, the display posture of the first object geometric model and the display posture of the second object geometric model are determined.

[0115] In an embodiment of the present application, a user can move and / or rotate the first object geometric model and / or the second object geometric model on the front-end interactive interface, thereby editing the display posture of the first object geometric model and / or the display posture of the second object geometric model.

[0116] For example, in a scene where the first object is a foreground object and the second object is a background object, the user can only edit the display pose of the geometric model of the first object, but not the display pose of the geometric model of the second object. Optionally, the user can edit the display pose of the geometric model of the first object and the display pose of the geometric model of the second object at the same time. Examples are not given one by one here.

[0117] In addition, the user can edit the display posture of a geometric model by moving the geometric model on the front-end interactive interface, by rotating the geometric model on the front-end interactive interface, or by simultaneously moving and rotating the geometric model on the front-end interactive interface.

[0118] For example, in a scene where the first object is a foreground object and the second object is a background object, when the user only edits the display pose of the geometric model of the first object, after the user moves and / or rotates the geometric model of the first object, the visual simulation device can extract the current display pose of the geometric model of the first object as the determined display pose of the geometric model of the first object. At the same time, the initial display pose of the geometric model of the second object on the front-end interactive interface is used as the determined display pose of the geometric model of the second object. The initial display poses of the geometric model of the first object and the geometric model of the second object displayed on the front-end interactive interface can be, for example, the origin position in the world coordinate system.

[0119] In some other embodiments, the display attributes may further include a display size. In this scenario, in response to the user's editing operation on the front-end interactive interface, the display attributes of the first object geometric model and the display attributes of the second object geometric model may be determined by: in response to the user's scaling operation on the first object geometric model and / or the second object geometric model, the display size of the first object geometric model and the display size of the second object geometric model are determined.

[0120] In an embodiment of the present application, the user can scale the first object geometric model and / or the second object geometric model on the front-end interactive interface, thereby editing the display size of the first object geometric model and / or the display size of the second object geometric model.

[0121] For example, in a scene where the first object is a foreground object and the second object is a background object, the user can only edit the display size of the geometric model of the first object, but not the display size of the geometric model of the second object. Alternatively, the user can edit the display size of the geometric model of the first object and the display size of the geometric model of the second object at the same time. Examples are not given one by one here.

[0122] For example, when the user only edits the display size of the first object geometric model, after the user performs a scaling operation on the first object geometric model, the visual simulation device can extract the size of the first object geometric model as the determined display size of the first object geometric model. At the same time, the display size of the second object geometric model is set to a default value, such as the size of the second object geometric model displayed on the current front-end interactive interface.

[0123] Figure 4 Schematic diagram of a front-end interactive interface provided by an embodiment of the present application. Figure 4 As shown, the front-end interactive interface displays roads, static buildings and vehicles, wherein the vehicle is a first object geometric model, and the road and static buildings are second object geometric models.

[0124] When the user moves, rotates, or scales the vehicle on the front-end interactive interface, the display properties of the first object geometric model can be edited. Figure 4 As shown on the right, after the user moves, rotates or scales the vehicle on the front-end interactive interface, the right window will display the display properties of the first object geometric model, which include display posture and display size. Among them, the display posture includes the position in the X / Y / Z three-axis of the world coordinate system, and the deflection angle with the X / Y / Z three-axis. The display size includes the ratio of the X / Y / Z three-axis to the default size. Specifically, the position of the vehicle in the X / Y / Z three-axis of the world coordinate system is edited by moving operations, the deflection angle of the vehicle with the X / Y / Z three-axis is edited by rotating operations, and the ratio of the vehicle on the X / Y / Z three-axis to the default size is edited by scaling operations.

[0125] Optionally, in other embodiments, the user can directly input the pose and size of the first object geometric model and the pose and size of the second object geometric model to be displayed in the front-end interactive interface, and the visual simulation device determines the display properties of the first object geometric model and the display properties of the second object geometric model based on the user input.

[0126] In addition, in some embodiments, in response to the user's editing operation on the front-end interactive interface, the implementation method of determining the rendering perspective posture can be: displaying a virtual rendering camera on the front-end interactive interface; and determining the rendering perspective posture in response to the user's movement and / or rotation operation of the virtual rendering camera.

[0127] In an embodiment of the present application, the user can move and / or rotate the virtual rendering camera on the front-end interactive interface, thereby editing the rendering perspective posture.

[0128] The virtual rendering camera may be, for example, a camera icon, so that the user can move and / or rotate the virtual rendering camera.

[0129] In addition, the rendering perspective pose edited by the user on the front-end interactive interface may include one perspective pose or multiple perspective poses.

[0130] When the rendered perspective pose includes one perspective pose, the subsequent cloud platform only needs to render one frame of image corresponding to the perspective pose. When the rendered perspective pose includes multiple perspective poses, the subsequent cloud platform can render multiple frames of image based on the multiple perspective poses to form a simulated video based on the multiple frames of image.

[0131] Figure 5 is a schematic diagram of another front-end interactive interface provided by an embodiment of the present application. Figure 5 As shown, the front-end interactive interface displays roads, static buildings and vehicles. The vehicle is a first object geometric model, and the road and static buildings are second object geometric models. The front-end interactive interface also displays a virtual rendering camera.

[0132] When the user moves or rotates the virtual rendering camera on the front-end interactive interface, the rendering perspective position can be edited. Figure 5 As shown, the user places the virtual rendering camera on Figure 5 At the three positions in , three viewing angle poses can be determined based on these three positions, that is, the rendering viewing angle poses edited by the user include three viewing angle poses. Figure 5 These three viewing positions are referred to as viewing position 1, viewing position 2, and viewing position 3 respectively.

[0133] like Figure 5As shown on the right, when editing any perspective pose, after the user moves or rotates the virtual rendering camera on the front-end interactive interface, the right window will display the pose of the virtual rendering camera and use the pose as a perspective pose. The perspective pose includes the position of the virtual rendering camera in the X / Y / Z three-axis of the world coordinate system, and the deflection angle with the X / Y / Z three-axis. Specifically, the position of the virtual rendering camera in the X / Y / Z three-axis of the world coordinate system is edited by moving operations, and the deflection angle of the virtual rendering camera with the X / Y / Z three-axis is edited by rotating operations.

[0134] Optionally, in some other embodiments, the user may directly input the desired viewing angle and posture for rendering in the front-end interactive interface, and the visual simulation device acquires the rendering viewing angle and posture based on the user input.

[0135] Step 205: Determine at least one first object image based on the display properties of the first object geometric model, the rendering view pose and the first object NeRF model.

[0136] In some embodiments, in a scenario where the sample data set also includes multiple second sample data, in step 205, the visual simulation device also determines at least one second object image based on the display properties of the second object, the rendering perspective pose, and the NeRF model of the second object, and fuses at least one first object image and at least one second object image to obtain panoramic simulation data.

[0137] The following description will be made by taking the panoramic simulation data as an example. The implementation method of the visual simulation device only rendering at least one first object image can refer to the following embodiment, which will not be described in detail later.

[0138] By way of example, an implementation method for determining at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model may be: based on the display properties of the first object geometric model and the rendering perspective pose, determining a first rendering pose in a first coordinate system, where the first coordinate system is the object coordinate system of the first object geometric model, and the rendering perspective pose is the pose in the world coordinate system; and determining at least one first object image based on the first rendering pose and the NeRF model of the first object.

[0139] Correspondingly, the implementation method of determining at least one second object image based on the display properties of the second object geometric model, the rendering perspective pose and the second object NeRF model can be: based on the display properties of the second object geometric model and the rendering perspective pose, determine the second rendering pose in the second coordinate system, the second coordinate system is the object coordinate system of the second object geometric model, and the rendering perspective pose is the pose in the world coordinate system; based on the second rendering pose and the NeRF model of the second object, determine at least one second object image.

[0140] The viewing pose required for rendering an image with the NeRF model is the viewing pose in the object coordinate system of the geometric model to be rendered, while the viewing pose edited by the user is usually the viewing pose in the world coordinate system. Therefore, after the visual simulation device obtains the rendering viewing pose based on step 204, it is necessary to convert the rendering viewing pose into a first rendering pose in the first coordinate system and a second rendering pose in the second coordinate system, that is, to convert them into a first rendering pose aligned with the NeRF model of the first object and a second rendering pose aligned with the NeRF model of the second object, respectively.

[0141] The object coordinate system of the first object geometric model can be understood as the coordinate system of the first object geometric model itself, and the origin of the object coordinate system can be the center position of the first object geometric model, or other positions such as the center of gravity of the first object geometric model.

[0142] Based on step 204, it can be known that the rendering perspective pose may include one perspective pose or multiple perspective poses.

[0143] When the rendering perspective pose includes a perspective pose, the first rendering pose and the second rendering pose determined at this time also only include one rendering pose. Accordingly, based on the first rendering pose and the first object NeRF model, a first object image is determined, and based on the second rendering pose and the second object NeRF model, a second object image is determined, and finally a panoramic image is obtained after fusion.

[0144] When the rendering perspective pose includes multiple perspective poses, the first rendering pose and the second rendering pose determined at this time include multiple rendering poses respectively. Accordingly, based on the first rendering pose and the first object NeRF model, multiple first object images are determined, and based on the second rendering pose and the second object NeRF model, multiple second object images are determined, and finally multiple frames of panoramic images are obtained after fusion. The subsequent visual simulation device can generate a simulated video based on these multiple frames of panoramic images.

[0145] In this scenario, in some embodiments, based on the display properties of the first object geometric model and the rendering perspective pose, determining the first rendering pose in the first coordinate system can be achieved by: based on the display properties of the first object geometric model, converting multiple perspective poses into multiple rendering poses in the first coordinate system, respectively, and the multiple rendering poses constitute the first rendering pose.

[0146] Accordingly, based on the first rendering pose and the NeRF model of the first object, determining at least one first object image may be implemented by: determining a first rendering trajectory based on multiple rendering poses; and determining multiple first object images based on the first rendering trajectory and the NeRF model of the first object.

[0147] The implementation method of determining the first rendering trajectory based on the multiple rendering postures may be: determining the first rendering trajectory through an interpolation algorithm based on the multiple rendering postures. Optionally, the multiple rendering postures may also be directly combined as the first rendering trajectory.

[0148] In addition, the implementation method of determining the second rendering pose in the second coordinate system can refer to the above embodiment, which will not be repeated here.

[0149] For example, the visual simulation device can convert multiple viewing angles into multiple rendering angles at a first coordinate based on the display properties of the geometric model of the first object; determine the first rendering trajectory through an interpolation algorithm based on the multiple rendering angles at the first coordinate. Correspondingly, based on the display properties of the geometric model of the second object, convert multiple viewing angles into multiple rendering angles at a second coordinate; determine the second rendering trajectory through an interpolation algorithm based on the multiple rendering angles at the second coordinate.

[0150] In the above manner, the user can only edit a few key viewing angles and poses, and the visual simulation device obtains more rendering poses through an interpolation algorithm based on the viewing angles and poses edited by the user to form a rendering trajectory.

[0151] For example, the duration of the video data to be simulated is 1 second, but the user has only edited 5 viewpoints. Assuming the frame rate is 30 frames per second, the visual simulation device can obtain 5 rendering poses through the 5 viewpoints edited by the user, and obtain another 25 rendering poses through the interpolation algorithm, thereby obtaining a total of 30 rendering poses, which constitute a rendering trajectory.

[0152] like Figure 5 As shown, the visual simulation device can determine a rendering trajectory based on the three viewing angles and poses edited by the user.

[0153] It should be noted that, since the first rendering pose includes multiple rendering poses obtained by converting multiple viewpoint poses according to the first coordinate system, and the second rendering pose includes multiple rendering poses obtained by converting multiple viewpoint poses according to the second coordinate system, there is a one-to-one correspondence between the first rendering pose including multiple rendering poses and the second rendering pose including multiple rendering poses. And in order to facilitate subsequent fusion, when the first rendering pose including multiple rendering poses and the second rendering pose including multiple rendering poses are interpolated, the interpolation is performed according to the same rule, so that the rendering poses in the first rendering trajectory and the rendering poses in the second rendering trajectory are also one-to-one corresponding.

[0154] Table 1 is a correspondence between a rendering pose in a first rendering trajectory and a rendering pose in a second rendering trajectory provided by an embodiment of the present application. As shown in Table 1, the user edits 5 viewing poses, and the visual simulation device can obtain 5 rendering poses in the first coordinate system and 5 rendering poses in the second coordinate system through the coordinate system conversion.

[0155] Through the interpolation algorithm, rendering poses 6-11 can be interpolated between rendering pose 1 and rendering pose 2 in the first coordinate system, rendering poses 12-17 can be interpolated between rendering pose 2 and rendering pose 3 in the first coordinate system, rendering poses 18-23 can be interpolated between rendering pose 3 and rendering pose 4 in the first coordinate system, and rendering poses 24-30 can be interpolated between rendering pose 4 and rendering pose 5 in the first coordinate system.

[0156] Through the same interpolation algorithm, rendering pose 6'-11' can be interpolated between rendering pose 1' and rendering pose 2' in the second coordinate system, rendering pose 12'-17' can be interpolated between rendering pose 2' and rendering pose 3' in the second coordinate system, rendering pose 18'-23' can be interpolated between rendering pose 3' and rendering pose 4' in the second coordinate system, and rendering pose 24'-30' can be interpolated between rendering pose 4' and rendering pose 5' in the second coordinate system.

[0157] Based on this, the 30 rendering poses in the first rendering trajectory and the 30 rendering poses in the second rendering trajectory are in one-to-one correspondence. And they all correspond to a certain rendering perspective pose in the world coordinate system. It should be noted that since some rendering poses are obtained by interpolation, the rendering perspective pose in the world coordinate system corresponding to the interpolated rendering pose is not the rendering perspective pose actually edited by the user.

[0158] Table 1

[0159]

[0160] The following is an explanation using how to obtain the rendering pose in the first rendering pose as an example. For the implementation method of the second rendering pose, reference may be made to the following content.

[0161] For example, in a scenario where the display properties only include display poses, based on the display properties of the first object geometric model, the implementation method of converting multiple viewing poses into multiple rendering poses at the first coordinate can be: each viewing pose is converted into a rendering pose at the first coordinate based on the display pose of the first object geometric model through a coordinate system conversion algorithm.

[0162] The embodiment of the present application does not limit the implementation of the coordinate system conversion algorithm. For example, the viewing angle pose and the display pose of the first object geometric model may be multiplied, and the obtained pose is used as a rendering pose under the first coordinate.

[0163] As another example, in a scenario where display attributes include display pose and display size, based on the display attributes of the first object geometric model, the implementation method of converting multiple viewing poses into multiple rendering poses at the first coordinate can be: each viewing pose is converted into an initial rendering pose at the first coordinate based on the display pose of the first object geometric model through a coordinate system conversion algorithm, and then the initial rendering pose is adjusted according to the proportional relationship between the display size of the first object geometric model and the default display size to obtain the final rendering pose.

[0164] For example, when the rendering pose is expressed in a matrix manner, after the initial rendering pose is obtained, the proportional relationship between the display size of the first object geometric model and the default display size is applied to the main diagonal of the initial rendering pose to obtain the final rendering pose.

[0165] Optionally, in some other embodiments, based on the display properties of the first object geometric model and the rendering perspective pose, determining the first rendering trajectory in the first coordinate system and the second rendering trajectory in the second coordinate system can be implemented by: determining the initial rendering trajectory through an interpolation algorithm based on multiple perspective poses, the initial rendering trajectory including a larger number of perspective poses; based on the display properties of the first object geometric model, converting the perspective poses in the initial rendering trajectory into rendering poses in the first coordinate system, thereby obtaining the first rendering trajectory; based on the display properties of the second object geometric model, converting the perspective poses in the initial rendering trajectory into rendering poses in the second coordinate system, thereby obtaining the second rendering trajectory.

[0166] The rendering pose in the first rendering trajectory determined in this way also corresponds one-to-one to the rendering pose in the second rendering trajectory.

[0167] In addition, when the rendering perspective pose includes multiple perspective poses, since the rendering pose in the first rendering trajectory corresponds to the rendering pose in the second rendering trajectory one by one, the multiple first object images determined based on the first object NeRF model correspond to the multiple second object images determined based on the second object NeRF model, and correspond one by one to the same rendering perspective pose in the world coordinate system.

[0168] For example, the first rendering trajectory and the second rendering trajectory both include 30 rendering poses, then 30 first object images are generated based on the first object NeRF model, and 30 second object images are generated based on the second object NeRF model, and the 30 first object images correspond to the 30 second object images one by one.

[0169] In this scenario, the method for fusing multiple first object images and multiple second object images to obtain simulated video data can be: obtaining a first object image and a second object image corresponding to the first object image from multiple first object images and multiple second object images, the first object image including a first object red-green-blue (RGB) image and a first object depth image, and the second object image including a second object RGB image and a second object depth image; based on the first object depth image and the second object depth image, fusing the first object RGB image and the second object RGB image into a panoramic RGB image.

[0170] Based on the first object depth image and the second object depth image, the occlusion relationship between the first object and the second object at the same viewpoint can be determined, and then based on the occlusion relationship, it is determined whether the pixel value displayed at the viewpoint in the panoramic RGB image is the first object RGB image or the second object RGB image.

[0171] For example, for a certain viewpoint, if the depth corresponding to the first object depth image at the viewpoint is greater than the depth corresponding to the second object depth image, the pixel value at the viewpoint in the second object RGB image is displayed in the panoramic RGB image. Correspondingly, if the depth corresponding to the first object depth image at the viewpoint is less than the depth corresponding to the second object depth image, the pixel value at the viewpoint in the first object RGB image is displayed in the panoramic RGB image.

[0172] In a scene where the first object is a foreground object and the second object is a background object, further, based on the first object depth image and the second object depth image, after the first object RGB image and the second object RGB image are fused into a panoramic RGB image, the visual simulation device can also determine the light direction characteristics of the second object RGB image; and generate a shadow of the foreground in the panoramic RGB image based on the light direction characteristics of the second object RGB image.

[0173] After the first object RGB image and the second object RGB image are fused based on the depth image, the shadow of the foreground may be further generated based on the light direction characteristics of the background, thereby improving the realism of the visual simulation result.

[0174] The above method can be specifically implemented by a shadow generation algorithm. It should be noted that the embodiment of the present application does not limit the shadow generation algorithm, and any shadow generation algorithm can be applied in the embodiment of the present application.

[0175] In a scene where the first object is a foreground object and the second object is a background object, further, based on the first object depth image and the second object depth image, after the first object RGB image and the second object RGB image are fused into a panoramic RGB image, the visual simulation device can also determine the light intensity characteristics of the second object RGB image; and adjust the color of the foreground in the panoramic RGB image based on the light intensity characteristics of the second object RGB image.

[0176] After the first object RGB image and the second object RGB image are fused based on the depth image, the color of the foreground may be further adjusted based on the light intensity characteristics of the background, thereby improving the realism of the visual simulation result.

[0177] The above method can be specifically implemented by an image harmonization algorithm. It should be noted that the present embodiment of the application does not limit the image harmonization algorithm, and any image harmonization algorithm can be applied in the present embodiment of the application.

[0178] The above-mentioned generating the shadow of the foreground in the panoramic RGB image and adjusting the color of the foreground in the panoramic RGB image may be performed simultaneously, or one of them may be performed, which will not be described one by one in examples herein.

[0179] Figure 6 This is another image processing flow diagram provided in an embodiment of the present application. Figure 6 Take the first object as the foreground object and the second object as the background object as an example. Figure 6 As shown, the image processing flow includes the following three steps.

[0180] Step 1: Preparation and production of 3D materials, including collecting sample data, training the first object NeRF model for the foreground object and the second object NeRF model for the background object through the sample data, and determining the visualized first object geometric model and the second object geometric model. The specific implementation method can refer to the aforementioned steps 201 and 202.

[0181] Step 2: Combination editing of materials, including positioning the geometric model of the first object, editing the rendering perspective pose, and aligning the rendering perspective pose with the two object NeRF models, and then determining the rendering trajectory, etc. For specific implementation methods, please refer to the aforementioned steps 203, 204, and part of step 205.

[0182] Step 3: 3D rendering, including predicting the first object image according to the rendering trajectory through the first object NeRF model, predicting the second object image according to the rendering trajectory through the second object NeRF model, fusing the first object image with the second object image, etc. For specific implementation methods, please refer to part of the content in the aforementioned step 205.

[0183] In summary, in the embodiment of the present application, a visualized object geometry model and an object NeRF model that can be implicitly rendered later are simultaneously formed through a sample data set. The user edits the display properties of the object to be simulated, such as the display posture, through the visualized object geometry model, thereby achieving high editability of the visual simulation. At the same time, when rendering the visual simulation results, the image can be directly rendered through the object NeRF model, thereby achieving high realism of the visual simulation results and low resource consumption in the rendering process.

[0184] The following is an explanation of the relevant devices and equipment provided in the embodiments of the present application.

[0185] The present application also provides an image processing device, such as Figure 7 As shown, the device 700 includes:

[0186] The acquisition module 701 is used to acquire a sample data set, where the sample data set includes a plurality of first sample data, where the plurality of first sample data are used to indicate the imaging information of a first object from multiple viewing angles. Figure 2 Step 201 in the embodiment.

[0187] The first determination module 702 is used to determine the first object neural radiation field NeRF model and the first object geometric model based on multiple first sample data. The first object NeRF model is used to predict the arbitrary perspective image of the first object, and the first object geometric model is a three-dimensional model of the first object. For specific implementation methods, please refer to Figure 2 Step 202 in the embodiment.

[0188] The display module 703 is used to display the geometric model of the first object on the front-end interactive interface. Figure 2 Step 203 in the embodiment.

[0189] The second determination module 704 is used to determine the display attributes and rendering perspective pose of the first object geometric model in response to the user's editing operation on the front-end interactive interface, where the display attributes include the display pose. Figure 2 Step 204 in an embodiment.

[0190] The third determination module 705 is used to determine at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model. Figure 2 Step 205 in the embodiment.

[0191] Optionally, the third determining module is used to:

[0192] Determine a first rendering pose in a first coordinate system based on display properties of the first object geometric model and a rendering perspective pose, where the first coordinate system is an object coordinate system of the first object geometric model, and the rendering perspective pose is a pose in a world coordinate system;

[0193] At least one first object image is determined based on the first rendering pose and the NeRF model of the first object.

[0194] Optionally, the rendering perspective pose includes a plurality of perspective poses, and the first rendering pose includes a plurality of rendering poses;

[0195] The third determination module is used for:

[0196] Based on the multiple rendering poses, determining a first rendering trajectory;

[0197] Based on the first rendering trajectory and the NeRF model of the first object, a plurality of first object images are determined.

[0198] Optionally, the third determining module is used to:

[0199] Based on the multiple rendering poses, a first rendering trajectory is determined by an interpolation algorithm.

[0200] Optionally, the sample data set further includes a plurality of second sample data, and the plurality of second sample data are used to indicate imaging information of a second object from multiple viewing angles;

[0201] The first determination module is further used to determine a second object NeRF model and a second object geometric model based on a plurality of second sample data, wherein the second object NeRF model is used to predict an arbitrary viewing angle image of the second object, and the second object geometric model is a three-dimensional model of the second object;

[0202] The display module is further used to display the second object geometric model on the front-end interactive interface;

[0203] The second determination module is further used to determine the display attribute of the second object in response to the user's editing operation on the front-end interactive interface;

[0204] The third determination module is further used to determine at least one second object image based on the display attribute of the second object, the rendering perspective pose, and the NeRF model of the second object;

[0205] The device also includes a fusion module, which is used to fuse at least one first object image and at least one second object image to obtain panoramic simulation data.

[0206] Optionally, at least one first object image and at least one second object image correspond one-to-one to the same rendering perspective pose in at least one rendering perspective pose;

[0207] The Fusion Module is used to:

[0208] Acquire a first object image and a second object image corresponding to the first object image from at least one first object image and at least one second object image, the first object image comprising a first object red, green, and blue (RGB) image and a first object depth image, and the second object image comprising a second object RGB image and a second object depth image;

[0209] Based on the first object depth image and the second object depth image, the first object RGB image and the second object RGB image are fused into a panoramic RGB image.

[0210] Optionally, the first object is a foreground object and the second object is a background object;

[0211] The device also includes:

[0212] The receiving module is used to receive object category information configured by a user, where the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

[0213] Optionally, the display attributes also include display size;

[0214] The second determination module is used for:

[0215] In response to a user's movement and / or rotation operation on the first object geometric model, determining a display pose of the first object geometric model;

[0216] In response to a user's zooming operation on the first object geometric model, a display size of the first object geometric model is determined.

[0217] Optionally, the second determining module is used to:

[0218] Display the virtual rendering camera on the front-end interactive interface;

[0219] In response to a user's movement and / or rotation operation on the virtual rendering camera, a rendering perspective pose is determined.

[0220] Optionally, the first object geometric model comprises a first object mesh model.

[0221] Among them, the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704 and the third determination module 705 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the third determination module 705 is introduced below by taking the third determination module 705 as an example. Similarly, the implementation of the acquisition module 701, the first determination module 702, the display module 703 and the second determination module 704 can refer to the implementation of the third determination module 705.

[0222] As an example of a software functional unit, the third determination module 705 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the third determination module 705 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0223] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0224] As an example of a hardware functional unit, the third determination module 705 may include at least one computing device, such as a server, etc. Alternatively, the third determination module 705 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0225] The multiple computing devices included in the third determination module 705 can be distributed in the same region or in different regions. The multiple computing devices included in the third determination module 705 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the third determination module 705 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0226] It should be noted that, in other embodiments, the third determination module 705 can be used to execute any step in the image processing method, the acquisition module 701 can be used to execute any step in the image processing method, the first determination module 702 can be used to execute any step in the image processing method, the display module 703 can be used to execute any step in the image processing method, and the second determination module 704 can be used to execute any step in the image processing method. The steps that the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705 are responsible for implementing can be specified as needed, and the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705 respectively implement different steps in the image processing method to realize the full functions of the image processing device.

[0227] The present application embodiment also provides a computing device 800. Figure 8 As shown, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate through the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0228] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 804 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0229] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0230] The memory 806 may include a volatile memory, such as a random access memory (RAM). The processor 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0231] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705, thereby implementing the image processing method. That is, the memory 806 stores instructions for executing the image processing method.

[0232] Alternatively, the memory 806 stores executable codes, and the processor 804 executes the executable codes to respectively implement the functions of the aforementioned image processing apparatus, thereby implementing the image processing method. That is, the memory 806 stores instructions for executing the image processing method.

[0233] The communication interface 803 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0234] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0235] like Fig. 9 As shown, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the image processing method.

[0236] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, the combination of one or more computing devices 800 may jointly execute instructions for executing the image processing method.

[0237] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the image processing apparatus. That is, the instructions stored in the memory 806 in different computing devices 800 may implement the functions of one or more of the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705.

[0238] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the acquisition module 701, the display module 703, and the second determination module 704. At the same time, the memory 806 in the computing device 800B stores instructions for executing the functions of the first determination module 702 and the third determination module 705.

[0239] Fig.10 The connection method between the computing device clusters shown can be that considering that the image processing method provided in this application needs to interact with the user and a large amount of calculation is required in the background, it is considered that the functions implemented by the acquisition module 701, the display module 703 and the second determination module 704 are executed by the computing device 800A, and the functions implemented by the first determination module 702 and the third determination module 705 are executed by the computing device 800B.

[0240] It should be understood that Fig.10 The functions of the computing device 800A shown in FIG. 8 may also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B may also be completed by multiple computing devices 800.

[0241] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the image processing method.

[0242] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a sample data set, where the sample data set includes a plurality of first sample data, where the plurality of first sample data are used to indicate imaging information of a first object from a plurality of viewing angles; Determine a first object neural radiation field NeRF model and a first object geometric model based on the plurality of first sample data, wherein the first object NeRF model is used to predict an arbitrary viewing angle image of the first object, and the first object geometric model is a three-dimensional model of the first object; Displaying the first object geometric model on the front-end interactive interface; In response to the user's editing operation on the front-end interactive interface, determining the display attributes and rendering perspective pose of the first object geometric model, wherein the display attributes include the display pose; At least one first object image is determined based on the display properties of the first object geometric model, the rendering view pose, and the first object NeRF model.

2. The method according to claim 1, characterized in that The determining at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model comprises: Determine a first rendering pose in a first coordinate system based on display properties of the first object geometric model and the rendering perspective pose, where the first coordinate system is an object coordinate system of the first object geometric model, and the rendering perspective pose is a pose in a world coordinate system; The at least one first object image is determined based on the first rendering pose and the NeRF model of the first object.

3. The method according to claim 2, characterized in that The rendering perspective pose includes a plurality of perspective poses, and the first rendering pose includes a plurality of rendering poses; The determining, based on the first rendering pose and the NeRF model of the first object, the at least one first object image comprises: Based on the multiple rendering poses, determining a first rendering trajectory; The plurality of first object images are determined based on the first rendering trajectory and a NeRF model of the first object.

4. The method according to claim 3, characterized in that The determining a first rendering trajectory based on the first rendering pose includes: Based on the multiple rendering poses, the first rendering trajectory is determined by an interpolation algorithm.

5. The method according to any one of claims 1 to 4, characterized in that: The sample data set further includes a plurality of second sample data, wherein the plurality of second sample data are used to indicate imaging information of a second object at multiple viewing angles; The method comprises: Determine a second object NeRF model and a second object geometric model based on the plurality of second sample data, wherein the second object NeRF model is used to predict an arbitrary viewing angle image of the second object, and the second object geometric model is a three-dimensional model of the second object; Displaying the second object geometric model on the front-end interactive interface; In response to the editing operation of the user on the front-end interactive interface, determining a display attribute of the second object; Determine at least one second object image based on the display attribute of the second object, the rendering view pose, and the NeRF model of the second object; The at least one first object image and the at least one second object image are fused to obtain panoramic simulation data.

6. The method according to claim 5, characterized in that The at least one first object image and the at least one second object image correspond one-to-one to the same rendering perspective pose in the at least one rendering perspective pose; The step of fusing the at least one first object image and the at least one second object image to obtain panoramic simulation data includes: Acquire a first object image and a second object image corresponding to the first object image from the at least one first object image and the at least one second object image, the first object image comprising a first object red, green, and blue (RGB) image and a first object depth image, and the second object image comprising a second object RGB image and a second object depth image; Based on the first object depth image and the second object depth image, the first object RGB image and the second object RGB image are fused into a panoramic RGB image.

7. The method according to claim 5 or 6, characterized in that The first object is a foreground object, and the second object is a background object; The method further comprises: Object category information configured by a user is received, where the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

8. The method according to any one of claims 1 to 7, characterized in that: The display attributes also include display size; The step of determining the display attribute of the first object geometric model in response to the user's editing operation on the front-end interactive interface includes: In response to the user's movement and / or rotation operation on the first object geometric model, determining a display posture of the first object geometric model; In response to the user's zooming operation on the first object geometric model, a display size of the first object geometric model is determined.

9. The method according to any one of claims 1 to 8, characterized in that: The step of determining the rendering perspective pose in response to the user's editing operation on the front-end interactive interface includes: Displaying a virtual rendering camera on the front-end interactive interface; In response to the user's movement and / or rotation operation on the virtual rendering camera, the rendering perspective pose is determined.

10. The method according to any one of claims 1 to 9, characterized in that: The first object geometric model includes a first object mesh model.

11. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a sample data set, wherein the sample data set includes a plurality of first sample data, and the plurality of first sample data are used for indicating imaging information of a first object at a plurality of viewing angles; A first determination module is used to determine a first object neural radiation field NeRF model and a first object geometric model based on the plurality of first sample data, wherein the first object NeRF model is used to predict an arbitrary viewing angle image of the first object, and the first object geometric model is a three-dimensional model of the first object; A display module, used for displaying the first object geometric model on a front-end interactive interface; A second determination module, configured to determine display attributes and rendering perspective pose of the first object geometric model in response to an editing operation of a user on the front-end interactive interface, wherein the display attributes include a display pose; The third determination module is used to determine at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model.

12. The device according to claim 11, characterized in that The third determination module is used for: Determine a first rendering pose in a first coordinate system based on display properties of the first object geometric model and the rendering perspective pose, where the first coordinate system is an object coordinate system of the first object geometric model, and the rendering perspective pose is a pose in a world coordinate system; The at least one first object image is determined based on the first rendering pose and the NeRF model of the first object.

13. The device according to claim 12, characterized in that The rendering perspective pose includes a plurality of perspective poses, and the first rendering pose includes a plurality of rendering poses; The third determination module is used for: Based on the multiple rendering poses, determining a first rendering trajectory; The plurality of first object images are determined based on the first rendering trajectory and a NeRF model of the first object.

14. The device according to claim 13, characterized in that The third determination module is used for: Based on the multiple rendering poses, the first rendering trajectory is determined by an interpolation algorithm.

15. The device according to any one of claims 11 to 14, characterized in that: The sample data set further includes a plurality of second sample data, wherein the plurality of second sample data are used to indicate imaging information of a second object at multiple viewing angles; The first determination module is further used to determine a second object NeRF model and a second object geometric model based on the plurality of second sample data, wherein the second object NeRF model is used to predict an arbitrary viewing angle image of the second object, and the second object geometric model is a three-dimensional model of the second object; The display module is further used to display the second object geometric model on the front-end interactive interface; The second determination module is further configured to determine a display attribute of the second object in response to the editing operation of the user on the front-end interactive interface; The third determination module is further used to determine at least one second object image based on the display attribute of the second object, the rendering perspective pose, and the NeRF model of the second object; The device also includes a fusion module, which is used to fuse the at least one first object image and the at least one second object image to obtain panoramic simulation data.

16. The device according to claim 15, characterized in that The at least one first object image and the at least one second object image correspond one-to-one to the same rendering perspective pose in the at least one rendering perspective pose; The fusion module is used to: Acquire a first object image and a second object image corresponding to the first object image from the at least one first object image and the at least one second object image, the first object image comprising a first object red, green, and blue (RGB) image and a first object depth image, and the second object image comprising a second object RGB image and a second object depth image; Based on the first object depth image and the second object depth image, the first object RGB image and the second object RGB image are fused into a panoramic RGB image.

17. The device according to claim 15 or 16, characterized in that The first object is a foreground object, and the second object is a background object; The device also includes: The receiving module is used to receive object category information configured by a user, wherein the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

18. The device according to any one of claims 11 to 17, characterized in that: The display attributes also include display size; The second determining module is used for: In response to the user's movement and / or rotation operation on the first object geometric model, determining a display posture of the first object geometric model; In response to the user's zooming operation on the first object geometric model, a display size of the first object geometric model is determined.

19. The device according to any one of claims 11 to 18, characterized in that: The second determining module is used for: Displaying a virtual rendering camera on the front-end interactive interface; In response to the user's movement and / or rotation operation on the virtual rendering camera, the rendering perspective pose is determined.

20. The device according to any one of claims 11 to 19, characterized in that: The first object geometric model includes a first object mesh model.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

Images