Image processing method and apparatus, and related device

Abstract

The embodiments of the present application belong to the technical field of computer vision. Disclosed are an image processing method and apparatus, and a related device. In the embodiments of the present application, a viewable object geometric model, and an object NeRF model that can be subsequently rendered in an implicit manner are formed at the same time by means of a sample dataset. By means of the viewable object geometric model, a user edits a display attribute such as a display posture of an object to be simulated, thereby achieving high editability of visual simulation. In addition, when a visual simulation result is rendered, an image can be directly rendered by means of the object NeRF model, thereby achieving a high degree of realism of the visual simulation result and low resource consumption during a rendering process.

Description

Image processing method, device and related equipment Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to an image processing method, apparatus, and related equipment. Background Art

[0002] Image processing in visual simulation technology can be understood as generating virtual scene images that approximate the real scene using sample data such as collected sample images. For example, in the field of autonomous driving, visual simulation technology can be used to generate virtual scene images that simulate the environment surrounding an autonomous vehicle, thereby providing more data for the autonomous driving algorithm.

[0003] In related technologies, to enable editing of the foreground during visual simulation, the foreground in an image is explicitly rendered using computer graphics (CG) technology. However, the process of displaying and rendering the foreground using CG technology requires manual operation using specialized human resources and software tools, and consumes significant time and hardware resources to achieve realistic foreground rendering.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, and related equipment that can achieve high editability of the visual simulation process and reduce resource consumption while ensuring high fidelity of the visual simulation results. The technical solution is as follows:

[0006] In a first aspect, an image processing method is provided. In this method, a sample data set is acquired, the sample data set including a plurality of first sample data, the plurality of first sample data being used to indicate imaging information of a first object from multiple perspectives; a first object neural radiation field (NeRF) model and a first object geometric model are determined based on the plurality of first sample data, the first object NeRF model being used to predict an arbitrary perspective image of the first object, the first object geometric model being a three-dimensional model of the first object; the first object geometric model is displayed on a front-end interactive interface; in response to a user editing operation on the front-end interactive interface, display properties and a rendering perspective pose of the first object geometric model are determined, the display properties including the display pose; and at least one first object image is determined based on the display properties, the rendering perspective pose, and the first object NeRF model of the first object geometric model.

[0007] In this embodiment of the present application, a sample dataset is used to simultaneously form a visual object geometry model and an object NeRF model that can be subsequently implicitly rendered. Users can edit the display properties of the simulated object, such as its display pose, using the visual object geometry model, thereby achieving high editability in the visual simulation. Furthermore, when rendering the visual simulation results, the NeRF model can be used directly to render the image, achieving high fidelity in the visual simulation results while minimizing resource consumption during the rendering process.

[0008] Based on the method provided in the first aspect, in a possible implementation manner, the implementation process of determining at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model can be: based on the display properties of the first object geometric model and the rendering perspective pose, determine the first rendering pose in the first coordinate system, the first coordinate system is the object coordinate system of the first object geometric model, and the rendering perspective pose is the pose in the world coordinate system; based on the first rendering pose and the NeRF model of the first object, determine at least one first object image.

[0009] The viewing pose required by the NeRF model to render the image is the viewing pose in the object coordinate system of the geometric model to be rendered, while the viewing pose edited by the user is usually the viewing pose in the world coordinate system. Therefore, after obtaining the user-edited rendering viewing pose, the rendering viewing pose needs to be converted into the first rendering pose in the first coordinate system, that is, converted into the first rendering pose aligned with the NeRF model of the first object.

[0010] Based on the method provided in the first aspect, in one possible implementation, the rendering view pose includes multiple view poses, and the first rendering pose includes multiple rendering poses. In this scenario, determining at least one first object image based on the first rendering pose and a NeRF model of the first object may include: determining a first rendering trajectory based on the multiple rendering poses; and determining multiple first object images based on the first rendering trajectory and the NeRF model of the first object.

[0011] Through the above method, the user can only edit a few key view poses, and a rendering trajectory can be formed based on the several view poses edited by the user.

[0012] Based on the method provided in the first aspect, in a possible implementation manner, the process of determining the first rendering trajectory based on the first rendering pose may be: determining the first rendering trajectory through an interpolation algorithm based on multiple rendering poses.

[0013] In an embodiment of the present application, more rendering poses can be obtained through an interpolation algorithm to form a rendering trajectory.

[0014] Based on the method provided in the first aspect, in one possible implementation, the sample data set also includes multiple second sample data, and the multiple second sample data are used to indicate multiple perspective imaging information of the second object. In this scenario, in this method, a second object NeRF model and a second object geometric model can also be determined based on the multiple second sample data, the second object NeRF model is used to predict an arbitrary perspective image of the second object, and the second object geometric model is a three-dimensional model of the second object; the second object geometric model is displayed on the front-end interactive interface; in response to the user's editing operation on the front-end interactive interface, the display properties of the second object are determined; based on the display properties of the second object, the rendering perspective pose, and the NeRF model of the second object, at least one second object image is determined; and the at least one first object image and the at least one second object image are fused to obtain panoramic simulation data.

[0015] In an embodiment of the present application, the above method can also be used to perform visual simulation on a scene including multiple objects.

[0016] Based on the method provided in the first aspect, in one possible implementation, at least one first object image and at least one second object image correspond one-to-one to the same rendering perspective pose in at least one rendering perspective pose. In this scenario, the implementation process of fusing at least one first object image and at least one second object image to obtain panoramic simulation data can be: obtaining a first object image and a second object image corresponding to the first object image from at least one first object image and at least one second object image, the first object image including a first object red, green, and blue (RGB) image and a first object depth image, and the second object image including a second object RGB image and a second object depth image; based on the first object depth image and the second object depth image, fusing the first object RGB image and the second object RGB image into a panoramic RGB image.

[0017] Based on the first object depth image and the second object depth image, the occlusion relationship between the first object and the second object at the same viewpoint can be determined, and then based on the occlusion relationship, it is determined whether the pixel value displayed at the viewpoint in the panoramic RGB image is the first object RGB image or the second object scene RGB image, thereby improving the realism of the visual simulation results.

[0018] Based on the method provided in the first aspect, in one possible implementation, the first object is a foreground object and the second object is a background object. In this scenario, user-configured object type information may also be received, the object type information being used to indicate that the first object is a foreground object and the second object is a background object.

[0019] In the embodiment of the present application, the user can edit which objects are foreground objects and which objects are background objects.

[0020] Based on the method provided in the first aspect, in one possible implementation, the display attributes further include display size. In this scenario, in response to a user editing operation on the front-end interactive interface, determining the display attributes of the first object's geometric model may include: determining the display pose of the first object's geometric model in response to the user's movement and / or rotation operation on the first object's geometric model; and determining the display size of the first object's geometric model in response to the user's scaling operation on the first object's geometric model.

[0021] In an embodiment of the present application, a user can move and / or rotate the first object geometric model on the front-end interactive interface, thereby editing the display pose of the first object geometric model. The user can also scale the first object geometric model on the front-end interactive interface, thereby editing the display size of the first object geometric model.

[0022] Based on the method provided in the first aspect, in one possible implementation, in response to the user's editing operation on the front-end interactive interface, the implementation process of determining the rendering perspective pose can be: displaying a virtual rendering camera on the front-end interactive interface; and determining the rendering perspective pose in response to the user's movement and / or rotation operation on the virtual rendering camera.

[0023] In an embodiment of the present application, the user can move and / or rotate the virtual rendering camera on the front-end interactive interface, thereby editing the rendering perspective posture.

[0024] Based on the method provided in the first aspect, in a possible implementation manner, the first object geometric model includes a first object mesh model.

[0025] In the embodiment of the present application, the user can edit the visual simulation requirements based on the visualized first object mesh model.

[0026] In a second aspect, an image processing device is provided, which has the function of implementing the image processing method in the first aspect. The image processing device includes at least one module, and the at least one module is used to implement the image processing method provided in the first aspect.

[0027] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the image processing method provided in the first aspect above.

[0028] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the image processing method provided in the first aspect.

[0029] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the image processing method provided in the first aspect.

[0030] The technical effects obtained in the above-mentioned second, third, fourth and fifth aspects are similar to those obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a schematic diagram of an image processing system provided in an embodiment of the present application;

[0032] FIG2 is a flow chart of an image processing method provided in an embodiment of the present application;

[0033] FIG3 is a schematic diagram of a front-end interactive interface provided in an embodiment of the present application;

[0034] FIG4 is a schematic diagram of another front-end interactive interface provided in an embodiment of the present application;

[0035] FIG5 is a schematic diagram of another front-end interactive interface provided in an embodiment of the present application;

[0036] FIG6 is a schematic diagram of another image processing flow provided by an embodiment of the present application;

[0037] FIG7 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0038] FIG8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0039] FIG9 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0040] FIG10 is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0042] To facilitate subsequent understanding, the application scenarios of the embodiments of the present application are first explained below.

[0043] Visual simulation technology has long been a key research topic in fields such as computer vision and computer graphics. Highly realistic visual simulation can generate sufficient data to train visual perception systems with enhanced capabilities. In recent years, as applications such as autonomous driving and robotics have gradually moved from the laboratory into practical applications, the demand for visual simulation technology has become increasingly stringent. This requires not only a higher degree of realism, but also the ability to generate this data without excessively high costs in terms of time, manpower, and equipment.

[0044] Several visual simulation technologies are explained below with examples.

[0045] (1) CG-based visual simulation technology.

[0046] CG-based visual simulation technology mainly includes three steps:

[0047] First: Preparation and production of three-dimensional (3D) materials. Because models generated directly through 3D reconstruction technology are often difficult to use directly, CG-based visual simulation technology often requires professional modelers to manually create or modify 3D materials using specialized software, and then specifically produce the corresponding materials to achieve good visual effects.

[0048] Second: 3D asset assembly and editing. The 3D assets created in the first step are usually single and need to be assembled into a 3D scene. This step usually requires the involvement of a professional modeler, who uses specialized software to adjust the 3D asset's position, scene lighting, and other aspects.

[0049] Third: 3D rendering. For example, 3D rendering can be performed using physically-based rendering (PBR) technology.

[0050] For example, many autonomous driving simulation platforms use CG-based visual simulation technology to simulate vehicle driving scenarios.

[0051] Currently, CG-based visual simulation technology is gradually capable of producing relatively realistic simulation effects, but overall, there is still a gap between it and photorealistic rendering. Moreover, the more realistic the rendering, the greater the consumption of hardware and time, which is inconsistent with the demand for cost control in simulation.

[0052] Based on the above, we can see that the advantages of CG-based visual simulation technology are its maturity and high editability. However, the problem is that it is difficult to simultaneously meet the requirements of high realism and low resource consumption.

[0053] (2) Visual simulation technology based on NeRF model.

[0054] Unlike CG-based visual simulation technology that expresses 3D models in the form of explicit mesh models, NeRF-based visual simulation technology implicitly expresses 3D models in the form of neural networks. Therefore, the images generated by NeRF-based visual simulation technology have photo-level realism.

[0055] Furthermore, NeRF-based visual simulation technology eliminates the need for manual work by professional modelers. Instead, algorithms can be executed end-to-end to achieve highly realistic rendering results, significantly reducing labor costs. Furthermore, the NeRF model itself requires no specialized software or tools; it can be executed entirely in the background, requiring only a device capable of performing neural network inference. However, CG-based visual simulation technology sometimes requires better graphics card support, such as those that support ray tracing, for higher performance. Therefore, NeRF-based visual simulation technology can significantly reduce hardware costs.

[0056] However, the visual simulation technology based on the NeRF model also has its own problems, namely, it does not have good editability, which means it is difficult to generate new scenarios through editing based on the existing NeRF model.

[0057] Based on this, an embodiment of the present application provides an image processing method, which can not only achieve high editability of the visual simulation process but also reduce resource consumption while ensuring high realism of the visual simulation results.

[0058] The image processing method provided in the embodiments of the present application can be applied to cloud scenarios, such as autonomous driving scenarios. In such scenarios, the image processing method is executed by a cloud platform, that is, the cloud platform provides cloud services to users, and the cloud services are used to implement the image processing method provided in the embodiments of the present application. Optionally, the image processing method provided in the embodiments of the present application can also be executed by a terminal device. The embodiments of the present application do not limit the application scenarios of the image processing method.

[0059] The image processing system involved in the embodiments of the present application is explained below.

[0060] FIG1 is a schematic diagram of an image processing system provided in an embodiment of the present application. As shown in FIG1 , the system includes a data acquisition unit 10 and a visual simulation device 20. The visual simulation device includes a front-end interaction unit 201, an asset storage unit 202, and a calculation unit 203.

[0061] The data acquisition unit 10 and the visual simulation device 20 may be connected to each other via a wired or wireless manner for communication, and the various units in the visual simulation device 20 may also be connected to each other via a wired or wireless manner for communication.

[0062] The data acquisition unit 10 is used to collect sample data for subsequent visual simulation through the sample data. The sample data is used to indicate at least one object in a specific target scene, and the at least one object may include a foreground object and a background object. For example, the sample data can be a sample image collected for the target scene, or a sample point cloud collected for the target scene. The front-end interaction unit 201 is used for human-computer interaction to realize functions such as uploading sample data, visual editing, and displaying visual simulation results. The asset storage unit 202 is used to store sample data collected by the data acquisition unit 10 and information such as the trained NeRF model. The computing unit 203 is used to implement algorithmic calculations to realize the image processing method provided in the embodiment of the present application.

[0063] The data acquisition unit 10 can be any device capable of collecting sample data. For example, in an autonomous driving scenario, the data acquisition unit 10 can be a camera or radar on the autonomous driving vehicle, or other sample data acquisition devices independent of the autonomous driving vehicle, such as a mobile phone, a road sampling vehicle, or a scanner.

[0064] The visual simulation device 20 can be, for example, a cloud platform. In this scenario, the visual simulation device 20 can be implemented by one or more computing devices. The front-end interaction unit 201, asset storage unit 202, and computing unit 203 can be integrated into the same computing device or deployed on different computing devices, which is not limited in this embodiment of the present application.

[0065] The image processing method provided in the embodiments of the present application is explained below.

[0066] FIG2 is a flow chart of an image processing method provided in an embodiment of the present application. The method can be applied to the visual simulation device shown in FIG1. ​​As shown in FIG2, the method includes the following steps.

[0067] Step 201: Acquire a sample data set, where the sample data set includes a plurality of first sample data, and the plurality of first sample data are used to indicate imaging information of a first object at multiple viewing angles.

[0068] When performing visual simulation on the first object, it is necessary to first obtain a sample data set so as to render the first object at a desired viewing angle based on the sample data in the sample data set.

[0069] The first object may be any object in a scene. For example, the first object is a foreground object. In this scenario, the method provided in the embodiment of the present application can realize visual simulation of the foreground object in a scene.

[0070] In the embodiment of the present application, visual simulation can also be performed on multiple objects in a scene at the same time. In this scenario, the sample data set also includes multiple second sample data, and the multiple second sample data are used to indicate the imaging information of the second object from multiple perspectives.

[0071] The first object and the second object may be any two objects in a scene. For example, the first object is a foreground object, and the second object is a background object. In this scenario, the method provided in the embodiments of the present application can simultaneously achieve visual simulation of the foreground object and the background object in a scene.

[0072] The following explanation is given by taking the first object as a foreground image and the second object as a background object as an example.

[0073] For example, in an autonomous driving scenario, foreground objects may include dynamic vehicles around the autonomous driving vehicle, and background objects may include the road on which the autonomous driving vehicle is located, static buildings around the road, and the like.

[0074] In some embodiments, the sample data set includes a first sample data set and a second sample data set, the first sample data set includes a plurality of first sample data, and the second sample data set includes a plurality of second sample data.

[0075] In an embodiment of the present application, the first sample data for the foreground object and the second sample data for the background object can come from different data sources, so that the user can collect first sample data with a wider perspective and second sample data with a wider perspective based on needs to improve the realism of subsequent visual simulation results.

[0076] For example, in an autonomous driving scenario, first sample data of other vehicles on the road around the autonomous driving vehicle can be collected through a mobile phone or other camera device, and second sample data of the road around the autonomous driving vehicle can be collected through on-board sensors such as radar or cameras.

[0077] The user may subsequently upload the first sample data set and the second sample data set to the visual simulation device through the front-end interaction unit in FIG. 1 , so as to implement the image processing method provided in the embodiment of the present application through the visual simulation device.

[0078] When the user uploads the first sample dataset and the second sample dataset to the visual simulation device through the front-end interaction unit, the user needs to specify that the first sample dataset is a sample dataset for foreground objects and the second sample dataset is a sample dataset for background objects.

[0079] In some embodiments, the visual simulation apparatus receives object category information configured by a user, where the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

[0080] Figure 3 is a schematic diagram of an interface for user configuration of object types provided by an embodiment of the present application. As shown in Figure 3, the front-end interaction unit provides a front-end interaction interface, which displays a foreground object file import option and a background object file import option.

[0081] After the user clicks the foreground object file import option, the first sample data set is uploaded to the visual simulation device. The visual simulation device then confirms that the user has set the first object corresponding to the first sample data set as the foreground object. Similarly, after the user clicks the background object file import option, the second sample data set is uploaded to the visual simulation device. The visual simulation device then confirms that the user has set the second object corresponding to the second sample data set as the background object.

[0082] The above example takes the first sample data and the second sample data as coming from different data sources. Alternatively, in other embodiments, the sample data in the sample data set may be collected by the same sample data collection device without distinguishing between the first sample data and the second sample data.

[0083] For example, in an autonomous driving scenario, onboard sensors such as radar or cameras can collect sample data about the vehicle's surroundings (including background data such as the road and foreground data such as other dynamic vehicles). In this scenario, image recognition algorithms can then be used to separate foreground and background objects from the sample data, facilitating subsequent training of different NeRF models.

[0084] In addition, the sample data in the sample dataset may include sample images or sample point clouds, as well as the pose of the device that collected the sample data. For example, the sample dataset includes multiple sample images and the camera pose corresponding to each sample image. The camera pose corresponding to the sample image can be understood as the pose of the camera that collected the sample image. For another example, the sample dataset includes multiple point cloud data and the radar pose corresponding to each point cloud data. The radar pose corresponding to the point cloud data can be understood as the pose of the radar that collected the point cloud data.

[0085] Step 202: Determine a first object NeRF model and a first object geometric model based on a plurality of first sample data. The first object NeRF model is used to predict an arbitrary viewing angle image of the first object. The first object geometric model is a three-dimensional model of the first object.

[0086] In an embodiment of the present application, in order to simultaneously achieve high editability of visual simulation, high fidelity of visual simulation results, and low resource consumption of the visual simulation process, a visualized object geometry model and an object NeRF model that can be subsequently implicitly rendered can be simultaneously formed using a sample data set. The user edits the display properties of the object to be simulated, such as the display posture, using the visualized object geometry model, thereby achieving high editability of the visual simulation. At the same time, when rendering the visual simulation results, the image can be rendered directly using the object NeRF model, thereby achieving high fidelity of the visual simulation results and low resource consumption during the rendering process.

[0087] Optionally, in a scenario where the sample data set in step 201 also includes multiple second sample data, a second object NeRF model and a second object geometric model can also be determined based on the multiple second sample data. The second object NeRF model is used to predict an arbitrary perspective image of the second object, and the second object geometric model is a three-dimensional model of the second object.

[0088] The following example illustrates how to simultaneously determine the NeRF model and geometric model of the first object, the NeRF model and geometric model of the second object based on the sample data set. Implementations in other scenarios can refer to the following content and will not be described one by one in the following examples.

[0089] To facilitate understanding, let's first explain the principles of the NeRF model. For any object to be simulated, sample images of the object are collected from different perspectives, and the camera pose for each sample image is determined. These sample images and corresponding camera poses are then fed into the NeRF model for training. After training, the NeRF model can then be fed with two-dimensional images of the object from any perspective.

[0090] Based on this, in some embodiments, the implementation process of determining the first object NeRF model and the second object NeRF model based on the sample data set can be: training the first initialized NeRF model based on multiple first sample data in the sample data set to obtain the first object NeRF model, and training the second initialized NeRF model based on multiple second sample data in the sample data set to obtain the second object NeRF model.

[0091] After obtaining the NeRF model of the first object, a first mesh model can be constructed based on the NeRF model of the first object, and the first mesh model can be used as the first object geometric model. After obtaining the NeRF model of the second object, a second mesh model can be constructed based on the NeRF model of the second object, and the second mesh model can be used as the second object geometric model.

[0092] In other words, in this embodiment of the present application, the first object geometric model may include a first object mesh model, and the second object geometric model may include a second object mesh model. In this way, the user can subsequently edit the visual simulation requirements based on the visualized first object mesh model and the second object mesh model.

[0093] For example, in a scene where the first object is a foreground object and the second object is a background object, the first object geometric model includes a foreground object mesh model, and the second object geometric model includes a background object mesh model.

[0094] For example, the first object NeRF model may be a neural implicit surfaces (NeuS) model. In this scenario, after obtaining the first object NeRF model, a foreground mesh model may be constructed using a signed distance function (SDF) in the first object NeRF model.

[0095] As another example, the second object NeRF model may be a NeRF in the wild (NeRF-W) model. In this scenario, after obtaining the second object NeRF model, the output of the second object NeRF model may be processed using a marching cubes algorithm, for example, to construct a background mesh model.

[0096] It should be noted that the embodiments of the present application do not limit the types of the NeRF model of the first object and the NeRF model of the second object, nor do they limit the algorithms for constructing the geometric model of the first object and the second object. Any NeRF model and algorithm for constructing a geometric model can be applied in the embodiments of the present application.

[0097] Optionally, in a scenario where the sample data in the sample data set is sample point cloud data, in addition to obtaining the first object geometric model and the second object geometric model by constructing the first object mesh model and the second object mesh model, the cloud platform can also directly construct the first object point cloud model and the second object point cloud model through the sample point cloud data, and use the first object point cloud model as the first object geometric model and the second object point cloud model as the second object geometric model.

[0098] In this scenario, optionally, the cloud platform may also determine the geometric model of the first object by constructing a point cloud model of the first object, and determine the geometric model of the second object by constructing a mesh model of the second object.

[0099] In addition, before determining the first object NeRF model and the first object geometric model based on multiple first sample data, the first object sample data can also be processed to eliminate other interfering objects other than the first object in the first object sample data. This not only reduces the amount of calculation required to train the first object NeRF model, but also improves the realism of subsequent visual simulation results.

[0100] For example, in a scenario where the first object is a foreground object and the first sample data is a foreground sample image, the first sample data may be processed by processing the foreground sample image using a semantic segmentation algorithm to identify the foreground of the foreground sample image, thereby generating a foreground mask.

[0101] For example, in an autonomous driving scenario, for a foreground sample image, a semantic segmentation algorithm can be used to identify other dynamic vehicles and other foregrounds around the autonomous driving vehicle in the foreground sample image, thereby generating a mask of the other dynamic vehicles and other foregrounds.

[0102] Accordingly, before determining the second object NeRF model and the second object geometric model based on multiple second sample data, the second sample data can also be processed to eliminate other interfering objects in the second sample data except the second object. This not only reduces the amount of calculation required to train the second object NeRF model, but also improves the realism of subsequent visual simulation results.

[0103] For example, in a scenario where the second object is a background object and the second sample data is a background sample image, the second sample data may be processed by processing the background sample image using a semantic segmentation algorithm to identify the background of the background sample image, thereby generating a background mask.

[0104] For example, in an autonomous driving scenario, for a background sample image, a semantic segmentation algorithm can be used to identify the background such as the road surrounding the autonomous driving vehicle in the background sample image, thereby generating a mask of the background such as the road.

[0105] Optionally, if the first sample data and the second sample data are not distinguished in the sample data set, the sample data can be directly identified to obtain the foreground mask and background mask of the sample data, and then the first object NeRF model and the first object geometric model are determined through the foreground mask, and the second object NeRF model and the second object geometric model are determined based on the background mask.

[0106] Step 203: Displaying the first object geometric model on the front-end interactive interface.

[0107] In an embodiment of the present application, in order to improve the editability of the visual simulation process, a front-end interactive interface is provided to the user, and the first object geometric model constructed in step 202 is displayed on the front-end interactive interface, so that the user can use the front-end interactive interface to edit the first object at the angle at which it needs to be simulated, that is, to edit the display properties of the first object geometric model.

[0108] Optionally, in the scene where the second object geometric model is determined to exist in step 202, the second object geometric model can also be displayed on the front-end interactive interface, so that the user can also use the front-end interactive interface to edit the angle at which the second object needs to be simulated, that is, edit the display properties of the second object geometric model.

[0109] For example, in a scene where a first object is a foreground object and a second object is a background object, the geometric model of the first object and the geometric model of the second object are simultaneously displayed on the front-end interactive interface. The user can use the first and second geometric models to edit the position of the foreground object on the background object on the front-end interactive interface, thereby combining the foreground and background objects. The specific implementation method will be further described in the subsequent step 204.

[0110] Step 204: In response to the user's editing operation on the front-end interactive interface, determine the display attributes and rendering perspective pose of the first object geometric model, where the display attributes include the display pose.

[0111] In some embodiments, step 204 may be implemented by determining a display pose of the first object geometric model in response to a user's movement and / or rotation operation on the first object geometric model.

[0112] Optionally, in some other embodiments, the display attributes further include display size. In this scenario, step 204 may be implemented by: determining the display pose of the first object geometric model in response to a user's movement and / or rotation operation on the first object geometric model; and determining the display size of the first object geometric model in response to a user's scaling operation on the first object geometric model.

[0113] In addition, in the scenario where the front-end interactive interface also displays the second object geometric model in step 203, the display properties of the first object geometric model, the display properties of the second object geometric model and the rendering perspective pose can be determined in response to the user's editing operation on the front-end interactive interface.

[0114] The following explanation is given by taking the simultaneous determination of the display attributes of the first object's geometric model, the display attributes of the second object's geometric model, and the rendering perspective pose as an example.

[0115] Among them, in response to the user's editing operation on the front-end interactive interface, the display properties of the first object geometric model and the display properties of the second object geometric model can be determined as follows: in response to the user's movement and / or rotation operations on the first object geometric model and / or the second object geometric model, the display posture of the first object geometric model and the display posture of the second object geometric model are determined.

[0116] In an embodiment of the present application, the user can move and / or rotate the first object geometric model and / or the second object geometric model on the front-end interactive interface, thereby editing the display posture of the first object geometric model and / or the display posture of the second object geometric model.

[0117] For example, in a scene where a first object is a foreground object and a second object is a background object, the user can edit only the display pose of the first object's geometric model, without editing the display pose of the second object's geometric model. Alternatively, the user can edit both the display pose of the first object's geometric model and the display pose of the second object's geometric model simultaneously. Examples are not provided here.

[0118] In addition, the user can edit the display posture of a geometric model by moving the geometric model on the front-end interactive interface, by rotating the geometric model on the front-end interactive interface, or by simultaneously moving and rotating the geometric model on the front-end interactive interface.

[0119] For example, in a scene where the first object is a foreground object and the second object is a background object, when the user only edits the display pose of the geometric model of the first object, after the user moves and / or rotates the geometric model of the first object, the visual simulation device can extract the current display pose of the geometric model of the first object as the determined display pose of the geometric model of the first object. At the same time, the initial display pose of the geometric model of the second object on the front-end interactive interface is used as the determined display pose of the geometric model of the second object. The initial display poses of the geometric model of the first object and the geometric model of the second object displayed on the front-end interactive interface can be, for example, the origin position in the world coordinate system.

[0120] In other embodiments, the display properties may further include display size. In this scenario, in response to a user editing operation on the front-end interactive interface, the display properties of the first object geometric model and the display properties of the second object geometric model may be determined by: in response to a user scaling operation on the first object geometric model and / or the second object geometric model, the display size of the first object geometric model and the display size of the second object geometric model may be determined.

[0121] In an embodiment of the present application, the user can perform scaling operations on the first object geometric model and / or the second object geometric model on the front-end interactive interface, thereby editing the display size of the first object geometric model and / or the display size of the second object geometric model.

[0122] For example, in a scene where a first object is a foreground object and a second object is a background object, the user can edit only the display size of the first object's geometric model, without editing the display size of the second object's geometric model. Alternatively, the user can edit the display size of both the first and second objects' geometric models simultaneously. Examples are not provided here.

[0123] For example, when a user only edits the display size of a first object's geometric model, after the user scales the first object's geometric model, the visual simulation device may extract the size of the first object's geometric model as the determined display size of the first object's geometric model. Simultaneously, the display size of the second object's geometric model may be set to a default value, such as the size of the second object's geometric model currently displayed on the front-end interactive interface.

[0124] FIG4 is a schematic diagram of a front-end interactive interface provided in an embodiment of the present application. As shown in FIG4 , the front-end interactive interface displays roads, static buildings, and vehicles. The vehicles are the first object geometric model, and the roads and static buildings are the second object geometric models.

[0125] When the user moves, rotates or scales the vehicle on the front-end interactive interface, the display properties of the first object geometric model can be edited. As shown on the right side of Figure 4, after the user moves, rotates or scales the vehicle on the front-end interactive interface, the right window will display the display properties of the first object geometric model, which include display posture and display size. Among them, the display posture includes the position in the X / Y / Z three axes of the world coordinate system, and the deflection angle with the X / Y / Z three axes. The display size includes the ratio of the X / Y / Z three axes to the default size. Specifically, the position of the vehicle in the X / Y / Z three axes of the world coordinate system is edited by moving operations, the deflection angle of the vehicle with the X / Y / Z three axes is edited by rotating operations, and the ratio of the vehicle in the X / Y / Z three axes to the default size is edited by scaling operations.

[0126] Optionally, in other embodiments, the user can directly input the pose and size of the first object geometric model and the pose and size of the second object geometric model to be displayed in the front-end interactive interface, and the visual simulation device determines the display properties of the first object geometric model and the display properties of the second object geometric model based on the user input.

[0127] In addition, in some embodiments, in response to the user's editing operations on the front-end interactive interface, the implementation method of determining the rendering perspective posture can be: displaying a virtual rendering camera on the front-end interactive interface; and determining the rendering perspective posture in response to the user's movement and / or rotation operations on the virtual rendering camera.

[0128] In an embodiment of the present application, the user can move and / or rotate the virtual rendering camera on the front-end interactive interface, thereby editing the rendering perspective posture.

[0129] The virtual rendering camera may be, for example, a camera icon, so that the user can move and / or rotate the virtual rendering camera.

[0130] In addition, the rendering perspective pose edited by the user on the front-end interactive interface may include one perspective pose or multiple perspective poses.

[0131] When the rendered view pose includes a single view pose, the subsequent cloud platform only needs to render a single frame of image corresponding to the single view pose. When the rendered view pose includes multiple view poses, the subsequent cloud platform can render multiple frames of image based on the multiple view poses to form a simulated video based on the multiple frames of image.

[0132] Figure 5 is a schematic diagram of another front-end interactive interface provided in an embodiment of the present application. As shown in Figure 5 , the front-end interactive interface displays a road, static buildings, and vehicles. The vehicle is a first object geometric model, while the road and static buildings are second object geometric models. The front-end interactive interface also displays a virtual rendering camera.

[0133] By moving or rotating the virtual rendering camera on the front-end interactive interface, the user can edit the rendering perspective pose. As shown in Figure 5, the user places the virtual rendering camera at three positions. Based on these three positions, three perspective poses can be determined. In other words, the user-edited rendering perspective poses include three perspective poses. In Figure 5, these three perspective poses are referred to as perspective pose 1, perspective pose 2, and perspective pose 3.

[0134] As shown on the right side of Figure 5, when editing any perspective pose, after the user moves or rotates the virtual rendering camera on the front-end interactive interface, the right window will display the pose of the virtual rendering camera and use this pose as a perspective pose. The perspective pose includes the position of the virtual rendering camera in the X / Y / Z three-axis of the world coordinate system, as well as the deflection angle with respect to the X / Y / Z three-axis. Specifically, the position of the virtual rendering camera in the X / Y / Z three-axis of the world coordinate system is edited by moving the camera, and the deflection angle of the virtual rendering camera with respect to the X / Y / Z three-axis is edited by rotating the camera.

[0135] Optionally, in other embodiments, the user may directly input the desired rendering perspective pose in the front-end interactive interface, and the visual simulation device obtains the rendering perspective pose based on the user input.

[0136] Step 205: Determine at least one first object image based on the display properties of the first object geometric model, the rendering view pose, and the first object NeRF model.

[0137] In some embodiments, in a scenario where the sample data set also includes multiple second sample data, in step 205, the visual simulation device also determines at least one second object image based on the display properties, rendering perspective pose and NeRF model of the second object, and fuses at least one first object image and at least one second object image to obtain panoramic simulation data.

[0138] The following description will be made by taking panoramic simulation data as an example. The implementation of the visual simulation device rendering only at least one first object image can be referred to the following embodiment, which will not be described in detail.

[0139] For example, the implementation method of determining at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model can be: based on the display properties of the first object geometric model and the rendering perspective pose, determine the first rendering pose in the first coordinate system, the first coordinate system is the object coordinate system of the first object geometric model, and the rendering perspective pose is the pose in the world coordinate system; based on the first rendering pose and the NeRF model of the first object, determine at least one first object image.

[0140] Correspondingly, the implementation method of determining at least one second object image based on the display properties of the second object geometric model, the rendering perspective pose and the second object NeRF model can be: based on the display properties of the second object geometric model and the rendering perspective pose, determine the second rendering pose in the second coordinate system, the second coordinate system is the object coordinate system of the second object geometric model, and the rendering perspective pose is the pose in the world coordinate system; based on the second rendering pose and the NeRF model of the second object, determine at least one second object image.

[0141] The viewing pose required for the NeRF model to render the image is the viewing pose in the object coordinate system of the geometric model to be rendered, while the viewing pose edited by the user is usually the viewing pose in the world coordinate system. Therefore, after the visual simulation device obtains the rendering viewing pose based on step 204, it needs to convert the rendering viewing pose into a first rendering pose in the first coordinate system and a second rendering pose in the second coordinate system, that is, convert it into a first rendering pose aligned with the NeRF model of the first object and a second rendering pose aligned with the NeRF model of the second object.

[0142] The object coordinate system of the first object geometric model can be understood as the coordinate system of the first object geometric model itself, and the origin of the object coordinate system can be the center position of the first object geometric model, or other positions such as the center of gravity of the first object geometric model.

[0143] Based on step 204 , it can be known that the rendering perspective pose may include one perspective pose or multiple perspective poses.

[0144] When the rendering view pose includes a single view pose, the first and second rendering poses determined also include only a single rendering pose. Accordingly, based on the first rendering pose and the first object NeRF model, a first object image is determined. Based on the second rendering pose and the second object NeRF model, a second object image is determined. Ultimately, a panoramic image is generated through fusion.

[0145] When the rendering perspective poses include multiple perspective poses, the first rendering pose and the second rendering pose determined in this case each include multiple rendering poses. Accordingly, based on the first rendering pose and the first object NeRF model, multiple first object images are determined, and based on the second rendering pose and the second object NeRF model, multiple second object images are determined. Ultimately, after fusion, a multi-frame panoramic image is obtained. The visual simulation device can subsequently generate a simulated video based on these multi-frame panoramic images.

[0146] In this scenario, in some embodiments, based on the display properties of the first object geometric model and the rendering perspective pose, determining the first rendering pose in the first coordinate system can be achieved by converting multiple perspective poses into multiple rendering poses in the first coordinate system based on the display properties of the first object geometric model, and the multiple rendering poses constitute the first rendering pose.

[0147] Accordingly, determining at least one first object image based on the first rendering pose and the NeRF model of the first object may be implemented by: determining a first rendering trajectory based on multiple rendering poses; and determining multiple first object images based on the first rendering trajectory and the NeRF model of the first object.

[0148] The method of determining the first rendering trajectory based on the multiple rendering postures may be: determining the first rendering trajectory based on the multiple rendering postures through an interpolation algorithm. Alternatively, the multiple rendering postures may be directly combined as the first rendering trajectory.

[0149] In addition, the implementation method of determining the second rendering pose in the second coordinate system can refer to the above embodiment and will not be repeated here.

[0150] For example, the visual simulation device may convert multiple viewing angles into multiple rendering angles at a first coordinate based on the display properties of the first object's geometric model; determine a first rendering trajectory based on the multiple rendering angles at the first coordinate using an interpolation algorithm. Correspondingly, based on the display properties of the second object's geometric model, the multiple viewing angles may be converted into multiple rendering angles at a second coordinate; and determine a second rendering trajectory based on the multiple rendering angles at the second coordinate using an interpolation algorithm.

[0151] In the above manner, the user can only edit a few key viewpoint poses, and the visual simulation device obtains more rendering poses through an interpolation algorithm based on the several viewpoint poses edited by the user to form a rendering trajectory.

[0152] For example, if the video data to be simulated is 1 second long, but the user has only edited 5 view poses, assuming a frame rate of 30 frames per second, the visual simulation device can derive 5 rendering poses from the 5 edited view poses and interpolate another 25 rendering poses, resulting in a total of 30 rendering poses. These 30 rendering poses constitute a rendering trajectory.

[0153] As shown in FIG5 , the visual simulation device can determine a rendering trajectory based on the three viewing angles and poses edited by the user.

[0154] It should be noted that since the first rendering pose includes multiple rendering poses obtained by converting multiple viewpoint poses according to the first coordinate system, and the second rendering pose includes multiple rendering poses obtained by converting multiple viewpoint poses according to the second coordinate system, there is a one-to-one correspondence between the first rendering pose including multiple rendering poses and the second rendering pose including multiple rendering poses. In addition, to facilitate subsequent fusion, when interpolating the first rendering pose including multiple rendering poses and the second rendering pose including multiple rendering poses, the interpolation is performed according to the same rules, so that the rendering poses in the first rendering trajectory and the rendering poses in the second rendering trajectory also have a one-to-one correspondence.

[0155] Table 1 shows the correspondence between rendering poses in a first rendering trajectory and rendering poses in a second rendering trajectory, provided in an embodiment of the present application. As shown in Table 1, a user edits five viewing poses, and the visual simulation device can obtain five rendering poses in the first coordinate system and five rendering poses in the second coordinate system through coordinate system conversion.

[0156] Through the interpolation algorithm, rendering poses 6-11 can be interpolated between rendering pose 1 and rendering pose 2 in the first coordinate system, rendering poses 12-17 can be interpolated between rendering pose 2 and rendering pose 3 in the first coordinate system, rendering poses 18-23 can be interpolated between rendering pose 3 and rendering pose 4 in the first coordinate system, and rendering poses 24-30 can be interpolated between rendering pose 4 and rendering pose 5 in the first coordinate system.

[0157] Through the same interpolation algorithm, rendering poses 6'-11' can be interpolated between rendering pose 1' and rendering pose 2' in the second coordinate system, rendering poses 12'-17' can be interpolated between rendering pose 2' and rendering pose 3' in the second coordinate system, rendering poses 18'-23' can be interpolated between rendering pose 3' and rendering pose 4' in the second coordinate system, and rendering poses 24'-30' can be interpolated between rendering pose 4' and rendering pose 5' in the second coordinate system.

[0158] Based on this, the 30 rendering poses obtained in the first rendering trajectory correspond one-to-one with the 30 rendering poses in the second rendering trajectory. Each of them corresponds one-to-one to a specific rendering viewpoint pose in the world coordinate system. It should be noted that since some rendering poses are obtained through interpolation, the rendering viewpoint pose in the world coordinate system corresponding to the interpolated rendering pose is not the rendering viewpoint pose actually edited by the user.

[0159] Table 1

[0160] The following is an explanation using how to obtain the rendering pose in the first rendering pose as an example. For the implementation of the second rendering pose, please refer to the following content.

[0161] For example, in a scenario where the display properties only include display poses, the implementation method of converting multiple viewing poses into multiple rendering poses under the first coordinate based on the display properties of the first object geometric model can be: each viewing pose is converted into a rendering pose under the first coordinate based on the display pose of the first object geometric model through a coordinate system conversion algorithm.

[0162] The embodiment of the present application does not limit the implementation of the coordinate system conversion algorithm. For example, the viewing angle pose can be multiplied by the display pose of the first object geometric model, and the obtained pose is used as a rendering pose under the first coordinate.

[0163] As another example, in a scenario where the display attributes include display pose and display size, the implementation method of converting multiple perspective poses into multiple rendering poses under the first coordinate based on the display attributes of the first object geometric model can be: each perspective pose is converted into an initial rendering pose under the first coordinate based on the display pose of the first object geometric model through a coordinate system conversion algorithm, and then the initial rendering pose is adjusted according to the proportional relationship between the display size of the first object geometric model and the default display size to obtain the final rendering pose.

[0164] For example, when the rendering pose is expressed in a matrix format, after obtaining the initial rendering pose, the proportional relationship between the display size of the first object geometric model and the default display size is applied to the main diagonal of the initial rendering pose to obtain the final rendering pose.

[0165] Optionally, in other embodiments, based on the display properties of the first object geometric model and the rendering perspective pose, determining the first rendering trajectory in the first coordinate system and the second rendering trajectory in the second coordinate system can be implemented as follows: determining the initial rendering trajectory through an interpolation algorithm based on multiple perspective poses, the initial rendering trajectory including a larger number of perspective poses; based on the display properties of the first object geometric model, converting the perspective poses in the initial rendering trajectory into rendering poses in the first coordinate system, thereby obtaining the first rendering trajectory; based on the display properties of the second object geometric model, converting the perspective poses in the initial rendering trajectory into rendering poses in the second coordinate system, thereby obtaining the second rendering trajectory.

[0166] The rendering pose in the first rendering trajectory determined in this way also corresponds one-to-one to the rendering pose in the second rendering trajectory.

[0167] In addition, when the rendering view pose includes multiple view poses, since the rendering pose in the first rendering trajectory corresponds one-to-one with the rendering pose in the second rendering trajectory, the multiple first object images determined based on the first object NeRF model and the multiple second object images determined based on the second object NeRF model also correspond one-to-one, and each one corresponds to the same rendering view pose in the world coordinate system.

[0168] For example, if both the first rendering trajectory and the second rendering trajectory include 30 rendering poses, 30 first object images are generated based on the first object NeRF model, and 30 second object images are generated based on the second object NeRF model, and these 30 first object images correspond one to one to the 30 second object images.

[0169] In this scenario, the method for fusing multiple first object images and multiple second object images to obtain simulated video data can be: obtaining a first object image and a second object image corresponding to the first object image from the multiple first object images and the multiple second object images, the first object image including a first object red-green-blue (RGB) image and a first object depth image, and the second object image including a second object RGB image and a second object depth image; based on the first object depth image and the second object depth image, fusing the first object RGB image and the second object RGB image into a panoramic RGB image.

[0170] Based on the first object depth image and the second object depth image, the occlusion relationship between the first object and the second object at the same viewpoint can be determined, and then based on the occlusion relationship, it is determined whether the pixel value displayed at the viewpoint in the panoramic RGB image is the first object RGB image or the second object RGB image.

[0171] For example, for a certain viewpoint, if the depth corresponding to the first object depth image at that viewpoint is greater than the depth corresponding to the second object depth image, the pixel value at that viewpoint in the second object RGB image is displayed in the panoramic RGB image. Correspondingly, if the depth corresponding to the first object depth image at that viewpoint is less than the depth corresponding to the second object depth image, the pixel value at that viewpoint in the first object RGB image is displayed in the panoramic RGB image.

[0172] In a scene where the first object is a foreground object and the second object is a background object, further, based on the first object depth image and the second object depth image, after the first object RGB image and the second object RGB image are fused into a panoramic RGB image, the visual simulation device can also determine the light direction characteristics of the second object RGB image; and based on the light direction characteristics of the second object RGB image, generate a shadow of the foreground in the panoramic RGB image.

[0173] After the first object RGB image and the second object RGB image are fused based on the depth image, the shadow of the foreground can be further generated based on the light direction characteristics of the background, thereby improving the realism of the visual simulation result.

[0174] The above method can be implemented specifically through a shadow generation algorithm. It should be noted that the embodiment of the present application does not limit the shadow generation algorithm, and any shadow generation algorithm can be applied in the embodiment of the present application.

[0175] In a scene where the first object is a foreground object and the second object is a background object, further, based on the first object depth image and the second object depth image, after the first object RGB image and the second object RGB image are fused into a panoramic RGB image, the visual simulation device can also determine the light intensity characteristics of the second object RGB image; and adjust the color of the foreground in the panoramic RGB image based on the light intensity characteristics of the second object RGB image.

[0176] After the first object RGB image and the second object RGB image are fused based on the depth image, the color of the foreground may be further adjusted based on the light intensity characteristics of the background, thereby improving the realism of the visual simulation result.

[0177] The above-mentioned method can be specifically implemented by an image harmonization algorithm. It should be noted that the embodiment of the present application does not limit the image harmonization algorithm, and any image harmonization algorithm can be applied in the embodiment of the present application.

[0178] The above-mentioned steps of generating the shadow of the foreground in the panoramic RGB image and adjusting the color of the foreground in the panoramic RGB image may be performed simultaneously, or one of them may be performed separately, which will not be described one by one here.

[0179] Figure 6 is a schematic diagram of another image processing process provided by an embodiment of the present application. Figure 6 takes the first object as a foreground object and the second object as a background object as an example. As shown in Figure 6, the image processing process includes the following three steps.

[0180] Step 1: Prepare and create 3D assets. This includes collecting sample data, using the sample data to train a first-object NeRF model for the foreground object and a second-object NeRF model for the background object, and determining the first and second visual geometric models. For specific implementation details, refer to steps 201 and 202 above.

[0181] Step 2: Combining and editing the assets, including positioning the first object's geometric model, editing the rendering perspective pose, aligning the rendering perspective pose with the two object NeRF models, and determining the rendering trajectory. For specific implementation methods, please refer to the aforementioned steps 203, 204, and part of step 205.

[0182] Step 3: 3D rendering, including predicting the first object image using the first object NeRF model according to the rendering trajectory, predicting the second object image using the second object NeRF model according to the rendering trajectory, fusing the first object image with the second object image, etc. For specific implementation methods, please refer to part of the content in the aforementioned step 205.

[0183] In summary, in the embodiments of the present application, a sample dataset is used to simultaneously form a visual object geometry model and an object NeRF model that can be subsequently rendered implicitly. Users can edit the display properties of the simulated object, such as its display pose, using the visual object geometry model, thereby achieving high editability in the visual simulation. Furthermore, when rendering the visual simulation results, the NeRF model can be used directly to render the image, achieving high fidelity and low resource consumption during the rendering process.

[0184] The following is an explanation of the relevant devices and equipment provided in the embodiments of the present application.

[0185] The present application also provides an image processing device, as shown in FIG7 , wherein the device 700 includes:

[0186] The acquisition module 701 is used to acquire a sample data set, which includes a plurality of first sample data, and the plurality of first sample data are used to indicate imaging information of a first object from multiple viewing angles. For specific implementation, please refer to step 201 in the embodiment of FIG. 2 .

[0187] A first determination module 702 is configured to determine a neural radiation field (NeRF) model of a first object and a first object geometric model based on the plurality of first sample data. The first object NeRF model is used to predict arbitrary-viewpoint images of the first object, and the first object geometric model is a three-dimensional model of the first object. For specific implementation, see step 202 in the embodiment of FIG. 2 .

[0188] The display module 703 is configured to display the geometric model of the first object on the front-end interactive interface. Specific implementation methods can refer to step 203 in the embodiment of FIG. 2 .

[0189] The second determining module 704 is configured to determine the display attributes and rendering perspective pose of the first object geometric model in response to the user's editing operation on the front-end interactive interface, wherein the display attributes include the display pose. Specific implementation methods can refer to step 204 in the embodiment of FIG. 2 .

[0190] The third determining module 705 is configured to determine at least one first object image based on the display properties, rendering view pose, and NeRF model of the first object geometric model. Specific implementation methods may refer to step 205 in the embodiment of FIG. 2 .

[0191] Optionally, the third determining module is configured to:

[0192] Determine a first rendering pose in a first coordinate system based on display properties of the first object geometric model and a rendering perspective pose, where the first coordinate system is an object coordinate system of the first object geometric model, and the rendering perspective pose is a pose in a world coordinate system;

[0193] At least one first object image is determined based on the first rendering pose and the NeRF model of the first object.

[0194] Optionally, the rendering perspective pose includes a plurality of perspective poses, and the first rendering pose includes a plurality of rendering poses;

[0195] The third determination module is used to:

[0196] Determining a first rendering trajectory based on the multiple rendering poses;

[0197] Based on the first rendering trajectory and the NeRF model of the first object, a plurality of first object images are determined.

[0198] Optionally, the third determining module is configured to:

[0199] Based on the multiple rendering poses, a first rendering trajectory is determined by an interpolation algorithm.

[0200] Optionally, the sample data set further includes a plurality of second sample data, where the plurality of second sample data are used to indicate imaging information of the second object from multiple perspectives;

[0201] The first determining module is further configured to determine a second object NeRF model and a second object geometric model based on the plurality of second sample data, wherein the second object NeRF model is used to predict an arbitrary perspective image of the second object, and the second object geometric model is a three-dimensional model of the second object;

[0202] The display module is further used to display the second object geometric model on the front-end interactive interface;

[0203] The second determining module is further configured to determine the display attribute of the second object in response to an editing operation performed by the user on the front-end interactive interface;

[0204] The third determining module is further configured to determine at least one second object image based on the display properties, rendering view pose, and NeRF model of the second object;

[0205] The device also includes a fusion module for fusing at least one first object image and at least one second object image to obtain panoramic simulation data.

[0206] Optionally, the at least one first object image and the at least one second object image correspond one-to-one to the same rendering perspective pose in the at least one rendering perspective pose;

[0207] The Fusion Module is used to:

[0208] Acquire a first object image and a second object image corresponding to the first object image from at least one first object image and at least one second object image, wherein the first object image includes a first object red, green, and blue (RGB) image and a first object depth image, and the second object image includes a second object RGB image and a second object depth image;

[0209] The first object RGB image and the second object RGB image are fused into a panoramic RGB image based on the first object depth image and the second object depth image.

[0210] Optionally, the first object is a foreground object and the second object is a background object;

[0211] The device also includes:

[0212] The receiving module is configured to receive object category information configured by a user, where the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

[0213] Optionally, the display attributes further include display size;

[0214] The second determination module is used for:

[0215] In response to a user's movement and / or rotation operation on the first object geometric model, determining a display pose of the first object geometric model;

[0216] In response to a user's zooming operation on the first object geometric model, a display size of the first object geometric model is determined.

[0217] Optionally, the second determining module is configured to:

[0218] Display the virtual rendering camera on the front-end interactive interface;

[0219] In response to a user's movement and / or rotation operation on the virtual rendering camera, a rendering perspective pose is determined.

[0220] Optionally, the first object geometric model includes a first object mesh model.

[0221] The acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705 can all be implemented via software or hardware. For example, the implementation of the third determination module 705 will be described below using the third determination module 705 as an example. Similarly, the implementation of the acquisition module 701, the first determination module 702, the display module 703, and the second determination module 704 can refer to the implementation of the third determination module 705.

[0222] As an example of a software functional unit, the third determination module 705 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the third determination module 705 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0223] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0224] As an example of a hardware functional unit, the third determination module 705 may include at least one computing device, such as a server. Alternatively, the third determination module 705 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0225] The multiple computing devices included in the third determination module 705 can be distributed in the same region or in different regions. The multiple computing devices included in the third determination module 705 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the third determination module 705 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0226] It should be noted that, in other embodiments, the third determination module 705 can be used to execute any step in the image processing method, the acquisition module 701 can be used to execute any step in the image processing method, the first determination module 702 can be used to execute any step in the image processing method, the display module 703 can be used to execute any step in the image processing method, and the second determination module 704 can be used to execute any step in the image processing method. The steps that the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705 are responsible for implementing can be specified as needed. By having the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705 respectively implement different steps in the image processing method, the full functionality of the image processing device is achieved.

[0227] The present application also provides a computing device 800. As shown in FIG8 , computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. Processor 804, memory 806, and communication interface 808 communicate with each other via bus 802. Computing device 800 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 800.

[0228] Bus 802 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 804 may include a path for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, and communication interface 808).

[0229] The processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0230] The memory 806 may include volatile memory, such as random access memory (RAM). The processor 804 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0231] Memory 806 stores executable program code. Processor 804 executes the executable program code to implement the functions of the aforementioned acquisition module 701, first determination module 702, display module 703, second determination module 704, and third determination module 705, thereby implementing the image processing method. In other words, memory 806 stores instructions for executing the image processing method.

[0232] Alternatively, the memory 806 stores executable codes, and the processor 804 executes the executable codes to respectively implement the functions of the aforementioned image processing apparatus, thereby implementing the image processing method. That is, the memory 806 stores instructions for executing the image processing method.

[0233] The communication interface 803 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0234] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0235] As shown in Fig. 9, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the image processing method.

[0236] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store some instructions for executing the image processing method. In other words, the combination of one or more computing devices 800 can jointly execute the instructions for executing the image processing method.

[0237] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster can store different instructions, each for executing a portion of the functions of the image processing apparatus. In other words, the instructions stored in the memory 806 in different computing devices 800 can implement the functions of one or more of the acquisition module 701, the first determination module 702, the display module 703, the second determination module 704, and the third determination module 705.

[0238] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), etc. FIG. 10 illustrates a possible implementation. As shown in FIG. 10 , two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the acquisition module 701, the display module 703, and the second determination module 704. Simultaneously, the memory 806 in the computing device 800B stores instructions for executing the functions of the first determination module 702 and the third determination module 705.

[0239] The connection method between the computing device clusters shown in Figure 10 can be based on the consideration that the image processing method provided in this application needs to interact with the user and a large amount of calculation is required in the background. Therefore, it is considered that the functions implemented by the acquisition module 701, the display module 703 and the second determination module 704 are executed by the computing device 800A, and the functions implemented by the first determination module 702 and the third determination module 705 are executed by the computing device 800B.

[0240] It should be understood that the functionality of the computing device 800A shown in FIG10 may also be implemented by multiple computing devices 800. Similarly, the functionality of the computing device 800B may also be implemented by multiple computing devices 800.

[0241] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform an image processing method.

[0242] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a sample data set, where the sample data set includes a plurality of first sample data, where the plurality of first sample data are used to indicate imaging information of a first object from a plurality of viewing angles; Determine a first object neural radiation field NeRF model and a first object geometric model based on the plurality of first sample data, wherein the first object NeRF model is used to predict an arbitrary viewing angle image of the first object, and the first object geometric model is a three-dimensional model of the first object; Displaying the first object geometric model on the front-end interactive interface; In response to the user's editing operation on the front-end interactive interface, determining the display attributes and rendering perspective pose of the first object geometric model, wherein the display attributes include the display pose; At least one first object image is determined based on the display properties of the first object geometric model, the rendering view pose, and the first object NeRF model.

2. The method according to claim 1, characterized in that The determining at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model comprises: Determine a first rendering pose in a first coordinate system based on display properties of the first object geometric model and the rendering perspective pose, where the first coordinate system is an object coordinate system of the first object geometric model, and the rendering perspective pose is a pose in a world coordinate system; The at least one first object image is determined based on the first rendering pose and the NeRF model of the first object.

3. The method according to claim 2, characterized in that The rendering perspective pose includes a plurality of perspective poses, and the first rendering pose includes a plurality of rendering poses; The determining, based on the first rendering pose and the NeRF model of the first object, the at least one first object image comprises: Based on the multiple rendering poses, determining a first rendering trajectory; The plurality of first object images are determined based on the first rendering trajectory and a NeRF model of the first object.

4. The method according to claim 3, characterized in that The determining a first rendering trajectory based on the first rendering pose includes: Based on the multiple rendering poses, the first rendering trajectory is determined by an interpolation algorithm.

5. The method according to any one of claims 1 to 4, characterized in that: The sample data set further includes a plurality of second sample data, wherein the plurality of second sample data are used to indicate imaging information of a second object at multiple viewing angles; The method comprises: Determine a second object NeRF model and a second object geometric model based on the plurality of second sample data, wherein the second object NeRF model is used to predict an arbitrary viewing angle image of the second object, and the second object geometric model is a three-dimensional model of the second object; Displaying the second object geometric model on the front-end interactive interface; In response to the editing operation of the user on the front-end interactive interface, determining a display attribute of the second object; Determine at least one second object image based on the display attribute of the second object, the rendering view pose, and the NeRF model of the second object; The at least one first object image and the at least one second object image are fused to obtain panoramic simulation data.

6. The method according to claim 5, characterized in that The at least one first object image and the at least one second object image correspond one-to-one to the same rendering perspective pose in the at least one rendering perspective pose; The step of fusing the at least one first object image and the at least one second object image to obtain panoramic simulation data includes: Acquire a first object image and a second object image corresponding to the first object image from the at least one first object image and the at least one second object image, the first object image comprising a first object red, green, and blue (RGB) image and a first object depth image, and the second object image comprising a second object RGB image and a second object depth image; Based on the first object depth image and the second object depth image, the first object RGB image and the second object RGB image are fused into a panoramic RGB image.

7. The method according to claim 5 or 6, characterized in that The first object is a foreground object, and the second object is a background object; The method further comprises: Object category information configured by a user is received, where the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

8. The method according to any one of claims 1 to 7, characterized in that: The display attributes also include display size; The step of determining the display attribute of the first object geometric model in response to the user's editing operation on the front-end interactive interface includes: In response to the user's movement and / or rotation operation on the first object geometric model, determining a display posture of the first object geometric model; In response to the user's zooming operation on the first object geometric model, a display size of the first object geometric model is determined.

9. The method according to any one of claims 1 to 8, characterized in that: The step of determining the rendering perspective pose in response to the user's editing operation on the front-end interactive interface includes: Displaying a virtual rendering camera on the front-end interactive interface; In response to the user's movement and / or rotation operation on the virtual rendering camera, the rendering perspective pose is determined.

10. The method according to any one of claims 1 to 9, characterized in that: The first object geometric model includes a first object mesh model.

11. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a sample data set, wherein the sample data set includes a plurality of first sample data, and the plurality of first sample data are used for indicating imaging information of a first object at a plurality of viewing angles; A first determination module is used to determine a first object neural radiation field NeRF model and a first object geometric model based on the plurality of first sample data, wherein the first object NeRF model is used to predict an arbitrary viewing angle image of the first object, and the first object geometric model is a three-dimensional model of the first object; A display module, used for displaying the first object geometric model on a front-end interactive interface; A second determination module, configured to determine display attributes and rendering perspective pose of the first object geometric model in response to an editing operation of a user on the front-end interactive interface, wherein the display attributes include a display pose; The third determination module is used to determine at least one first object image based on the display properties of the first object geometric model, the rendering perspective pose and the first object NeRF model.

12. The device according to claim 11, characterized in that The third determination module is used for: Determine a first rendering pose in a first coordinate system based on display properties of the first object geometric model and the rendering perspective pose, where the first coordinate system is an object coordinate system of the first object geometric model, and the rendering perspective pose is a pose in a world coordinate system; The at least one first object image is determined based on the first rendering pose and the NeRF model of the first object.

13. The device according to claim 12, characterized in that The rendering perspective pose includes a plurality of perspective poses, and the first rendering pose includes a plurality of rendering poses; The third determination module is used for: Based on the multiple rendering poses, determining a first rendering trajectory; The plurality of first object images are determined based on the first rendering trajectory and a NeRF model of the first object.

14. The device according to claim 13, characterized in that The third determination module is used for: Based on the multiple rendering poses, the first rendering trajectory is determined by an interpolation algorithm.

15. The device according to any one of claims 11 to 14, characterized in that: The sample data set further includes a plurality of second sample data, wherein the plurality of second sample data are used to indicate imaging information of a second object at multiple viewing angles; The first determination module is further used to determine a second object NeRF model and a second object geometric model based on the plurality of second sample data, wherein the second object NeRF model is used to predict an arbitrary viewing angle image of the second object, and the second object geometric model is a three-dimensional model of the second object; The display module is further used to display the second object geometric model on the front-end interactive interface; The second determination module is further configured to determine a display attribute of the second object in response to the editing operation of the user on the front-end interactive interface; The third determination module is further used to determine at least one second object image based on the display attribute of the second object, the rendering perspective pose, and the NeRF model of the second object; The device also includes a fusion module, which is used to fuse the at least one first object image and the at least one second object image to obtain panoramic simulation data.

16. The device according to claim 15, characterized in that The at least one first object image and the at least one second object image correspond one-to-one to the same rendering perspective pose in the at least one rendering perspective pose; The fusion module is used to: Acquire a first object image and a second object image corresponding to the first object image from the at least one first object image and the at least one second object image, the first object image comprising a first object red, green, and blue (RGB) image and a first object depth image, and the second object image comprising a second object RGB image and a second object depth image; Based on the first object depth image and the second object depth image, the first object RGB image and the second object RGB image are fused into a panoramic RGB image.

17. The device according to claim 15 or 16, characterized in that The first object is a foreground object, and the second object is a background object; The device also includes: The receiving module is used to receive object category information configured by a user, wherein the object category information is used to indicate that the first object is a foreground object and the second object is a background object.

18. The device according to any one of claims 11 to 17, characterized in that: The display attributes also include display size; The second determining module is used for: In response to the user's movement and / or rotation operation on the first object geometric model, determining a display posture of the first object geometric model; In response to the user's zooming operation on the first object geometric model, a display size of the first object geometric model is determined.

19. The device according to any one of claims 11 to 18, characterized in that: The second determining module is used for: Displaying a virtual rendering camera on the front-end interactive interface; In response to the user's movement and / or rotation operation on the virtual rendering camera, the rendering perspective pose is determined.

20. The device according to any one of claims 11 to 19, characterized in that: The first object geometric model includes a first object mesh model.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 10.

Images