Processing method and device

Abstract

The invention discloses a processing method and device, and the method comprises the steps: receiving a control instruction which is used for controlling a virtual object to move in a virtual space; if the control instruction represents that the virtual object is controlled to enter the target area, the virtual object is limited outside the target area, a target view angle exists in the target area, and under the target view angle, the virtual space is rendered into an image with flaws.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a processing method and device. Background Art

[0002] With the development of computer vision technology, 3D reconstruction technology can achieve photorealistic rendering effects in visualization tasks such as free browsing and augmented reality. However, images with defects may still be rendered during the rendering process, affecting the user's interactive experience in visualization tasks. Summary of the invention

[0003] The technical solution of this application is as follows:

[0004] The first aspect of the present application provides a processing method, comprising:

[0005] receiving a control instruction, wherein the control instruction is used to control the movement of a virtual object in a virtual space;

[0006] If the control instruction represents controlling the virtual object to enter a target area, the virtual object is restricted outside the target area, and the target area has a target viewing angle. Under the target viewing angle, the virtual space is rendered as an image with defects.

[0007] The method further comprises:

[0008] Determining that the virtual object is located in a first area in the virtual space, rendering the virtual space into an image corresponding to a current viewing angle of the virtual object according to a first rendering strategy corresponding to the first area, the first area having the target viewing angle, and the first area being different from the target area;

[0009] Determine that the virtual object is located in a second area in the virtual space, render the virtual space into an image corresponding to the current perspective of the virtual object according to a second rendering strategy corresponding to the second area, the second area does not have the target perspective, and the first rendering strategy is different from the second rendering strategy.

[0010] The density of the target viewing angles in the first area is lower than the density of the target viewing angles in the target area.

[0011] The rendering of the virtual space into an image corresponding to the current viewing angle of the virtual object according to the first rendering strategy corresponding to the first area includes:

[0012] Determining an image quality of an image corresponding to a current viewing angle of the virtual object;

[0013] When the image quality meets the first quality condition, rendering the image corresponding to the current viewing angle of the virtual object according to a first method;

[0014] When the image quality reaches a second quality condition, the image corresponding to the current viewing angle of the virtual object is rendered according to a second method, and the first method is different from the second method.

[0015] The virtual space is a representation of a target three-dimensional model obtained by training an initial model according to image information in a target physical space; the image quality corresponding to the first quality condition is higher than the image quality corresponding to the second quality condition;

[0016] The rendering of the image corresponding to the current viewing angle of the virtual object according to the first method includes:

[0017] Rendering an image corresponding to the current viewing angle of the virtual object by using the target three-dimensional model;

[0018] The rendering of the image corresponding to the current viewing angle of the virtual object according to the second method includes at least one of the following:

[0019] Shielding rendering of an image corresponding to a current viewing angle of the virtual object;

[0020] Rendering an image corresponding to a current viewing angle of the virtual object based on the point cloud data corresponding to the virtual space;

[0021] An image corresponding to the current viewing angle of the virtual object is rendered according to the spatial structure data corresponding to the virtual space.

[0022] The method further comprises:

[0023] Obtaining regional parameters of each region in the virtual space; the regional parameters are used to detect the rendering quality corresponding to the region;

[0024] It is determined that the region parameters meet set conditions, and the region is determined as a target region; the set conditions indicate that there is a problem with the rendering quality corresponding to the region.

[0025] Obtaining the area parameters of each area in the virtual space includes:

[0026] Filtering out candidate areas in an unoccupied state from various areas in the virtual space;

[0027] Obtaining region parameters of each of the candidate regions in the virtual space.

[0028] Obtaining the area parameters of each area in the virtual space includes at least one of the following:

[0029] Pre-rendering each area in the virtual space to obtain a detection image corresponding to each area;

[0030] Using the target model to perform quality detection on the detection images corresponding to each of the regions, and obtain image quality parameters of each of the regions; the image quality parameters are used to characterize the quality of each pixel in the detection image;

[0031] Obtaining a camera trajectory; the camera trajectory is used to represent a path for photographing a target physical space;

[0032] Based on the camera trajectory, a common view parameter corresponding to each area in the virtual space is determined; the common view parameter is used to characterize the degree to which the area is observed by multiple camera perspectives.

[0033] Determining that the regional parameters meet the set conditions includes at least one of the following:

[0034] Determining that an image quality parameter of the region is lower than a set quality threshold;

[0035] It is determined that the common vision parameter is lower than a set common vision threshold.

[0036] Another aspect of the present application provides a processing device, comprising:

[0037] A receiving module, used for receiving a control instruction, wherein the control instruction is used for controlling the movement of a virtual object in a virtual space;

[0038] The control module is used for restricting the virtual object outside the target area if the control instruction represents controlling the virtual object to enter the target area, wherein the target area has a target viewing angle, and under the target viewing angle, the virtual space is rendered as an image with defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 A schematic diagram of a process for a treatment method provided in Example 1 of the present application;

[0041] Figure 2 A schematic diagram of a process for a treatment method provided in Example 2 of the present application;

[0042] Figure 3 A schematic diagram of a process for a treatment method provided in Example 4 of the present application;

[0043] Figure 4 A schematic diagram of a process for a treatment method provided in Example 6 of the present application;

[0044] Figure 5 A schematic diagram of a process for a treatment method provided in Example 7 of the present application;

[0045] Figure 6 A schematic diagram of an implementation scenario of a processing method provided in this application;

[0046] Figure 7 A schematic diagram of another implementation scenario of a processing method provided in this application;

[0047] Figure 8 A schematic diagram of the structure of a processing device provided in this application. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0049] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0050] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0051] With the rapid development of computer vision and deep learning technologies, especially the introduction and widespread application of Neural Radiance Fields (NeRF), 3D reconstruction technology has ushered in a revolutionary breakthrough. By combining deep learning models with volume rendering technology, NERF can reconstruct a three-dimensional scene with high realism and fine details from a series of two-dimensional images, achieving unprecedented photo-level rendering effects. The popularity of this technology is due to its use of widely available RGB camera data, which significantly lowers the threshold for technology application.

[0052] However, despite the remarkable achievements made by NERF in the field of 3D reconstruction, it still faces an important challenge in practical applications - the widespread existence of "floater" defects. These defects are due to the problem of missing or inaccurate data encountered by NERF during the training process. Specifically, due to factors such as occlusion and lighting changes in real physical scenes, the RGB camera data in some areas may not be fully captured, resulting in inaccurate learning of the radiation field information of the NERF model in these areas. When using the NERF model for virtual space rendering, using inaccurate radiation field information may render images with defects, affecting the overall quality and realism of the rendered images. Especially in visualization tasks that require free browsing and deep interaction, the negative impact of this problem is particularly prominent.

[0053] In order to alleviate this problem, a human-computer interaction post-editing method based on a graphical user interface (GUI) is often used in the prior art. This method allows the user to locally adjust and optimize the rendering results of NERF through an intuitive operating interface to reduce or eliminate defects in the image. Although this method can improve the image quality to a certain extent, due to the inherent perspective-dependent characteristics of NERF, that is, the rendering results under different perspectives may be significantly different, it is often difficult to fully and thoroughly remove all defects by relying solely on manual editing. In addition, improper operations during the editing process may accidentally destroy the rendering effect of the original non-degraded area, further increasing the complexity and uncertainty of the processing. Therefore, the present application provides a processing method to improve the overall quality of the image and the user's visual experience, and improve the user's interactive experience in visualization tasks.

[0054] Next, the present application will be further described in detail with reference to the accompanying drawings and specific implementation methods.

[0055] Reference Figure 1 , is a flow chart of a processing method provided in Example 1 of the present application, such as Figure 1 As shown, the method may include but is not limited to the following steps:

[0056] Step S101: receiving a control instruction, where the control instruction is used to control the movement of a virtual object in a virtual space.

[0057] In this embodiment, the virtual space may be constructed for a visualization task. For example, the visualization task may include but is not limited to: a free browsing task or an augmented reality interaction task.

[0058] In the free browsing task, users can interact with the virtual space through the browser, and they can freely change the perspective and position of virtual objects to explore different parts of the virtual space. For example, users can operate a virtual camera to observe the details in the virtual space by adjusting its position, angle and focal length.

[0059] In the free browsing task, users can generate control instructions by dragging the mouse, pressing keyboard keys, or sliding the touch screen.

[0060] In augmented reality interaction tasks, virtual objects can appear in the form of the user's virtual avatar. By controlling the actions of the virtual avatar, the user can explore and interact with the entire virtual space.

[0061] Users can control the actions of their avatars through a variety of input methods, including but not limited to VR controllers, gesture recognition, voice commands, etc. These input methods allow users to interact with the virtual space naturally and achieve complex actions such as walking, running, jumping, turning, and interacting with other virtual objects.

[0062] Step S102: If the control instruction represents controlling the virtual object to enter a target area, the virtual object is restricted outside the target area. The target area has a target viewing angle. Under the target viewing angle, the virtual space is rendered as an image with defects.

[0063] In the process of virtual space modeling, you may encounter quality problems (such as missing or blurred) in the collected real physical scene image data, missing data in the virtual space, unreasonable rendering settings, etc. These problems may cause some areas in the virtual space to be rendered as defective images from the target perspective. Image defects may appear as blur, artifacts, color distortion, etc., and image defects will affect the visual quality of the virtual space observed from the target perspective.

[0064] When the virtual object wants to enter the target area, the virtual object is restricted outside the target area by determining that the target area has the target viewing angle. That is, although the control instruction intends to make the virtual object enter the target area, this behavior can be prevented from occurring when it is confirmed that the target area has the target viewing angle.

[0065] In this embodiment, by receiving a control instruction, the control instruction is used to control the movement of a virtual object in a virtual space. If the control instruction represents controlling the virtual object to enter a target area, and the target area has a target viewing angle, the virtual object can be restricted outside the target area, and the virtual object can be prevented from entering an area that may cause rendering defects, thereby preventing an image with defects from being rendered under the target viewing angle, thereby improving the overall image quality and the user's visual experience, and improving the user's interactive experience in visualization tasks.

[0066] As another optional embodiment of the present application, refer to Figure 2 , is a flow chart of a processing method provided in Example 2 of the present application, such as Figure 2 As shown, the method may include but is not limited to the following steps:

[0067] Step S201: receiving a control instruction, where the control instruction is used to control the movement of a virtual object in a virtual space.

[0068] Step S202: If the control instruction represents controlling the virtual object to enter a target area, the virtual object is restricted outside the target area. The target area has a target viewing angle. Under the target viewing angle, the virtual space is rendered as an image with defects.

[0069] The detailed process of steps S201-S202 can refer to the relevant introduction of steps S101-S102 in Example 1, which will not be repeated here.

[0070] Step S203: determine that the virtual object is located in a first area in the virtual space, render the virtual space into an image corresponding to the current perspective of the virtual object according to a first rendering strategy corresponding to the first area, the first area has a target perspective, and the first area is different from the target area.

[0071] In the process of virtual space modeling, you may encounter quality problems (such as missing or blurred) in the collected real physical scene image data, missing data in the virtual space, unreasonable rendering settings, and other problems. These problems may cause the virtual space to be rendered as an image with defects in the target area and the first area in the virtual space under the target perspective. However, rendering in the first area is less affected by these problems than rendering in the target area. Therefore, although there is a target perspective in the first area of ​​the virtual space, the virtual space will be rendered as an image with defects under the target perspective, but these defects may be milder and less significant than in the target area, or have less impact on the user experience. Accordingly, relative to the target area, virtual objects can be allowed to enter the target area.

[0072] Since the first region has a target viewing angle, the first rendering strategy may include special rendering techniques related to handling defects to ensure that a certain visual quality is maintained when the defects are presented.

[0073] As the virtual object moves in the virtual space, the current location of the virtual object can be determined in real time. If the virtual object is located in a first area, the virtual space can be rendered into an image corresponding to the current viewing angle of the virtual object according to a first rendering strategy corresponding to the first area.

[0074] Step S204: determine that the virtual object is located in a second area in the virtual space, and render the virtual space into an image corresponding to the current perspective of the virtual object according to a second rendering strategy corresponding to the second area. The second area does not have a target perspective, and the first rendering strategy is different from the second rendering strategy.

[0075] If there is no target viewing angle in the second area, there is no need to specially deal with defects during rendering. The second rendering strategy can focus more on the overall clarity and detail performance of the virtual space, and can use higher precision and more complex rendering technology to present the virtual space.

[0076] In this embodiment, if the control instruction represents controlling the virtual object to enter the target area, the virtual object can be restricted outside the target area, which can prevent the virtual object from entering the area that may cause rendering defects and prevent the rendering of defective images under the target viewing angle.

[0077] In the first area, although the virtual space may be rendered as an image with defects, these defects are relatively minor or have little impact on the user experience, and if the first rendering strategy can handle these defects to ensure a certain visual quality, it can ensure that the user can continue to explore in the first area without being too disturbed.

[0078] In the second area, no matter what the viewing angle is, the virtual space will not be rendered as an image with defects. Users can enjoy higher quality visual effects and richer details, improving the user experience.

[0079] As another optional embodiment of the present application, a processing method is provided for Example 3 of the present application. This embodiment is mainly an implementation method of the first area in Example 2. In this implementation method, the density of the target viewing angle in the first area can be lower than the density of the target viewing angle in the target area.

[0080] The density of target views indicates how distributed the target views are. A high density of target views in an area means that rendering artifacts may be encountered when viewing from many different angles or positions in the area, while a low density means that artifacts are only encountered when viewing from a few specific angles or positions.

[0081] In the process of virtual space modeling, you may encounter quality problems (such as missing or blurred) in the collected real physical scene image data, missing data in the virtual space, unreasonable rendering settings, and other problems. These problems may cause the density of target perspectives in the target area and the first area in the virtual space to be different. Rendering in the first area is less affected by these problems than rendering in the target area. Therefore, the density of target perspectives in the first area can be lower than the density of target perspectives in the target area.

[0082] In the target area, the density of target perspectives is relatively high, which means that when a virtual object moves in the target area, the probability that the current perspective of the virtual object overlaps with the target perspective is relatively high. Therefore, in order to prevent users from experiencing the adverse effects of these defects, virtual objects can be restricted outside the target area, so as to ensure that users encounter as few rendering defects as possible when exploring the virtual space, thereby improving the overall user experience.

[0083] In the first area, the density of target perspectives is relatively low, which means that when the virtual object moves in the first area, the probability that the current perspective of the virtual object coincides with the target perspective is relatively low. Therefore, the virtual object can be allowed to enter the first area and rendered using the first rendering strategy to ensure a certain visual quality.

[0084] As another optional embodiment of the present application, refer to Figure 3 , is a flow chart of a processing method provided in Example 4 of the present application. This example is mainly an implementation method of step S203 in Example 2. Figure 3 As shown, step S203 may include but is not limited to the following steps:

[0085] Step S2031: Determine that the virtual object is located in a first area in the virtual space, and determine the image quality of an image corresponding to a current viewing angle of the virtual object.

[0086] In this embodiment, the first area may correspond to multiple rendering modes. When rendering, one of the multiple rendering modes may be selected for image rendering according to image quality. For example, the multiple rendering modes may include but are not limited to: a first mode and a second mode. The first mode and the second mode may differ in rendering accuracy, computational complexity, resource consumption, etc.

[0087] The rendering accuracy of the first method is higher than that of the second method.

[0088] In the first area, the first rendering method may be used by default. When the virtual object is located in the first area in the virtual space, the image corresponding to the current viewing angle of the virtual object may be rendered in the first manner, and the image quality parameter of the image corresponding to the current viewing angle of the virtual object rendered in the first manner is determined based on the image quality assessment algorithm. The image quality parameter may quantitatively represent the image quality of the image corresponding to the current viewing angle of the virtual object.

[0089] The image quality assessment algorithm can select an algorithm with small computational complexity and fast computational speed to ensure that the real-time performance of the rendering process is not affected.

[0090] Step S2032: When the image quality meets the first quality condition, render the image corresponding to the current viewing angle of the virtual object according to the first method.

[0091] Corresponding to an implementation method of determining image quality parameters of an image corresponding to a current perspective of a virtual object rendered in a first manner based on an image quality assessment algorithm, the image quality meeting the first quality condition may include but is not limited to: the image quality is higher than a set rendering quality threshold.

[0092] The rendering quality threshold can be set as needed and is not limited in this application.

[0093] When the image quality reaches the first quality condition, it means that the current rendering method is appropriate and there is no need to switch the rendering method. The image corresponding to the current perspective of the virtual object can continue to be rendered according to the first method to maintain the image quality.

[0094] Step S2033: When the image quality reaches a second quality condition, the image corresponding to the current viewing angle of the virtual object is rendered according to a second method, and the first method is different from the second method.

[0095] The second quality condition may be lower than the first quality condition.

[0096] When the image quality reaches the second quality condition, it may indicate that the rendering effect using the first method is not ideal. In order to reduce the adverse impact on user experience, the image corresponding to the current viewing angle of the virtual object may be rendered according to the second method.

[0097] Corresponding to an implementation manner of determining image quality parameters of an image corresponding to a current perspective of a virtual object rendered in a first manner based on an image quality assessment algorithm, the image quality meeting the second quality condition may include but is not limited to: the image quality is lower than a set rendering quality threshold.

[0098] In this embodiment, based on the dynamic evaluation of the image quality in the first area, the rendering process can flexibly adjust the rendering method according to different situations, and use a more appropriate rendering method in time, which can reduce the adverse impact on user experience caused by poor rendering effect, thereby improving user satisfaction.

[0099] As another optional embodiment of the present application, a processing method is provided for Example 5 of the present application. This embodiment is mainly an implementation method of steps S2032 and S2033 in Example 4. In this embodiment, the virtual space can be a representation of the target three-dimensional model obtained by training the initial model according to the image information in the target physical space.

[0100] The image information in the target physical space may come from multi-perspective two-dimensional images captured or acquired in the target physical space.

[0101] By training the initial model based on the image information in the target physical space, the target three-dimensional model can learn how to infer information such as the structure, texture, and lighting of the three-dimensional space from the image information of the target physical space.

[0102] When a target three-dimensional model is obtained through training, the target three-dimensional model can be used to generate a digital representation of a three-dimensional scene, namely a virtual space.

[0103] In this embodiment, the target three-dimensional model may include but is not limited to: a NeRF model. The NeRF model has a powerful rendering capability and can generate a high-quality virtual space.

[0104] The above step S2032 may include but is not limited to:

[0105] Step S20321: When the image quality reaches a first quality condition, an image corresponding to the current viewing angle of the virtual object is rendered using the target three-dimensional model.

[0106] As an advanced neural radiation field model, the NeRF model can learn and reconstruct a continuous virtual space from two-dimensional images from multiple angles. However, since the NeRF model may not be able to fully and accurately capture all radiation field information during the learning process, in some cases, the images rendered by the NeRF model may contain some defects, such as anomalies similar to floating objects.

[0107] When the corresponding image of the virtual object rendered by the NeRF model at the current viewing angle meets the first quality condition, it usually means that the defects in the image are small and within an acceptable range, so the NeRF model can continue to be used to render the image at the current viewing angle.

[0108] As the virtual object moves in the first area, the radiation field information learned by the NeRF model may have limitations, especially when the NeRF model is not accurate enough for the radiation field information at the current position, the rendered image may have defects. If the image quality of the image rendered by the NeRF model at the current position reaches the second quality condition, and the image quality corresponding to the first quality condition is higher than the image quality corresponding to the second quality condition, in order to obtain a higher quality rendering effect, other rendering methods can be used to replace the NeRF model to render the image at the current position.

[0109] Rendering the image corresponding to the current perspective of the virtual object according to the second method in the above step S2033 may include but is not limited to at least one of the following:

[0110] Step S20331: shield the rendering of the image corresponding to the current viewing angle of the virtual object.

[0111] For example, when the rendering result of the NeRF model contains so many defects that the image is almost unacceptable, shielded rendering can avoid showing low-quality images to users. Shielded rendering can avoid unnecessary calculations and save computing resources.

[0112] Step S20332: Rendering an image corresponding to the current viewing angle of the virtual object based on the point cloud data corresponding to the virtual space.

[0113] In this embodiment, a large amount of three-dimensional point data, namely, point cloud data, can be acquired by scanning or measuring the target physical space.

[0114] Point cloud data can accurately reflect information such as the shape, position, and surface characteristics of objects in real physical space. Rendering the image corresponding to the current perspective of the virtual object based on the point cloud data can ensure the quality of the rendered image to a certain extent.

[0115] Step S20333 renders the image corresponding to the current viewing angle of the virtual object according to the spatial structure data corresponding to the virtual space.

[0116] The spatial structure data corresponding to the virtual space can provide geometric information about objects, surfaces and volumes in three-dimensional space. When the NeRF model cannot accurately capture certain complex spatial structure information, rendering based on spatial structure data can provide images with better rendering effects.

[0117] In this embodiment, when the effect of rendering the image corresponding to the current perspective of the virtual object through the target three-dimensional model is not good, the target three-dimensional model can be replaced by various methods such as shielding rendering, rendering based on point cloud data, and rendering based on spatial structure data, so as to reduce the adverse effects on user experience caused by poor rendering effects, thereby improving user satisfaction.

[0118] As another optional embodiment of the present application, refer to Figure 4 , is a flow chart of a processing method provided in Example 6 of the present application, such as Figure 4 As shown, the method may include but is not limited to the following steps:

[0119] Step S301, obtaining regional parameters of each region in the virtual space; the regional parameters are used to detect the rendering quality corresponding to the region.

[0120] In this embodiment, the regional parameters of each region may be determined through preliminary calculation and analysis during the virtual space design or construction stage.

[0121] Step S302: Determine that the region parameters meet the set conditions, and determine the region as the target region; the set conditions indicate that there is a problem with the rendering quality corresponding to the region.

[0122] In this embodiment, during the virtual space design or construction stage, it can be determined that the region parameters meet the set conditions and the region is determined as the target region.

[0123] Of course, when the virtual object interacts with the virtual space and is about to enter a certain area, it is also possible to determine in real time whether the area parameters meet the set conditions to determine whether the area is the target area.

[0124] Step S303: receiving a control instruction, where the control instruction is used to control the movement of the virtual object in the virtual space;

[0125] Step S304: if the control instruction represents controlling the virtual object to enter a target area, the virtual object is restricted outside the target area. The target area has a target viewing angle. Under the target viewing angle, the virtual space is rendered as an image with defects.

[0126] In this embodiment, by determining the regional parameters of each area during the virtual space design or construction stage and determining the target area based on the regional parameters, computing resources in the subsequent user-virtual space interaction stage can be saved, thereby improving the interaction effect of the user-virtual space interaction stage.

[0127] As another optional embodiment of the present application, refer to Figure 5 , is a flow chart of a processing method provided in Example 7 of the present application. This example is mainly an implementation method of step S301 in Example 6. Figure 5 As shown, step S301 may include but is not limited to the following steps:

[0128] Step S3011: Filter out candidate areas that are unoccupied from various areas in the virtual space.

[0129] In this embodiment, an occupancy grid may be constructed for the virtual space. The occupancy grid divides the virtual space into a series of small, discrete cells. In the occupancy grid, each cell is marked as occupied or unoccupied, which is used to limit the activity area of ​​the virtual object, that is, the cell marked as unoccupied in the occupancy grid is the activity area, allowing the virtual object to move; the cell marked as occupied in the occupancy grid is the inactive area, and the virtual object is not allowed to enter.

[0130] Occupancy grids can be divided into two types: 2D occupancy grids and 3D occupancy grids, depending on their dimensions and application scenarios. Among them, 2D occupancy grids are more suitable for most indoor application scenarios due to their simplicity and intuitiveness; while 3D occupancy grids, with their powerful spatial expression capabilities, are mainly used in scenes that require three-dimensional spatial understanding, such as aerial photography.

[0131] The 2D occupancy grid plays a key role in human-computer interaction. It guides the virtual subject to move only in the "empty" area of ​​the horizontal surface, thus providing users with an immersive experience close to the real person's observation perspective. The 3D occupancy grid, while allowing users to freely explore in three-dimensional space, cleverly sets the closest and farthest distance limits between the virtual subject and the target content, ensuring that users can always stay within the best viewing range.

[0132] In this embodiment, the occupancy grid can be constructed based on the NeRF model reconstruction result. For the 2D occupancy grid, it also needs to rely on some information of the 3D occupancy grid, and its prerequisite is that the NERF reconstruction result has completed the alignment processing of the gravity axis.

[0133] Filtering out candidate areas in the virtual space that are unoccupied from various areas may include but is not limited to:

[0134] Traverse the occupancy grid and identify all cells marked as unoccupied;

[0135] The area in the virtual space corresponding to the unit marked as unoccupied in the occupancy grid is determined as a candidate area with a state of unoccupied.

[0136] In the occupancy grid, the cells marked as occupied cannot be entered by virtual objects themselves, and no target perspective will be generated. Therefore, when determining the area parameters, the cells marked as occupied in the occupancy grid can be safely eliminated.

[0137] Step S3012: Obtain area parameters of each candidate area in the virtual space.

[0138] The region parameters of the candidate region can be used to detect the rendering quality corresponding to the candidate region.

[0139] If the area parameters of the candidate area meet the set conditions, the candidate area is determined as the target area. Accordingly, the occupancy grid can be updated, and the cell corresponding to the candidate area in the occupancy grid is changed from being marked as unoccupied to being occupied. That is, even if the cell corresponding to the candidate area is not actually occupied, but the area parameters of the candidate area meet the set conditions, the virtual object will not be allowed to enter the candidate area. For example, Figure 6 As shown in the figure, the target area is actually not occupied but its area parameters meet the set conditions, and the target area is marked as an occupied unit in the updated occupancy grid; the inactive area is actually occupied, and the inactive area is marked as an occupied unit before and after the occupancy grid is updated. Accordingly, when the virtual object moves in the virtual space, the virtual object can be restricted outside the target area and the inactive area according to the updated occupancy grid.

[0140] In this embodiment, by screening out candidate regions in the virtual space that are unoccupied, unnecessary calculations can be reduced and the efficiency of determining region parameters can be improved.

[0141] As another optional embodiment of the present application, a processing method is provided for Embodiment 8 of the present application. This embodiment is mainly an implementation method of step S301 in Embodiment 6. Step S301 may include but is not limited to at least one of the following:

[0142] Step S3013: pre-render each area in the virtual space to obtain a detection image corresponding to each area.

[0143] In this embodiment, a rendering method can be selected as needed to pre-render each area in the virtual space. For example, at least one of the first method and the second method can be selected to pre-render each area in the virtual space.

[0144] Step S3014: Use the target model to perform quality detection on the detection image corresponding to each area to obtain image quality parameters of each area; the image quality parameters are used to characterize the quality of each pixel in the detection image.

[0145] In this embodiment, the target model can be an artificial intelligence model, which can perform quality detection on the detection images corresponding to each area to obtain image quality parameters of each area. The image quality parameters can quantify the quality of each pixel in the detection image.

[0146] Step S3015, obtaining a camera trajectory; the camera trajectory is used to represent a path for photographing the target physical space.

[0147] In this embodiment, one or more cameras may be selected to shoot the target physical space as needed. If there are multiple cameras, the camera trajectory of each camera may be obtained respectively.

[0148] Step S3016: Based on the camera trajectory, determine the common view parameters corresponding to each area in the virtual space; the common view parameters are used to characterize the degree to which the area is observed by multiple camera perspectives.

[0149] If there is one camera, the common view parameter may include but is not limited to: at least one of the number of angles seen by the camera (an implementation of multiple camera view angles) and the number of times the camera is seen. The greater the number of angles seen by the camera, the higher the degree to which the area is commonly observed by multiple camera view angles; the greater the number of times the camera is seen, the higher the degree to which the area is commonly observed by multiple camera view angles.

[0150] If there are multiple cameras, the common view parameter may include but is not limited to: at least one of the number of times the area is seen by multiple cameras (how many different cameras observe the area at least once) and the number of times the area is seen by multiple cameras at the same time (the area is observed by multiple cameras at the same time).

[0151] The higher the common view parameter of an area, the more it is observed by multiple camera perspectives, so the rendering result in the virtual space may be more accurate and reliable. On the contrary, if the common view parameter of an area is low, it may be observed by only a few camera perspectives, or the observation angle and time are limited, which may lead to errors or uncertainties when rendering in the virtual space.

[0152] In this embodiment, both the image quality parameter and the common view parameter can reflect the rendering effect of the image. What are the benefits of using these two parameters as region parameters?

[0153] As another optional embodiment of the present application, a processing method is provided for Embodiment 9 of the present application. This embodiment is mainly an implementation method for determining that the regional parameters meet the set conditions in Embodiment 6. Determining that the regional parameters meet the set conditions may include but is not limited to at least one of the following:

[0154] Step S11: Determine whether the image quality parameter of the region is lower than a set quality threshold.

[0155] In this embodiment, the quality threshold can be set as needed and is not limited in this application.

[0156] When the image quality parameter of a certain area is lower than the set quality threshold, it means that the image quality of the area during the rendering process fails to meet the expected standards and the rendering effect is poor.

[0157] Step S12: determining that the common view parameter is lower than a set common view threshold.

[0158] In this embodiment, the common view threshold can be set as needed and is not limited in this application.

[0159] If the common view parameter of a certain area is lower than the set common view threshold, it means that there is a problem with the visibility or consistency of the area under multiple camera perspectives. This may cause visual inconsistency or blur when the observer observes the area from different angles, thus affecting the overall rendering effect and user experience. Therefore, the area can be determined as the target area.

[0160] Next, combine Figure 7 , the processing method provided in this embodiment is described. For example, Figure 7 As shown, in the virtual space construction stage (ie, pre-calculation stage):

[0161] A transparency field is generated based on the reconstruction result of the NeRF model. The transparency field can be constructed according to the application scenario to generate a 3D occupancy grid and / or a 2D occupancy grid, wherein each cell in the generated 3D occupancy grid or 2D occupancy grid is marked as occupied or unoccupied.

[0162] The area of ​​the virtual space corresponding to the cells marked as unoccupied in the occupancy grid is taken as the candidate area.

[0163] Based on the NeRF model, pre-rendering is performed in the candidate area to obtain a detection image, and the quality of the detection image is detected using an artificial intelligence (AI) model to obtain the image quality parameters of the candidate area.

[0164] And / or, obtaining a NeRF camera trajectory, and determining a common view parameter corresponding to the candidate area based on the NeRF camera trajectory.

[0165] If the image quality parameter of the candidate area is lower than the set quality threshold, and / or the common view parameter of the candidate area is lower than the set common view threshold, the candidate area can be determined as the target area, and the cells in the occupancy grid corresponding to the candidate area are changed from being marked as unoccupied to occupied, to obtain an updated occupancy grid.

[0166] During the phase where virtual objects interact in virtual space (i.e., real-time operation phase):

[0167] Receive a control instruction, the control instruction is used to control the virtual object to enter a certain area, and determine whether the area is a target area through the occupancy of the cells in the occupancy grid corresponding to the area. If the cells in the occupancy grid corresponding to the area are marked as occupied, determine that the area is the target area, and restrict the virtual object outside the target area.

[0168] If the cells in the occupancy grid corresponding to the area are marked as unoccupied, the area is determined to be the first area or the second area. In the first area or the second area, the image quality of the image corresponding to the current viewing angle of the virtual object is determined, and when the image quality is defective (that is, the image quality meets the first quality condition), the image corresponding to the current viewing angle of the virtual object is rendered according to the NeRF model; when the image quality is flawless or the defects are negligible (that is, the image quality meets the second quality condition), the rendering is shielded, based on the point cloud data, or based on the spatial structure data.

[0169] Next, the processing device provided in the present application is introduced. The processing device introduced below and the processing method introduced above can refer to each other.

[0170] like Figure 8 As shown, the processing device includes: a receiving module 100 and a control module 200.

[0171] The receiving module is used to receive a control instruction, where the control instruction is used to control the movement of the virtual object in the virtual space.

[0172] The control module is used for restricting the virtual object outside the target area if the control instruction represents controlling the virtual object to enter the target area, wherein the target area has a target viewing angle, and under the target viewing angle, the virtual space is rendered as an image with defects.

[0173] The processing device may also include:

[0174] The first rendering module is used to determine that the virtual object is located in a first area in the virtual space, and render the virtual space into an image corresponding to the current perspective of the virtual object according to a first rendering strategy corresponding to the first area, the first area has the target perspective, and the first area is different from the target area.

[0175] A second rendering module is used to determine that the virtual object is located in a second area in the virtual space, and render the virtual space into an image corresponding to the current perspective of the virtual object according to a second rendering strategy corresponding to the second area, the second area does not have the target perspective, and the first rendering strategy is different from the second rendering strategy.

[0176] The density of the target viewing angles in the first area may be lower than the density of the target viewing angles in the target area.

[0177] The first rendering module can be used for:

[0178] Determining an image quality of an image corresponding to a current viewing angle of the virtual object;

[0179] When the image quality meets the first quality condition, rendering the image corresponding to the current viewing angle of the virtual object according to a first method;

[0180] When the image quality reaches a second quality condition, the image corresponding to the current viewing angle of the virtual object is rendered according to a second method, and the first method is different from the second method.

[0181] In this embodiment, the virtual space may be a representation of a target three-dimensional model obtained by training an initial model based on image information in a target physical space; the image quality corresponding to the first quality condition may be higher than the image quality corresponding to the second quality condition.

[0182] The first rendering module renders the image corresponding to the current viewing angle of the virtual object according to the first method, which may include:

[0183] Rendering an image corresponding to the current viewing angle of the virtual object by using the target three-dimensional model;

[0184] The first rendering module rendering the image corresponding to the current perspective of the virtual object according to the second method may include at least one of the following:

[0185] Shielding rendering of an image corresponding to a current viewing angle of the virtual object;

[0186] Rendering an image corresponding to a current viewing angle of the virtual object based on the point cloud data corresponding to the virtual space;

[0187] An image corresponding to the current viewing angle of the virtual object is rendered according to the spatial structure data corresponding to the virtual space.

[0188] The processing device may also include:

[0189] The obtaining module is used to obtain the regional parameters of each region in the virtual space; the regional parameters are used to detect the rendering quality corresponding to the region.

[0190] The determination module is used to determine that the area parameters meet the set conditions and determine the area as the target area; the set conditions indicate that there is a problem with the rendering quality corresponding to the area.

[0191] Get the module, which can be used for:

[0192] Filtering out candidate areas in an unoccupied state from various areas in the virtual space;

[0193] Obtaining region parameters of each of the candidate regions in the virtual space.

[0194] The obtaining module obtains the area parameters of each area in the virtual space, which may include at least one of the following:

[0195] Pre-rendering each area in the virtual space to obtain a detection image corresponding to each area;

[0196] Using the target model to perform quality detection on the detection images corresponding to each of the regions, and obtain image quality parameters of each of the regions; the image quality parameters are used to characterize the quality of each pixel in the detection image;

[0197] Obtaining a camera trajectory; the camera trajectory is used to represent a path for photographing a target physical space;

[0198] Based on the camera trajectory, a common view parameter corresponding to each area in the virtual space is determined; the common view parameter is used to characterize the degree to which the area is observed by multiple camera perspectives.

[0199] Determining that the regional parameters meet the set conditions may include at least one of the following:

[0200] Determining that an image quality parameter of the region is lower than a set quality threshold;

[0201] It is determined that the common vision parameter is lower than a set common vision threshold.

[0202] In another embodiment of the present application, an electronic device is provided, comprising at least one processor and a memory connected to the processor, wherein:

[0203] The memory is used to store computer programs;

[0204] The processor is used to execute the computer program so that the electronic device can implement the processing method introduced in any one of the above embodiments 1 to 9.

[0205] In another embodiment of the present application, a computer storage medium is provided, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the processing method introduced in any one of the above embodiments 1 to 9.

[0206] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0207] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0208] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0209] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A processing method comprising: receiving a control instruction, wherein the control instruction is used to control the movement of a virtual object in a virtual space; If the control instruction represents controlling the virtual object to enter a target area, the virtual object is restricted outside the target area, and the target area has a target viewing angle. Under the target viewing angle, the virtual space is rendered as an image with defects.

2. The method according to claim 1, further comprising: Determining that the virtual object is located in a first area in the virtual space, rendering the virtual space into an image corresponding to a current viewing angle of the virtual object according to a first rendering strategy corresponding to the first area, the first area having the target viewing angle, and the first area being different from the target area; Determine that the virtual object is located in a second area in the virtual space, render the virtual space into an image corresponding to the current perspective of the virtual object according to a second rendering strategy corresponding to the second area, the second area does not have the target perspective, and the first rendering strategy is different from the second rendering strategy. 3 . The method according to claim 2 , wherein a density of the target viewing angles in the first area is lower than a density of the target viewing angles in the target area.

4. The method according to claim 2, wherein the step of rendering the virtual space into an image corresponding to the current viewing angle of the virtual object according to the first rendering strategy corresponding to the first area comprises: Determining an image quality of an image corresponding to a current viewing angle of the virtual object; When the image quality meets the first quality condition, rendering the image corresponding to the current viewing angle of the virtual object according to a first method; When the image quality reaches a second quality condition, the image corresponding to the current viewing angle of the virtual object is rendered according to a second method, and the first method is different from the second method.

5. The method according to claim 4, wherein the virtual space is a representation of a target three-dimensional model obtained by training an initial model according to image information in the target physical space; and the image quality corresponding to the first quality condition is higher than the image quality corresponding to the second quality condition; The rendering of the image corresponding to the current viewing angle of the virtual object according to the first method includes: Rendering an image corresponding to the current viewing angle of the virtual object by using the target three-dimensional model; The rendering of the image corresponding to the current viewing angle of the virtual object according to the second method includes at least one of the following: Shielding rendering of an image corresponding to a current viewing angle of the virtual object; Rendering an image corresponding to a current viewing angle of the virtual object based on the point cloud data corresponding to the virtual space; An image corresponding to the current viewing angle of the virtual object is rendered according to the spatial structure data corresponding to the virtual space.

6. The method according to claim 1, further comprising: Obtaining area parameters of each area in the virtual space; The region parameter is used to detect the rendering quality corresponding to the region; Determining that the area parameters meet the set conditions, and determining the area as the target area; The setting condition indicates that there is a problem with the rendering quality corresponding to the area.

7. The method according to claim 6, obtaining the area parameters of each area in the virtual space comprises: Filtering out candidate areas in an unoccupied state from various areas in the virtual space; Obtaining region parameters of each of the candidate regions in the virtual space.

8. The method according to claim 6, obtaining the area parameters of each area in the virtual space comprises at least one of the following: Pre-rendering each area in the virtual space to obtain a detection image corresponding to each area; Using the target model to perform quality detection on the detection images corresponding to each of the regions, and obtain image quality parameters of each of the regions; the image quality parameters are used to characterize the quality of each pixel in the detection image; Obtaining a camera trajectory; the camera trajectory is used to represent a path for photographing a target physical space; Based on the camera trajectory, a common view parameter corresponding to each area in the virtual space is determined; the common view parameter is used to characterize the degree to which the area is observed by multiple camera perspectives.

9. The method according to claim 8, wherein determining that the regional parameters meet the set conditions comprises at least one of the following: Determining that an image quality parameter of the region is lower than a set quality threshold; It is determined that the common vision parameter is lower than a set common vision threshold.

10. A processing device comprising: A receiving module, used for receiving a control instruction, wherein the control instruction is used for controlling the movement of a virtual object in a virtual space; The control module is used for restricting the virtual object outside the target area if the control instruction represents controlling the virtual object to enter the target area, wherein the target area has a target viewing angle, and under the target viewing angle, the virtual space is rendered as an image with defects.

Images