Image processing method, electronic device, storage medium and computer program product

Abstract

The application provides an image processing method, an electronic device, a storage medium and a computer program product. The method comprises: determining at least one target sampling point corresponding to any pixel of a to-be-generated image; determining a plurality of target regions corresponding to a plurality of scales of a receptive field one by one in a target element space based on the position of the target sampling point; performing sparse convolution of the feature values of the element points in the target region in the target element space at a corresponding scale to obtain the feature values of the target sampling point at the corresponding scale; splicing the feature values of the target sampling point at a plurality of scales together to obtain target input features; inputting the target input features into a target network model to obtain color information corresponding to the target sampling point output by the target network model; and determining the color information of the pixel based on the color information corresponding to each of the at least one target sampling point to obtain the to-be-generated image. The calculation redundancy can be reduced.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically to an image processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] In the field of image processing, it is often necessary to compress the original 2D image or render a new 2D image based on a 3D model. During the generation of a new 2D image, the image data is usually highly sparsity-based, resulting in slow generation speed and large memory consumption. Summary of the Invention

[0003] This application is made in view of the above-mentioned problems. This application provides an image processing method, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of this application, an image processing method is provided, comprising: for any pixel of an image to be generated, determining at least one target sampling point corresponding to the pixel, wherein the at least one target sampling point is a sampling point on the incident light path of the three-dimensional rendering corresponding to the pixel or the at least one target sampling point is the pixel itself; for each of the at least one target sampling point, determining multiple target regions in a target element space based on the position of the target sampling point, wherein the multiple target regions correspond one-to-one with receptive fields of multiple scales, and any target region is the region closest to the target sampling point in the convolution region of the receptive field of the corresponding scale in the target element space, wherein the target element space includes multiple element points, each element point corresponding to a feature value; for each of the multiple target regions... The feature values ​​of element points located within the target region are sparsely convolved at the corresponding scale to obtain the feature values ​​of the target sampling point at the corresponding scale. The target element space includes multiple grids, and the coverage areas of different grids do not overlap. The element points are three-dimensional voxels or two-dimensional pixels. Any convolution region corresponding to the receptive field at any scale includes one or more grids in the target element space. The feature values ​​of the target sampling point at multiple scales are concatenated to obtain the target input features. The target input features are input into the target network model to obtain the color information corresponding to the target sampling point output by the target network model. The color information of the pixel is determined based on the color information corresponding to at least one target sampling point to obtain the image to be generated.

[0005] For example, there are at least two types of meshes of different scales in the target element space.

[0006] For example, performing sparse convolution on the feature values ​​of element points located within the target region at a corresponding scale to obtain the feature values ​​of the target sampling point at the corresponding scale includes: obtaining a sparse convolution lookup table associated with the target region; determining at least one element point within the target region participating in sparse convolution based on the sparse convolution lookup table; for each of the at least one element point, calculating the weight corresponding to the element point based on the distance between the element point and the target sampling point; and performing a weighted summation of the feature values ​​of the at least one element point based on the weights corresponding to each of the at least one element point to obtain the feature values ​​of the target sampling point at the corresponding scale.

[0007] For example, the element point is a three-dimensional voxel point, the target element space is a three-dimensional voxel space, and the number of at least one target sampling point is multiple. Before determining at least one target sampling point corresponding to any pixel of the image to be generated, the method further includes: acquiring target pose information; determining at least one target sampling point corresponding to the pixel includes: determining a three-dimensional rendering incident light path based on the target pose information; determining the sampling point on the three-dimensional rendering incident light path as at least one target sampling point; and concatenating the feature values ​​of the target sampling point at multiple scales to obtain target input features includes: concatenating the feature values ​​of the target sampling point at multiple scales with the target pose information to obtain target input features.

[0008] For example, the target element space and the target network model are trained and obtained by: acquiring sample images; iteratively performing optimization operations based on the sample images until the total loss term meets the requirements, the optimization operations including: for any pixel in the sample image, determining at least one sample sampling point corresponding to the pixel, the sample sampling point being of the same type as the target sampling point; for each of the at least one sample sampling point, determining multiple sample regions in the current element space based on the position of the sample sampling point, the multiple sample regions corresponding one-to-one with receptive fields of multiple scales, any sample region being the region closest to the sample sampling point in the convolutional region of the corresponding scale receptive field in the current element space; for each of the multiple sample regions, performing sparse convolution of the feature values ​​of the element points located in the sample region at the corresponding scale to obtain the feature values ​​of the sample sampling point at the corresponding scale; The feature values ​​of the sampled points at multiple scales are concatenated to obtain the sample input features; the sample input features are input into the current network model to obtain the color information corresponding to the sampled points output by the current network model; the color information of the pixel is determined based on the color information corresponding to at least one sampled point to obtain the predicted image; the total loss term is calculated based on the color information of the sample image and the color information of the predicted image, and / or based on the feature values ​​of all element points in the current element space; the feature values ​​of all element points in the current element space and the parameters of the current network model are optimized based on the total loss term; wherein, the target element space and the target network model are the current element space and the current network model at the end of the training operation, and before the training operation begins, the initial element space is determined as the current element space and the initial network model is determined as the current network model.

[0009] For example, calculating the total loss term based on the color information of the sample image and the color information of the predicted image, and based on the feature values ​​of all element points in the current element space, includes: calculating a prediction loss term based on the color information of the sample image and the color information of the predicted image; calculating a penalty loss term based on the feature values ​​of all element points in the current element space; and weighted summing the prediction loss term and the penalty loss term to obtain the total loss term.

[0010] For example, for any receptive field of multiple scales, the receptive field has a corresponding sparse convolution lookup table in each convolution region. The sparse convolution lookup table is used to record the element points participating in sparse convolution. After optimizing the feature values ​​of all element points in the current element space and the parameters of the current network model based on the total loss term, the optimization operation further includes: setting the feature values ​​of element points whose feature values ​​are lower than a first feature value threshold to 0; for any receptive field of multiple scales, updating any sparse convolution lookup table corresponding to the receptive field to delete the element points whose feature values ​​are set to 0 from the sparse convolution lookup table.

[0011] For example, all grids in the initial element space have the same scale. The optimization operation further includes: if the sparsity ratio of the current element space relative to the element space after the previous split is less than or equal to the ratio threshold, for each element point in at least some of the element points in the current element space, the grid space corresponding to the element point is split into multiple parts in an octree or quadtree manner, and the feature value of the element point is copied to the element points corresponding to the multiple split grid spaces respectively to obtain a new split element space.

[0012] For example, for any receptive field of a variety of scales, the receptive field has a corresponding sparse convolution lookup table in each convolution region. The sparse convolution lookup table is used to record the element points participating in sparse convolution. The optimization operation also includes: for any receptive field of a variety of scales, updating any sparse convolution lookup table corresponding to the receptive field to add new element points that fall into the convolution region corresponding to the sparse convolution lookup table after splitting.

[0013] According to another aspect of this application, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the above-described image processing method.

[0014] According to another aspect of this application, a storage medium is provided on which program instructions are stored, which, when run, are used to execute the above-described image processing method.

[0015] According to another aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when run, performs the above-described image processing method.

[0016] The image processing method, electronic device, storage medium, and computer program product according to embodiments of this application can perform sparse convolutions in the same target element space using receptive fields of multiple scales. This sparse convolution scheme can reduce computational redundancy, thereby reducing the consumption of storage and computing resources and improving the speed of image processing. Furthermore, this scheme can share the same target element space, thus further reducing storage overhead. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A schematic block diagram of an example electronic device for implementing the image processing method and apparatus according to embodiments of this application is shown;

[0019] Figure 2 A schematic flowchart illustrating an image processing method according to an embodiment of this application is shown;

[0020] Figure 3 A schematic diagram of an element space according to an embodiment of this application is shown;

[0021] Figure 4a A schematic diagram of an element space according to an embodiment of this application is shown;

[0022] Figure 4b An embodiment of this application is shown. Figure 4a A schematic diagram illustrating the partial partitioning of the element space;

[0023] Figure 5 A schematic block diagram of an image processing apparatus according to one embodiment of the present application is shown;

[0024] Figure 6 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0025] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security and prevention, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial recognition payment, facial recognition unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beauty filters, cosmetics, medical aesthetics, and intelligent temperature measurement.

[0026] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0027] As mentioned above, in the process of generating new 2D images based on 2D images or 3D models, the image data is usually highly sparsity-dependent, resulting in slow image generation speed and large memory consumption. The following example of image rendering based on 3D models further illustrates these technical problems.

[0028] Existing 3D model image rendering techniques typically rely on fitting the model with multiple tetrahedrons and searching for tetrahedrons on the light path corresponding to the pixels during rendering. However, this method requires a large amount of storage and takes a lot of time to search for tetrahedrons.

[0029] Image rendering methods based on Neural Radiance Fields (NeRF) have achieved groundbreaking developments in the past two years. This method models the scene as a latent representation based on a neural network, enabling image rendering directly through the neural network after obtaining the camera's corresponding position in the scene. However, this NeRF-based image rendering method has many network parameters, is computationally complex, does not handle sparse data well, and has a relatively high time cost.

[0030] To at least partially address the aforementioned technical problems, embodiments of this application provide an image processing method, an electronic device, a storage medium, and a computer program product. The image processing method according to embodiments of this application can reduce computational redundancy through sparse convolution, thereby reducing the consumption of storage and computing resources and improving image processing speed. Furthermore, this scheme can share the same target element space, thus further reducing storage overhead. The image processing method according to embodiments of this application can be applied to any field requiring image processing, including but not limited to 3D model image rendering, image compression, and other technical fields.

[0031] First, refer to Figure 1 This describes an example electronic device 100 for implementing the image processing method and apparatus according to embodiments of this application.

[0032] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may also include an input device 106, an output device 108, and an image capturing device 110, these components being interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0033] The processor 102 may be implemented in at least one of the following hardware forms: digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic array (PLA), and microprocessor. The processor 102 may be one or a combination of several of the following: central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or other processing units with data processing capabilities and / or instruction execution capabilities. It may also control other components in the electronic device 100 to perform the desired functions.

[0034] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of this application described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0035] The input device 106 may be a device used by a user to input commands, and may include one or more of the following: keyboard, mouse, microphone, and touch screen.

[0036] The output device 108 can output various information (e.g., images and / or sound) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (e.g., a touch screen).

[0037] The image capturing device 110 can capture images and store the captured images in the storage device 104 for use by other components. The image capturing device 110 can be a standalone camera or a camera in a mobile terminal, etc. It should be understood that the image capturing device 110 is only an example, and the electronic device 100 may not include the image capturing device 110. In this case, other devices with image capturing capabilities can be used to capture images and send the captured images to the electronic device 100.

[0038] For example, the example electronic device for implementing the image processing method and apparatus according to the embodiments of this application can be implemented on devices such as personal computers, terminal devices, time and attendance machines, panel machines, cameras, or remote servers.

[0039] Below, we will refer to Figure 2 An image processing method according to an embodiment of this application is described. Figure 2 A schematic flowchart of an image processing method 200 according to an embodiment of this application is shown. Figure 2 As shown, the image processing method 200 includes steps S210, S220, S230, S240, S250 and step S260.

[0040] In step S210, for any pixel of the image to be generated (hereinafter referred to as x), at least one target sampling point corresponding to the pixel is determined. The at least one target sampling point is a sampling point on the incident light path of the three-dimensional rendering corresponding to the pixel, or the at least one target sampling point is the pixel.

[0041] In the field of 3D model image rendering, at least one target sampling point can be a sampling point on the incident light path (referred to as the "3D rendering incident light path" in this paper) corresponding to pixel x, and the number of target sampling points can be multiple. In the field of image compression, at least one target sampling point can be pixel x itself, and the number of target sampling points can be one.

[0042] In step S220, for any pixel of the image to be generated, for each of at least one target sampling point, multiple target regions in the target element space are determined based on the position of the target sampling point. The multiple target regions correspond one-to-one with receptive fields of multiple scales. Any target region is the region in the convolution region of the corresponding scale receptive field that is closest to the target sampling point in the target element space. The target element space includes multiple element points, and each element point corresponds to a feature value.

[0043] The target element space can be a three-dimensional voxel space or a two-dimensional pixel space. The target element space can contain multiple elements (i.e., element points), each of which is a voxel or pixel. Each element point corresponds to its own feature value. This feature value can be obtained through pre-training and can be a feature vector of any suitable size.

[0044] Receptive fields of various scales can be of any size. In embodiments involving convolution of voxel points in three-dimensional voxel space, the receptive field can also be three-dimensional, such as 2×2×2, 3×3×3, or 4×4×4. In embodiments involving convolution of pixels in two-dimensional pixel space, the receptive field can be two-dimensional, such as 2×2, 3×3, or 4×4.

[0045] For any receptive field of any scale, it has multiple corresponding convolutional regions in the target element space, and the areas covered by any different convolutional regions can be non-overlapping. For example, in a 4×4 pixel space, if a 2×2 receptive field is used for convolution, there can be 4 non-overlapping convolutional regions.

[0046] Convolutional regions corresponding to receptive fields of different sizes can overlap. For example, in the 4×4 pixel space mentioned above, if convolution is performed using a 4×4 receptive field, there is only one corresponding convolutional region, which overlaps with the four convolutional regions corresponding to a 2×2 receptive field.

[0047] The target region is the area within the convolutional region of the receptive field at the corresponding scale that is closest to the target sampling point; that is, the region that includes the target sampling point. For example, in the four convolutional regions corresponding to the 2×2 receptive field mentioned above, if the target sampling point is located in the upper right region of the four convolutional regions, then the upper right region can be taken as the target region. For any given target sampling point, each scale of receptive field corresponds to its own target region.

[0048] Figure 3 A schematic diagram of an element space according to an embodiment of this application is shown. Figure 3 The diagram shows a 2D pixel space of 300. Assume that the 2D pixel space of 300 is the target element space. For example... Figure 3 As shown, the two-dimensional pixel space 300 can include convolutional regions 310 and 320 of different sizes. The size of the receptive field (first receptive field) corresponding to the convolutional region 310 can be 3×3, and the size of the receptive field (second receptive field) corresponding to the convolutional region 320 can be 4×4. See also Figure 3 The two-dimensional pixel space 300 may include nine convolutional regions corresponding to the first receptive field, and the two-dimensional pixel space 300 may include four convolutional regions corresponding to the second receptive field. Figure 3 The target sampling point 330 is also shown. For the first receptive field, the corresponding target area is 310, and for the second receptive field, the corresponding target area is 320. Figure 3 Any corner point of any convolutional region can be considered as an element point.

[0049] The location of the convolution kernel (i.e., the target region) can be determined based on the location of the target sampling point. After determining the target region, a sparse convolution is performed once at that location. For the same receptive field, the sparse convolution of the corresponding scale is performed only once in the corresponding target region.

[0050] In step S230, for any pixel of the image to be generated, for each of at least one target sampling point, and for each of multiple target regions, sparse convolution of the feature values ​​of the element points located in the target region is performed at the corresponding scale to obtain the feature values ​​of the target sampling point at the corresponding scale. The target element space includes multiple grids, the ranges covered by different grids do not overlap with each other, the element points are three-dimensional voxel points or two-dimensional pixel points, and any convolution region corresponding to the receptive field at any scale includes one or more grids in the target element space.

[0051] The following is a brief explanation of sparse convolution using a 3D model as an example. Due to the sparsity of point cloud tasks, after voxelization, there are usually a large number of voxel points with eigenvalues ​​of 0 in space. Convolving all these voxel points would result in significant waste of memory (e.g., GPU memory) and computational resources. Sparse convolution, however, allows convolution to avoid reading and processing voxel points with eigenvalues ​​of 0, thus greatly improving space storage and computational efficiency. Whether convolution is performed at each location and the sparse objects participating in sparse convolution can be stored using a sparse convolution lookup table. When performing sparse convolution, this lookup table can be used to determine the element points in the target region that participate in sparse convolution.

[0052] Since any convolution region corresponding to a receptive field of any scale includes one or more grids in the target element space, at least some element points in one or more grids corresponding to the target convolution region participate in sparse convolution.

[0053] In step S240, for any pixel of the image to be generated, for each of at least one target sampling point, the feature values ​​of the target sampling point at multiple scales are concatenated together to obtain the target input features.

[0054] Concatenating multiple feature values ​​can be achieved by directly joining the channels of these feature values ​​together. In one example, the concatenated result of multiple feature values ​​corresponding one-to-one with multiple scales can be directly determined as the target input feature. In another example, additional information can be further concatenated with multiple feature values ​​corresponding one-to-one with multiple scales to obtain the target input feature.

[0055] In step S250, for any pixel of the image to be generated, for each of at least one target sampling point, the target input features are input into the target network model to obtain the color information corresponding to the target sampling point output by the target network model.

[0056] The color information described herein may include color values ​​in any color space. By way of example, and not limitation, the color information may include RGB color values ​​and opacity information.

[0057] The target network model can be implemented using any suitable network model, which may include any number and size of network layers. For example, and not as a limitation, the target network model may include network models such as multilayer perceptrons (MLP).

[0058] The parameters in the target network model (including the weights and biases of the convolution kernels) and the feature values ​​of each element point in the target element space can be obtained through pre-training.

[0059] In step S260, for any pixel of the image to be generated, the color information of the pixel is determined based on the color information corresponding to at least one target sampling point, so as to obtain the image to be generated.

[0060] When there is only one target sampling point, its corresponding color information can be determined as the color information of pixel x. When there are multiple target sampling points, the color information corresponding to each of the at least one target sampling point can be combined to determine the color information of pixel x. The combination method can be, for example, a weighted average.

[0061] For each pixel of the image to be generated, its color information can be determined through steps S210-S260 above. Obtaining the color information of each pixel in the image to be generated allows us to obtain the image itself.

[0062] The image processing method according to embodiments of this application can perform sparse convolutions with receptive fields of multiple scales within the same target element space. This sparse convolution scheme can reduce computational redundancy, thereby reducing the consumption of storage and computing resources and improving the speed of image processing. Furthermore, this scheme can share the same target element space, thus further reducing storage overhead.

[0063] For example, the image processing method according to the embodiments of this application can be implemented in a device, apparatus or system having a memory and a processor.

[0064] The image processing method according to the embodiments of this application can be deployed at the image acquisition end, for example, at a personal terminal or server with image acquisition function.

[0065] Alternatively, the image processing method according to embodiments of this application can also be deployed distributedly on a server (or cloud) and at a personal terminal. For example, on a client side, the client transmits the captured video frames to the server (or cloud) for image processing.

[0066] According to an embodiment of this application, performing sparse convolution on the feature values ​​of element points located within the target region at a corresponding scale to obtain the feature values ​​of the target sampling point at the corresponding scale includes: obtaining a sparse convolution lookup table related to the target region; determining at least one element point within the target region participating in sparse convolution based on the sparse convolution lookup table; for each of the at least one element point, calculating a weight corresponding to the element point based on the distance between the element point and the target sampling point; and performing a weighted summation of the feature values ​​of the at least one element point based on the weights corresponding to each of the at least one element point to obtain the feature values ​​of the target sampling point at the corresponding scale.

[0067] Sparse convolutional lookup tables can be learned through training operations. The training method for sparse convolutional lookup tables will be described below.

[0068] As mentioned above, for each receptive field at each scale, there are one or more convolutional regions in the target element space. Each convolutional region can have its own sparse convolution lookup table. The sparse convolution lookup table can be used to record the element points participating in sparse convolution within the corresponding convolutional region.

[0069] In the process of sparse convolution, the feature values ​​of the target sampling point can be calculated by analogy with the principle of linear interpolation. When the image processing method 200 is applied to the field of 3D model image rendering, sparse convolution can be implemented by analogy with the principle of trilinear interpolation. When the image processing method 200 is applied to the field of image compression, sparse convolution can be implemented by analogy with the principle of bilinear interpolation. For example, the distance from the target sampling point to any element point participating in the sparse convolution can be calculated, and the weight corresponding to the element point can be calculated based on the inverse ratio of the distance. This weight can be regarded as the weight of the convolution kernel of the sparse convolution. Then, the feature values ​​of these element points can be weighted and summed based on the weights of each element point participating in the convolution to obtain the feature value of the target sampling point.

[0070] According to an embodiment of this application, the element point is a three-dimensional voxel point, the target element space is a three-dimensional voxel space, and the number of at least one target sampling point is multiple. Before determining at least one target sampling point corresponding to any pixel of the image to be generated, the method further includes: acquiring target pose information; determining at least one target sampling point corresponding to the pixel includes: determining a three-dimensional rendering incident light path based on the target pose information; determining the sampling point on the three-dimensional rendering incident light path as at least one target sampling point; and concatenating the feature values ​​of the target sampling point at multiple scales to obtain target input features includes: concatenating the feature values ​​of the target sampling point at multiple scales with the target pose information to obtain target input features.

[0071] Pose information refers to the pose information of the camera. Pose can be understood as the viewing angle. In the field of 3D model image rendering, the target element space and the target network model can be regarded as a functional expression of a certain 3D model. Given a certain viewing angle, a two-dimensional image of the 3D model under that viewing angle can be generated through the target element space and the target network model. Those skilled in the art can understand the principle of 3D model image rendering and the significance of the viewing angle, and this article will not elaborate further.

[0072] The image to be generated is a 2D image corresponding to the 3D model under the specified target pose information. For each pixel of the image to be generated, the 3D rendering incident light path corresponding to that pixel in 3D space under the target pose information can be obtained based on the camera model. Optionally, n sampling points can be selected on the 3D rendering incident light path using a method such as equal spacing to obtain the coordinates of these sampling points.

[0073] For each sampling point on the incident light path in 3D rendering, the convolution region containing that sampling point within any receptive field of the target voxel space can be obtained based on its coordinate information; this is the target region. The feature values ​​of each voxel point in this target region are then obtained. Subsequently, the feature values ​​of the sampling points can be calculated using sparse convolution. As mentioned above, during sparse convolution, the feature values ​​of the sampling points can be calculated by analogy to the principle of trilinear interpolation. For example, the distance from the sampling point to any voxel point participating in the sparse convolution can be calculated, and the weight corresponding to that voxel point can be calculated based on the inverse ratio of this distance. Then, the feature values ​​of these voxel points can be weighted and summed based on the weights of each participating voxel point to obtain the feature value of the target sampling point.

[0074] Subsequently, multiple feature values ​​of the target sampling points extracted based on receptive fields of different scales can be concatenated, and camera pose information can also be added. The concatenated feature values ​​are then fed into the subsequent target network model. The target network model can output the color information of the sampling points, such as their RGB color values ​​and opacity information.

[0075] For example, the color information determination process described above can be performed sequentially on sampling points along a 3D rendering incident light path, in order of distance from the camera from near to far, until the sum of the opacities of all samples reaches a certain opacity threshold, such as 0.99. Subsequently, all subsequent sampling points can be ignored (considered as occluded), and the color values ​​of the acquired sampling points can be weighted and summed according to their opacity. The result is the color value of the current pixel x.

[0076] In this way, after obtaining the color information of each pixel in the image to be generated using the above method, the color information of the entire image to be generated can be obtained, i.e., the image to be generated can be obtained. The color information of the image to be generated can be output as the image rendering result.

[0077] According to an embodiment of this application, the element point is a two-dimensional pixel point, the target element space is a two-dimensional pixel space, and the number of at least one target sampling point is one. Determining at least one target sampling point corresponding to the pixel includes: determining the pixel as a target sampling point.

[0078] As mentioned above, image processing methods can also be applied to image compression. In this case, the target element space and the target network model can be considered as compression models. Exemplarily, and not limitingly, the compression model can be represented by an executable file in ".model" format. This executable file stores the feature values ​​in the target element space and the parameters of the target network model. When a user clicks the executable file, it runs automatically, and the result is the decompression result of a given image, i.e., the original image information. For each original image, a compression model can be trained; different original images correspond to different compression models. Running any compression model yields the corresponding original image.

[0079] Therefore, image compression is similar to rendering a 3D model image from a fixed viewing perspective. Unlike 3D model image rendering, image compression does not require camera pose information, and its element space is a two-dimensional pixel space rather than a three-dimensional voxel space. Furthermore, image compression does not have a three-dimensional rendering incident light path; the sampling point corresponding to any pixel is the pixel itself. Apart from this, the overall image processing flow of image compression is very similar to that of 3D model image rendering. Those skilled in the art can refer to the above description of 3D model image rendering to understand the implementation of image compression; it will not be elaborated upon here.

[0080] The image compression scheme implemented by image processing method 200 can reduce the image loss rate while achieving a high compression ratio.

[0081] According to embodiments of this application, the target element space and the target network model are obtained through training in the following manner: acquiring sample images; iteratively performing optimization operations based on the sample images until the total loss term meets the requirements, the optimization operations including: for any pixel of the sample image, determining at least one sample sampling point corresponding to the pixel, the sample sampling point being of the same type as the target sampling point; for any pixel of the sample image, for each of the at least one sample sampling point, determining multiple sample regions in the current element space based on the position of the sample sampling point, the multiple sample regions corresponding one-to-one with receptive fields of multiple scales, any sample region being the region closest to the sample sampling point in the convolutional region of the corresponding scale receptive field in the current element space; for any pixel of the sample image, for each of the at least one sample sampling point, for each of the multiple sample regions, performing sparse convolution of the feature values ​​of the element points located in the sample region at the corresponding scale to obtain the feature values ​​of the sample sampling point at the corresponding scale; for any pixel of the sample image, For each of at least one sample sampling point, the feature values ​​of that sample sampling point at multiple scales are concatenated to obtain sample input features; for any pixel of the sample image, for each of at least one sample sampling point, the sample input features are input into the current network model to obtain the color information corresponding to that sample sampling point output by the current network model; for any pixel of the sample image, the color information of that pixel is determined based on the color information corresponding to each of at least one sample sampling point to obtain a predicted image; based on the color information of the sample image and the color information of the predicted image, and / or based on the feature values ​​of all element points in the current element space, a total loss term is calculated; based on the total loss term, the feature values ​​of all element points in the current element space and the parameters of the current network model are optimized; wherein, the target element space and the target network model are the current element space and the current network model at the end of the training operation, and before the training operation begins, the initial element space is determined as the current element space and the initial network model is determined as the current network model.

[0082] The sample sampling point and the target sampling point are of the same type. This means that when the target sampling point is a sampling point on the 3D rendering incident light path (which can be called the target 3D rendering incident light path), the sample sampling point is also a sampling point on the 3D rendering incident light path (which can be called the sample 3D rendering incident light path). When the target sampling point is the pixel itself, the sample sampling point is also the pixel itself.

[0083] The feature values ​​of each element in the target element space and the parameters in the target network model can be obtained through training. Before training begins, an initial element space and an initial network model can be set. The feature values ​​of each element in the initial element space can be randomly generated or predefined. The parameters in the initial network model can also be randomly generated or predefined.

[0084] Before training begins, the initial element space and the initial network model can be defined as the current element space and the current network model, respectively. Subsequently, iterative optimization can be performed on the feature values ​​of each element in the current element space and the parameters of the current network model. Each optimization update can be based on the loss term and implemented using the backpropagation algorithm. Those skilled in the art will understand this iterative optimization method, and it will not be elaborated upon here.

[0085] After one or more iterations of optimization, if the performance of the current network model and the current element space meets the preset requirements, such as the convergence of the total loss term, then optimization can be stopped, and the training operation ends. At the end of the training operation, the current element space obtained at this point is considered the target element space, and the current network model obtained at this point is considered the target network model.

[0086] Those skilled in the art will understand that the processing flow for sample images in the training operation is similar to the processing flow for the image to be generated in steps S210-S260 (which can be called the inference operation) described above. The processing flow for sample images in the training operation can be understood with reference to the above description.

[0087] Unlike inference, in training, the total loss term can be calculated based on the color information of the predicted image output by the current network model, and / or based on the feature values ​​of all element points in the current element space. Furthermore, based on the total loss term, the feature values ​​of all element points in the current element space and the parameters of the current network model can be optimized using the backpropagation algorithm.

[0088] According to an embodiment of this application, calculating the total loss term based on the color information of the sample image and the color information of the predicted image, and based on the feature values ​​of all element points in the current element space, includes: calculating a prediction loss term based on the color information of the sample image and the color information of the predicted image; calculating a penalty loss term based on the feature values ​​of all element points in the current element space; and weighted summing the prediction loss term and the penalty loss term to obtain the total loss term.

[0089] During training, the initial loss calculation may include a prediction loss term. This prediction loss term constrains the performance of the current network model's output, ensuring that the predicted image is as close as possible to the sample image. The penalty loss term can be any suitable type, such as the L2 norm (mean squared error), first-order, or cubic norm. For example, the penalty loss term can be calculated based on the L2 norm of the feature values ​​of all points in the current element space, and then added to the loss calculation according to a certain proportional coefficient. The penalty loss term constrains the feature values ​​of the points in the current element space, ensuring that the feature values ​​of some less important points are set as close to 0 as possible.

[0090] When both loss terms are present, training has two objectives: one is to maximize the performance of the entire model (including the current element space and the current network model); the other is to minimize the eigenvalues ​​of its elements to zero. The weights of the prediction loss and penalty loss terms can be adjusted as needed. For example, if the weights of the penalty loss term are adjusted to values ​​infinitely close to zero, then that term is essentially nonexistent. It's understandable that while the goal of the penalty loss term is to make as many eigenvalues ​​as possible zero for sparsity, achieving all zeros is difficult.

[0091] The penalty loss term and the prediction loss term can be independent of each other, and both are optional. For example, the penalty loss term can be used alone as the total loss term for optimization, with the goal of making the feature values ​​of all elements in the current element space as close to 0 as possible.

[0092] By incorporating a penalty loss term calculated based on the feature values ​​of all elements in the current element space into the total loss term, the feature values ​​of some less important elements are set to 0 as much as possible after training. This way, in sparse convolution, these less important elements can be ignored and not included in the calculation. This approach further reduces storage and computational overhead, improving the speed of image processing.

[0093] According to an embodiment of this application, for any receptive field of multiple scales, the receptive field has a corresponding sparse convolution lookup table in each convolution region. The sparse convolution lookup table is used to record the element points participating in sparse convolution. After optimizing the feature values ​​of all element points in the current element space and the parameters of the current network model based on the total loss term, the optimization operation further includes: setting the feature values ​​of element points whose feature values ​​are lower than a first feature value threshold to 0; for any receptive field of multiple scales, updating any sparse convolution lookup table corresponding to the receptive field to delete the element points whose feature values ​​are set to 0 from the sparse convolution lookup table.

[0094] The magnitude of an eigenvalue can be measured using any standard. Exemplarily, but not restrictively, the 2-norm of an eigenvalue can be used to represent its magnitude. Since eigenvalues ​​can be positive or negative, while the 2-norm is always positive, the 2-norm more accurately represents the value of an eigenvalue. Generally, regardless of sign, a larger eigenvalue indicates a higher value, and this should be preserved as much as possible. For example, during training, eigenvalues ​​with smaller 2-norms can be set to 0 and removed from the lookup table of the sparse convolution, thus achieving sparsity.

[0095] We can assign initial values ​​to the feature values ​​of all elements in the initial element space, and then learn to make the feature values ​​of all elements as close to 0 as possible. Once a feature value approaches 0, it can be removed from the sparse convolution lookup table.

[0096] When learning new features through training operations, it is often difficult to make them learn to be exactly 0. Therefore, a threshold can be set, such as e-7, so that if the feature value is smaller than the threshold, it can be set directly to 0.

[0097] By setting feature values ​​below a threshold to 0, the efficiency of sparsification can be effectively improved, thereby increasing the speed of model training.

[0098] According to an embodiment of this application, all grids in the initial element space have the same scale. The optimization operation further includes: when the sparsity ratio of the current element space relative to the element space after the previous split is less than or equal to a ratio threshold, for each element point in at least some of the element points in the current element space, the grid space corresponding to the element point is split into multiple parts in an octree or quadtree manner, and the feature value of the element point is copied to the element points corresponding to each of the multiple split grid spaces to obtain a new split element space.

[0099] The sparsity ratio of the current element space relative to the element space after the previous split refers to the ratio of the number of non-zero feature values ​​in the current element space to the number of non-zero feature values ​​in the element space after the previous split.

[0100] In an embodiment where image processing method 200 is applied to 3D model image rendering, mesh splitting can be performed using an octree. In an embodiment where image processing method 200 is applied to image compression, mesh splitting can be performed using a quadtree.

[0101] Before training, a single-scale, rather than multi-scale, initial element space is constructed. This single-scale initial element space is shared by receptive fields of multiple scales. Exemplarily, but not restrictively, the initial element space can be constructed to the highest possible resolution within memory (e.g., GPU memory) constraints, meaning the initial element space contains as many elements as possible without exceeding memory limits. Receptive fields of multiple scales are then sparsely convolved on the initial element space. After training with a penalized loss term to achieve sufficient sparsity of features, significant amounts of memory (e.g., GPU memory) become available, allowing the element space to be further split. Note that the element space after each split can be considered the current element space for the next iteration of optimization.

[0102] In embodiments employing octree partitioning, the proportion threshold can be less than or equal to 1 / 8. In embodiments employing quadtree partitioning, the proportion threshold can be less than or equal to 1 / 4.

[0103] In one embodiment, the sparsity ratio can be monitored in real time. For example, if after several optimization operations it is found that 7 / 8 of the eigenvalues ​​are set to 0 compared to the initial element space, the elements can be further expanded by splitting the octree.

[0104] In another embodiment, expansion can continue at any time as needed. In this scheme, the sparsity ratio of the current element space relative to the element space after the previous split may not necessarily meet the requirement of being less than or equal to the ratio threshold. If the sparsity ratio does not meet this requirement, a portion of the element points in the current element space can be selected for splitting.

[0105] The following description uses the mesh splitting of a 3D voxel space as an example. Voxels can be sparsified through eigenvalue optimization and setting eigenvalues ​​to 0. When voxels are sparsified to a certain extent, for example, when the remaining number of voxel points is 1 / 8 of the initial number, the current voxel space can be expanded to supplement detailed information. Specifically, following the octree concept, the cube space corresponding to any remaining voxel point can be further split into 8 parts, with the center point of each part serving as a new voxel point. Subsequently, for the new voxel space obtained from the splitting, based on the predefined receptive fields, a sparse convolution lookup table for each convolutional region corresponding to each receptive field can be reconstructed. This optimization operation is then repeated until the penalty loss term can no longer further remove voxel points.

[0106] The feature values ​​of the original element point originally occupied one memory space. When splitting the original element point, its feature values ​​can be copied eight times and stored in eight memory spaces, serving as the feature values ​​for the eight new element points after the split. The feature values ​​of the original element point can then be deleted.

[0107] It should be noted that the above splitting can be performed multiple times. For example, for a certain grid space, after the first octree split, it becomes 8 grid spaces, and then further octree splitting can be performed on any one or more of the 8 grid spaces.

[0108] Sparse convolution requires additional storage of the mapping relationships between elements, thus consuming extra memory. This can easily lead to a low element resolution in the model before training begins, making it difficult to represent scene details. The above decomposition method allows for the use of more elements to represent certain regions rich in detail, thereby enriching scene details. Subsequent image processing based on this detailed element space can generate images with more vivid and accurate local representations.

[0109] According to an embodiment of this application, for any receptive field of a variety of scales, the receptive field has a corresponding sparse convolution lookup table in each convolution region. The sparse convolution lookup table is used to record the element points participating in sparse convolution. The optimization operation further includes: for any receptive field of a variety of scales, updating any sparse convolution lookup table corresponding to the receptive field, so as to add new element points that fall into the convolution region corresponding to the sparse convolution lookup table after splitting.

[0110] Figure 4a A schematic diagram of an element space according to an embodiment of this application is shown. Figure 4b An embodiment of this application is shown. Figure 4a A schematic diagram showing the partial splitting of the element space. Figure 4a and Figure 4b This shows a two-dimensional pixel space.

[0111] See Figure 4a The diagram shows a 2×2 convolution region 410. The receptive field size corresponding to convolution region 410 is also 2×2. The four corner points of convolution region 410 are four element points a1, b1, c1, and d1. See also... Figure 4b The top-right element b1 and the bottom-left element c1 are each split into four parts. After splitting the top-right element b1 into four parts, only one new element b2 falls into the convolution region 410, while the other three elements are outside the convolution region 410. The same process is performed on the bottom-left element c1; only one element c2 falls into the convolution region 410. Thus, in... Figure 4b In the new element space shown, only four element points still fall into convolution region 410. In the sparse convolution lookup table corresponding to convolution region 410, Figure 4aThe top-left element point a1 and the bottom-right element point d1 remain unchanged. The top-right element point b1 and the bottom-left element point c1 are deleted, and the newly split element points b2 and c2 are recorded.

[0112] The storage method of sparse convolutional lookup tables is naturally more friendly to update operations such as adding and deleting elements, so it is very convenient to update the split elements.

[0113] According to an embodiment of this application, before splitting the grid space corresponding to each element point in the current element space into multiple parts in an octree or quadtree manner for each element point in the current element space, the optimization operation further includes: determining element points in the current element space whose feature values ​​are greater than a second feature value threshold as at least some element points; or, determining all element points in the current element space as at least some element points.

[0114] If each grid space is divided into 8 parts, the required storage space will increase by 8 times, thus requiring a sparsity ratio of less than 1 / 8. If the sparsity is only 1 / 4, then some element points with higher eigenvalues ​​can be selected for splitting. As mentioned above, the magnitude of eigenvalues ​​can be represented by their L2 norm. Therefore, some element points with higher L2 norms can be selected for splitting. Of course, if the current sparsity ratio is less than or equal to the threshold, all element points in the current element space can be split directly.

[0115] According to embodiments of this application, the target element space contains at least two types of grids with different scales. It is understood that during training, if the element space is expanded using the octree or quadtree decomposition method described above, the final trained target element space may contain at least two types of grids with different scales. As mentioned above, these grids with different scales allow for the use of more element points to represent regions rich in detail, while fewer element points can be used to represent regions with less detail. This helps improve the local expressiveness of the generated image, while also reducing redundant information and increasing image processing speed.

[0116] The image processing method described above according to embodiments of this application can associate sparse elements based on a preset receptive field and extract features from the elements through sparse convolution with hardware acceleration capabilities. This helps to solve the problem of excessive invalid computation caused by sparsity in image processing and helps to improve the performance of image processing. Research has found that this type of image processing technology has shown great potential in 3D detection and segmentation tasks.

[0117] According to another aspect of this application, an image processing apparatus is provided. Figure 5A schematic block diagram of an image processing apparatus 500 according to one embodiment of this application is shown.

[0118] like Figure 5 As shown, the image processing apparatus 500 according to an embodiment of this application includes a first determining module 510, a second determining module 520, a convolution module 530, a stitching module 540, an input module 550, and a third determining module 560. Each module can respectively perform the functions described above. Figure 2 The image processing method described herein comprises each step. The following description focuses on the main functions of each component of the image processing apparatus 500, omitting the details already described above.

[0119] The first determining module 510 is used to determine, for any pixel in the image to be generated, at least one target sampling point corresponding to that pixel, wherein the at least one target sampling point is a sampling point on the incident light path of the three-dimensional rendering corresponding to that pixel, or the at least one target sampling point is that pixel. The first determining module 510 may be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0120] The second determining module 520 is used to, for any pixel of the image to be generated, for each of at least one target sampling point, determine multiple target regions in the target element space based on the position of the target sampling point. These multiple target regions correspond one-to-one with receptive fields of multiple scales. Each target region is the region closest to the target sampling point in the convolutional region of the corresponding scale's receptive field in the target element space. The target element space includes multiple element points, each element point corresponding to a feature value. The second determining module 520 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0121] The convolution module 530 is used to perform sparse convolution at a corresponding scale on the feature values ​​of element points located within a target region for each pixel of the image to be generated, for each of at least one target sampling point, and for each of multiple target regions, to obtain the feature values ​​of the target sampling point at the corresponding scale. The target element space includes multiple grids, the coverage areas of different grids do not overlap, and the element points are three-dimensional voxels or two-dimensional pixels. Any convolution region corresponding to a receptive field at any scale includes one or more grids in the target element space. The convolution module 530 can be... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0122] The stitching module 540 is used to stitch together the feature values ​​of at least one target sampling point at multiple scales for any pixel in the image to be generated, to obtain the target input features. Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0123] The input module 550 is used to input the target input features into the target network model for each pixel of the image to be generated, for each of at least one target sampling point, to obtain the color information corresponding to that target sampling point output by the target network model. The input module 550 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0124] The third determining module 560 is used to determine the color information of any pixel in the image to be generated based on the color information corresponding to at least one target sampling point, so as to obtain the image to be generated. The third determining module 560 can be composed of... Figure 1 The processor 102 in the illustrated electronic device executes program instructions stored in the storage device 104 to achieve this.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of this application is shown. The electronic device 600 includes a memory 610 and a processor 620.

[0127] The memory 610 stores computer program instructions for implementing corresponding steps in the image processing method according to embodiments of the present application.

[0128] The processor 620 is used to run computer program instructions stored in the memory 610 to perform corresponding steps of the image processing method according to the embodiments of this application.

[0129] In one embodiment, the computer program instructions, when executed by the processor 620, perform the following steps: for any pixel of the image to be generated, determine at least one target sampling point corresponding to that pixel, wherein the at least one target sampling point is a sampling point on the incident light path of the three-dimensional rendering corresponding to that pixel or the at least one target sampling point is that pixel; for each of the at least one target sampling point, determine multiple target regions in the target element space based on the position of the target sampling point, wherein the multiple target regions correspond one-to-one with receptive fields of multiple scales, and any target region is the region closest to the target sampling point in the convolution region of the receptive field of the corresponding scale in the target element space, wherein the target element space includes multiple element points, each element point corresponding to a feature value; for the multiple target regions For each element in the target area, a sparse convolution of the feature values ​​of the element points located within the target area is performed at the corresponding scale to obtain the feature values ​​of the target sampling point at the corresponding scale. The target element space includes multiple grids, and the coverage areas of different grids do not overlap. The element points are three-dimensional voxels or two-dimensional pixels. Any convolution region corresponding to any receptive field at any scale includes one or more grids in the target element space. The feature values ​​of the target sampling point at multiple scales are concatenated together to obtain the target input features. The target input features are input into the target network model to obtain the color information corresponding to the target sampling point output by the target network model. The color information of the pixel is determined based on the color information corresponding to at least one target sampling point to obtain the image to be generated.

[0130] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When the program instructions are run by a computer or processor, they are used to execute corresponding steps of the image processing method of the embodiments of this application and to implement corresponding modules in the image processing apparatus according to the embodiments of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0131] In one embodiment, when the program instructions are executed by a computer or processor, the computer or processor may implement the various functional modules of the image processing apparatus according to the embodiments of the present application, and / or may execute the image processing method according to the embodiments of the present application.

[0132] In one embodiment, the program instructions are used to perform the following steps at runtime: for any pixel of the image to be generated, determine at least one target sampling point corresponding to the pixel, wherein the at least one target sampling point is a sampling point on the incident light path of the 3D rendering corresponding to the pixel or the pixel itself; for each of the at least one target sampling point, determine multiple target regions in the target element space based on the position of the target sampling point, wherein the multiple target regions correspond one-to-one with receptive fields of multiple scales, and any target region is the region closest to the target sampling point in the convolution region of the receptive field of the corresponding scale in the target element space, wherein the target element space includes multiple element points, each element point corresponding to a feature value; for each of the multiple target regions A sparse convolution of the feature values ​​of element points located within the target region at a corresponding scale is performed to obtain the feature values ​​of the target sampling point at the corresponding scale. The target element space comprises multiple grids, with different grids covering non-overlapping areas. Element points are either three-dimensional voxels or two-dimensional pixels. Each convolution region corresponding to the receptive field at any scale includes one or more grids in the target element space. The feature values ​​of the target sampling point at multiple scales are concatenated to obtain the target input features. These target input features are then input into the target network model to obtain the color information corresponding to the target sampling point output by the target network model. Finally, the color information of the pixel is determined based on the color information corresponding to at least one target sampling point to obtain the image to be generated.

[0133] Furthermore, according to an embodiment of this application, a computer program product is also provided, which includes a computer program that, when running, performs the image processing method 200 described above.

[0134] Each module in the electronic device according to the embodiments of this application can be implemented by the processor of the electronic device implementing image processing according to the embodiments of this application running computer program instructions stored in memory, or by computer instructions stored in a computer-readable storage medium of a computer program product according to the embodiments of this application being executed by a computer.

[0135] Furthermore, according to embodiments of this application, a computer program is also provided, which, when running, is used to execute the above-described image processing method 200.

[0136] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0139] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0140] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more aspects of the various applications, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0141] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0142] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0143] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the image processing apparatus according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0144] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0145] The above are merely specific embodiments or descriptions of specific embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. An image processing method, comprising: For any pixel of the image to be generated, Determine at least one target sampling point corresponding to the pixel, wherein the at least one target sampling point is a sampling point on the incident light path of the three-dimensional rendering corresponding to the pixel or the at least one target sampling point is the pixel itself; For each of the at least one target sampling point Based on the location of the target sampling point, multiple target regions in the target element space are determined. The multiple target regions correspond one-to-one with receptive fields of multiple scales. Any target region is the region in the convolutional region of the corresponding scale receptive field in the target element space that is closest to the target sampling point. The target element space includes multiple element points, and each element point corresponds to a feature value. For each of the plurality of target regions, a sparse convolution of the feature value of the element point located in the target region is performed at the corresponding scale to obtain the feature value of the target sampling point at the corresponding scale. The target element space includes a plurality of grids, and the ranges covered by different grids do not overlap with each other. The element point is a three-dimensional voxel point or a two-dimensional pixel point. Any convolution region corresponding to any receptive field at any scale includes one or more grids in the target element space. The feature values ​​of the target sampling point at the multiple scales are concatenated together to obtain the target input features; The target input features are input into the target network model to obtain the color information corresponding to the target sampling point output by the target network model; The color information of the pixel is determined based on the color information corresponding to each of the at least one target sampling point, so as to obtain the image to be generated.

2. The method as described in claim 1, wherein, The target element space contains at least two types of grids of different scales.

3. The method as described in claim 1 or 2, wherein, The step of performing sparse convolution on the feature values ​​of element points located within the target region at a corresponding scale to obtain the feature values ​​of the target sampling point at the corresponding scale includes: Obtain the sparse convolution lookup table associated with the target region; Based on the sparse convolution lookup table, at least one element point in the target region that participates in sparse convolution is determined; For each of the at least one element point, a weight corresponding to the element point is calculated based on the distance between the element point and the target sampling point; The feature values ​​of the at least one element point are weighted and summed based on the weights corresponding to each element point to obtain the feature value of the target sampling point at the corresponding scale.

4. The method as described in any one of claims 1 to 3, wherein, The element point is a three-dimensional voxel point, the target element space is a three-dimensional voxel space, and the number of the at least one target sampling point is multiple. Before determining at least one target sampling point corresponding to any pixel in the image to be generated, the method further includes: Obtain target pose information; Determining at least one target sampling point corresponding to the pixel includes: The 3D rendering incident light path is determined based on the target pose information; The sampling point on the incident light path of the three-dimensional rendering is determined as the at least one target sampling point; The step of concatenating the feature values ​​of the target sampling points at multiple scales to obtain the target input features includes: The feature values ​​of the target sampling point at the various scales are concatenated with the target pose information to obtain the target input features.

5. The method as described in any one of claims 1 to 3, wherein, The target element space and the target network model are obtained through training in the following manner: Acquire sample images; Based on the sample images, iterative optimization operations are performed until the total loss term meets the requirements. The optimization operations include: For any pixel of the sample image, Determine at least one sample point corresponding to the pixel, wherein the sample point is of the same type as the target sample point; For each of the at least one sample point Based on the location of the sample sampling point, multiple sample regions in the current element space are determined. The multiple sample regions correspond one-to-one with the receptive fields of the various scales. Any sample region is the region in the convolution region of the corresponding scale that is closest to the sample sampling point. For each of the plurality of sample regions, a sparse convolution of the feature values ​​of the element points located in that sample region is performed at the corresponding scale to obtain the feature values ​​of the sample sampling points at the corresponding scale. The feature values ​​of the sampled points at the multiple scales are concatenated together to obtain the sample input features; The sample input features are input into the current network model to obtain the color information corresponding to the sample sampling point output by the current network model; The color information of the pixel is determined based on the color information corresponding to each of the at least one sample point, so as to obtain the predicted image; The total loss term is calculated based on the color information of the sample image and the color information of the predicted image, and / or based on the feature values ​​of all element points in the current element space; Based on the total loss term, optimize the feature values ​​of all element points in the current element space and the parameters of the current network model; The target element space and the target network model are the current element space and the current network model at the end of the training operation. Before the training operation begins, the initial element space is determined as the current element space and the initial network model is determined as the current network model.

6. The method of claim 5, wherein, The calculation of the total loss term based on the color information of the sample image and the color information of the predicted image, and based on the feature values ​​of all element points in the current element space, includes: Calculate the prediction loss term based on the color information of the sample image and the color information of the prediction image; Calculate the penalty loss term based on the feature values ​​of all element points in the current element space; The predicted loss term and the penalized loss term are weighted and summed to obtain the total loss term.

7. The method of claim 5, wherein, For any receptive field of any of the multiple receptive fields, the receptive field has a corresponding sparse convolution lookup table for each convolution region. The sparse convolution lookup table is used to record the element points participating in sparse convolution. After optimizing the feature values ​​of all element points in the current element space and the parameters of the current network model based on the total loss term, the optimization operation further includes: Set the feature values ​​of all element points in the current element space whose feature values ​​are lower than the first feature value threshold to 0. For any receptive field of the various scales, update any sparse convolution lookup table corresponding to that receptive field to remove elements whose feature values ​​are set to 0 from the sparse convolution lookup table.

8. The method as described in any one of claims 5 to 7, wherein, All grids in the initial element space have the same scale, and the optimization operation further includes: If the sparsity ratio of the current element space relative to the element space after the previous split is less than or equal to the ratio threshold, for each element point in at least some of the element points in the current element space, the grid space corresponding to the element point is split into multiple parts in the manner of octree or quadtree, and the feature value of the element point is copied to the element points corresponding to each of the multiple split grid spaces to obtain a new split element space.

9. The method of claim 8, wherein, For any receptive field of any of the multiple scales, the receptive field has a corresponding sparse convolution lookup table for each convolution region. The sparse convolution lookup table is used to record the element points participating in sparse convolution. The optimization operation further includes: For any receptive field of any of the multiple receptive fields, update any sparse convolution lookup table corresponding to that receptive field to add new element points that fall into the convolution region corresponding to that sparse convolution lookup table after splitting.

10. An electronic device comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the image processing method as described in any one of claims 1 to 9.

11. A storage medium on which program instructions are stored, wherein, The program instructions are used to execute the image processing method as described in any one of claims 1 to 9 when the program is run.

12. A computer program product, the computer program product comprising a computer program, wherein, The computer program, when running, is used to perform the image processing method as described in any one of claims 1 to 9.

Images