Image processing method and device, storage medium and electronic device

Abstract

The present disclosure provides an image processing method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-processed image carrying time information and a target view angle; constructing a plurality of spatial points of the to-be-processed image at the original view angle of the to-be-processed image, and determining the space-time characteristics of each spatial point according to the time information and the spatial coordinates of each spatial point; determining the probability of each spatial point belonging to each motion domain and the motion characteristics under each motion domain according to the space-time characteristics of each spatial point; wherein the motion domain includes: static domain, rigid body motion domain and non-rigid body motion domain; determining the motion combination characteristics of each spatial point according to the probability and the motion characteristics; rendering according to the target view angle, the spatial coordinates of each spatial point and the motion combination characteristics of each spatial point to obtain a redrawn image of the to-be-processed image at the target view angle. The method can classify and render the motion combination characteristics of the spatial points of the to-be-processed image as the basis, thereby improving the processing efficiency and saving the computing resources.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image processing method and apparatus, storage medium and electronic device. Background Technology

[0002] With the development of computer technology and computer graphics technology, 3D reconstruction and new perspective image rendering have become the focus of image processing technology research. With the emergence of new concepts such as digital twins, holographic communication, and metaverse, the industry's demand for 3D reconstruction and new perspective image rendering is also gradually increasing.

[0003] In related technologies, when reconstructing and rendering videos or images in videos from a new perspective, a new 3D model is usually created for each frame of the video before rendering. This method is inefficient for image processing, consumes a lot of computing resources, and occupies a lot of storage space.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image processing method, apparatus, electronic device, and storage medium that can improve rendering efficiency and save computing resources.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, an image processing method is provided, comprising: acquiring an image to be processed carrying time information and a target viewpoint; constructing multiple spatial points of the image to be processed under the original viewpoint of the image to be processed, and determining the spatiotemporal characteristics of each spatial point based on the time information and the spatial coordinates of each spatial point; determining the probability that each spatial point belongs to each motion domain based on the spatiotemporal characteristics of each spatial point, and determining the motion characteristics of each spatial point under each motion domain based on the spatiotemporal characteristics of each spatial point; wherein, the motion domain includes: a static domain, a rigid body motion domain, and a non-rigid body motion domain; determining the motion combination characteristics of each spatial point based on the probability and the motion characteristics; and rendering based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination characteristics of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0008] In one embodiment of this disclosure, constructing multiple spatial points of the image to be processed under the original viewpoint of the image to be processed includes: processing the image to be processed by a motion recovery structure algorithm to obtain a camera pose corresponding to the image to be processed, and using the camera pose as the original viewpoint of the image to be processed; constructing a three-dimensional virtual ray based on the original viewpoint and the pixels in the image to be processed; and sampling on each three-dimensional virtual ray to obtain multiple spatial points.

[0009] In one embodiment of this disclosure, determining the probability that each spatial point belongs to a certain motion domain based on the spatiotemporal characteristics of each spatial point includes: processing the spatiotemporal characteristics of each spatial point through a motion domain classification model to obtain the probability that each spatial point belongs to a static domain, a rigid body motion domain, and a non-rigid body motion domain.

[0010] In one embodiment of this disclosure, determining the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point includes: processing the spatiotemporal characteristics of each spatial point through a static feature extraction model to obtain the static domain motion characteristics of each spatial point; processing the spatiotemporal characteristics of each spatial point through a displacement prediction model to obtain the post-displacement spatiotemporal characteristics of each spatial point, and processing the post-displacement spatiotemporal characteristics of each spatial point through a static feature extraction model to obtain the rigid body motion domain motion characteristics of each spatial point; and processing the spatiotemporal characteristics of each spatial point through a dynamic feature extraction model to obtain the non-rigid body motion domain motion characteristics of each spatial point.

[0011] In one embodiment of this disclosure, determining the motion combination characteristics of each spatial point based on probability and motion characteristics includes: determining the feature components of each spatial point in the static domain based on the static domain motion characteristics of each spatial point and the probability that each spatial point belongs to the static domain; determining the feature components of each spatial point in the rigid body motion domain based on the rigid body motion domain motion characteristics of each spatial point and the probability that each spatial point belongs to the rigid body motion domain; determining the feature components of each spatial point in the non-rigid body motion domain based on the non-rigid body motion domain motion characteristics of each spatial point and the probability that each spatial point belongs to the non-rigid body motion domain; and determining the motion combination characteristics of each spatial point based on the feature components of each spatial point in the static domain, rigid body motion domain, and non-rigid body motion domain, respectively.

[0012] In one embodiment of this disclosure, rendering is performed based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint. This includes: processing the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features through a neural radiation field network model to obtain the color value and density value at each spatial point; and rendering is performed based on the color value and density value at each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0013] In one embodiment of this disclosure, a neural radiation field network model is used to process the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features to obtain the color value and density value at each spatial point. This includes: determining the two-dimensional coordinates corresponding to each spatial point based on the spatial coordinates of each spatial point and the target viewpoint; processing the motion combination features of each spatial point using a bilinear interpolation algorithm based on the two-dimensional coordinates of each spatial point to obtain the difference features of the two-dimensional coordinates corresponding to each spatial point; encoding the spatial coordinates of each spatial point to obtain spatial coordinate codes; and processing the difference features of the two-dimensional coordinates corresponding to each spatial point and the spatial coordinate codes of each spatial point using the neural radiation field network model to obtain the color value and density value at each spatial point.

[0014] In one embodiment of this disclosure, rendering is performed based on the color and density values ​​at each spatial point to obtain a redrawn image of the image to be processed from the target viewpoint. This includes: for each three-dimensional virtual ray, integrating the color and density values ​​at all spatial points sampled from the three-dimensional virtual ray to obtain the redrawn color value of the pixel corresponding to the three-dimensional virtual ray; and rendering is performed based on the redrawn color values ​​of the pixels corresponding to all three-dimensional virtual rays to obtain a redrawn image.

[0015] According to another aspect of this disclosure, an image processing apparatus is provided, comprising: an acquisition module for acquiring an image to be processed carrying time information and a target viewpoint; a construction module for constructing multiple spatial points of the image to be processed under the original viewpoint of the image to be processed, and determining the spatiotemporal features of each spatial point based on the time information and the spatial coordinates of each spatial point; a feature extraction module for determining the probability that each spatial point belongs to each motion domain based on the spatiotemporal features of each spatial point, and determining the motion features of each spatial point under each motion domain based on the spatiotemporal features of each spatial point; wherein the motion domain includes: a static domain, a rigid body motion domain, and a non-rigid body motion domain; a feature combination module for determining the motion combination features of each spatial point based on the probability and the motion features; and a rendering module for rendering based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0016] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image processing method described above.

[0017] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described image processing method by executing the executable instructions.

[0018] The image processing method provided in this disclosure first preprocesses the image to be processed, which contains temporal information, to obtain the spatiotemporal features of multiple spatial points related to the image. Then, a trained neural network model can predict the probability of each spatial point belonging to different motion domains and its motion features in each motion domain based on the spatiotemporal features. The probabilities and motion features are then combined to obtain the motion combination features of the spatial points. Finally, rendering is performed based on a specified target viewpoint, the spatial coordinates of the spatial points, and the motion combination features to obtain a redrawn image of the image to be processed from the target viewpoint. Therefore, this method can predict and extract features from spatial points in an image to be processed that carries temporal information, obtaining the motion combination features of each spatial point. This allows for rendering of spatial points belonging to different motion domain categories based on the target viewpoint during rendering, improving processing efficiency and saving computational resources.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 A schematic diagram of a network architecture for an image processing method to which embodiments of the present disclosure can be applied is shown;

[0022] Figure 2 A flowchart of an image processing method according to an embodiment of this disclosure is shown;

[0023] Figure 3 A flowchart illustrating rendering in an image processing method according to an embodiment of this disclosure is shown;

[0024] Figure 4 A flowchart illustrating an image processing method for obtaining color and density values ​​at various spatial points according to an embodiment of this disclosure is shown.

[0025] Figure 5 A schematic diagram of the flow framework of an image processing method according to an embodiment of the present disclosure is shown;

[0026] Figure 6 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown; and

[0027] Figure 7A structural block diagram of an image processing computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0029] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0030] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] Figure 1 A schematic diagram of a network architecture for an image processing method to which embodiments of the present disclosure can be applied is shown.

[0034] like Figure 1As shown, the system architecture may include a server 101, a network 102, and a client 103. The network 102 serves as the medium for providing a communication link between the client 103 and the server 101. The network 102 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] In an exemplary embodiment, server 101 may be a server providing various services, such as a backend management server supporting the device operated by the user using client 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to client 103. Server 101 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 may also be a server for a network platform, such as a trading platform, live streaming platform, social platform, or music platform; this embodiment does not limit this. The server may be a single server or a cluster of multiple servers; this disclosure does not limit the specific architecture of the server.

[0036] In an exemplary embodiment, the client 103 that transmits data with the server 101 may include, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices.

[0037] In some practical applications, client 103 can be an electronic device that simultaneously has a display screen and a camera unit. The camera unit can capture images of objects in a three-dimensional scene to generate two-dimensional images or videos. Then, video frames from the two-dimensional images or videos are sent as images to be processed to server 101 via network 102. Server 101 then uses the image processing method provided in this embodiment to process the images to be processed and generate a redrawn image from the target viewpoint. Client 103 can also receive the redrawn image returned by server 101 and display it on the display screen.

[0038] In an exemplary embodiment, the process by which server 101 implements the image processing method may be as follows: server 101 acquires an image to be processed carrying time information and a target viewpoint; server 101 constructs multiple spatial points of the image to be processed under the original viewpoint of the image to be processed, and determines the spatiotemporal characteristics of each spatial point based on the time information and the spatial coordinates of each spatial point; server 101 determines the probability that each spatial point belongs to each motion domain based on the spatiotemporal characteristics of each spatial point, and determines the motion characteristics of each spatial point under each motion domain based on the spatiotemporal characteristics of each spatial point; wherein, the motion domain includes: static domain, rigid body motion domain, and non-rigid body motion domain; server 101 determines the motion combination characteristics of each spatial point based on the probability and motion characteristics; server 101 renders the image based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination characteristics of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0039] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the image processing method provided in this disclosure. Figure 1 The number of servers 101, networks 102, and clients 103 shown is merely illustrative; any number of clients, networks, and servers can be used as needed.

[0040] To enable those skilled in the art to better understand the technical solutions of this disclosure, the following will describe in more detail the various steps of the image processing method in the example embodiments of this disclosure in conjunction with the accompanying drawings and examples.

[0041] Figure 2 A flowchart of an image processing method according to an embodiment of this disclosure is shown. The method provided in this disclosure can be performed by, for example... Figure 1 The server 101 or client 103 shown may execute this, but this disclosure is not limited thereto.

[0042] like Figure 2 As shown, the image processing method provided in this disclosure embodiment may include the following steps.

[0043] Step S201: Obtain the image to be processed and the target viewpoint carrying time information.

[0044] The image to be processed can be one or more images taken by the user using a terminal with camera functionality, or it can be one or more video frames selected from a video clip.

[0045] The target viewpoint can be user-specified or parsed from other image data; the target viewpoint can directly indicate viewpoint information (such as coordinate information or direction information), or it can be a processing method for a known viewpoint, such as 360° angle deflection of the original viewpoint of the image to be processed, depth of field adjustment method, distance of left or right movement, and other viewpoint information.

[0046] Step S203: Construct multiple spatial points of the image to be processed from the original viewpoint of the image to be processed, and determine the spatiotemporal characteristics of each spatial point based on the time information and the spatial coordinates of each spatial point.

[0047] The original viewpoint of the image to be processed can be known or obtained from the analysis of the image to be processed. Multiple spatial points can be constructed based on the original viewpoint and the pixel size of the image to be processed. The information carried by these multiple spatial points can be regarded as the features of the image to be processed. In the subsequent process, multiple spatial points can be processed to generate a redrawn image corresponding to the image to be processed.

[0048] After constructing the spatial points, their spatial coordinates can be concatenated with the temporal information of the image to be processed to obtain spatiotemporal features. For example, assuming the spatial coordinates of a spatial point are (x, y, z) and the temporal information of the image to be processed is t, then the spatiotemporal features of that spatial point can be (x, y, z, t).

[0049] In some embodiments, step S203, "constructing multiple spatial points of the image to be processed under the original viewpoint of the image to be processed", may include: processing the image to be processed by a motion recovery structure algorithm to obtain a camera pose corresponding to the image to be processed, and using the camera pose as the original viewpoint of the image to be processed; constructing a three-dimensional virtual ray based on the original viewpoint and the pixels in the image to be processed; and sampling on each three-dimensional virtual ray to obtain multiple spatial points.

[0050] Step S203 can be viewed as an input preprocessing operation for the image to be processed. After the user captures one or more images to be processed, these images can be input into the SFM (Structure from Motion) algorithm module, which then outputs a camera pose estimate for the image. Next, three-dimensional virtual rays can be generated in three-dimensional space based on the camera pose (i.e., the original viewpoint). For example, if the pixel size of the image to be processed is H×W, where H and W are the pixel sizes of the height and width of the image, respectively, then H×W three-dimensional virtual rays originating from the original viewpoint and passing through H×W pixels can be generated. Finally, uniform sampling can be performed on these three-dimensional virtual rays. For example, the same number of spatial points can be uniformly sampled on each three-dimensional virtual ray, and the spatial coordinates and orientations of these points can be recorded. The orientation information of a spatial point reflects the three-dimensional virtual ray it belongs to.

[0051] Step S205: Determine the probability that each spatial point belongs to each motion domain based on the spatiotemporal characteristics of each spatial point, and determine the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point; wherein, the motion domain includes: static domain, rigid body motion domain and non-rigid body motion domain.

[0052] Among them, objects belonging to the static domain can be understood as objects whose shape and position do not change over time. These can mainly be static background objects in the scene, such as the ground, houses, tables and chairs, or objects that can move in the scene but have a very low probability of moving.

[0053] For objects belonging to the rigid body motion domain, it can be understood that their shape does not change, but their position changes, such as a moving car or a floating balloon.

[0054] For objects that belong to the non-rigid body motion domain, it can be understood that their shape will change, or that new objects appear in the scene. These objects mainly include people or animals in the video, or an object that suddenly "flies" into the video, such as a bird flying into the frame.

[0055] A trained neural network model can be used to process the spatiotemporal features of spatial points to obtain the probability that each spatial point belongs to a different motion domain. This trained neural network model can be obtained by training on the scene video corresponding to the image to be processed. Specifically, the model can learn from the differences between video frames in the scene video to determine the probability that pixels within different objects in the scene belong to different motion domains at different times, and to predict the possible positions of objects. Then, based on the temporal information of the image to be processed, the neural network model can make predictions to obtain the probability that each spatial point belongs to a different motion domain and the possible offset when the cloud moves.

[0056] Alternatively, another trained neural network model for feature extraction can be used to process the spatiotemporal features of spatial points, in order to extract the motion features of each spatial point in each motion domain. This neural network model for feature extraction can be implemented using any mainstream convolutional neural network or fully connected neural network, and this disclosure does not limit it.

[0057] Step S207: Determine the motion combination characteristics of each spatial point based on probability and motion characteristics.

[0058] In some embodiments, determining the motion combination characteristics of each spatial point based on probability and motion characteristics includes: determining the feature components of each spatial point in the static domain based on the static domain motion characteristics of each spatial point and the probability that each spatial point belongs to the static domain; determining the feature components of each spatial point in the rigid body motion domain based on the rigid body motion domain motion characteristics of each spatial point and the probability that each spatial point belongs to the rigid body motion domain; determining the feature components of each spatial point in the non-rigid body motion domain based on the non-rigid body motion domain motion characteristics of each spatial point and the probability that each spatial point belongs to the non-rigid body motion domain; and determining the motion combination characteristics of each spatial point based on the feature components of each spatial point in the static domain, rigid body motion domain, and non-rigid body motion domain, respectively.

[0059] Specifically, for each spatial point, we can first multiply the static domain motion characteristics of the spatial point by the probability that the spatial point belongs to the static domain to obtain the feature components of the spatial point in the static domain; then multiply the rigid body motion domain motion characteristics of the spatial point by the probability that the spatial point belongs to the rigid body motion domain to obtain the feature components of the spatial point in the rigid body motion domain; then multiply the non-rigid body motion domain motion characteristics of the spatial point by the probability that the spatial point belongs to the non-rigid body motion domain to obtain the feature components of the spatial point in the non-rigid body motion domain; finally, by adding the feature components of the spatial point in the static domain, rigid body motion domain and non-rigid body motion domain respectively, we can obtain the motion combination characteristics of the spatial point.

[0060] Step S209: Render the image based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0061] Rendering can be achieved using a pre-trained neural rendering network model. The input data of this model can be the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point. The output data can be the color value and density value of each spatial point. Then, volume rendering can be performed based on the color value and density value of each spatial point and its corresponding 3D virtual ray information to obtain the color value information of the pixels corresponding to each 3D virtual ray. Rendering based on the color value information yields the redrawn image of the image to be processed from the target viewpoint.

[0062] Among them, motion combination features can be used as an indicator vector for rendering. Specifically, motion combination features can indicate the motion domain to which a spatial point is more likely to belong. Different processing can be performed during the rendering process for different motion domains. For example, for spatial points belonging to the static domain, since they remain unchanged throughout the entire scene video, the data that has already been modeled and rendered can be used or only a fixed transformation process can be performed. As another example, for spatial points belonging to the rigid body motion domain, since only the position changes and the shape itself remains unchanged throughout the entire scene video, the possible position under the same time information can be estimated first, and then the data that has already been modeled and rendered can be used or only a fixed transformation process can be performed.

[0063] As can be seen, the approach of first determining the category of the motion domain and then performing classified rendering can use motion combination features as a basis for classifying and rendering each spatial point. This method of modeling objects separately and then directly using the modeled data for rendering not only improves processing efficiency but also saves computational resources. In some practical applications, when the image to be processed consists of multiple video frames from a scene video, the image processing method provided in this disclosure can model objects separately. The modeling data of some objects (including objects belonging to the static domain and rigid body motion domain) can be directly reused, thereby avoiding the problems of low rendering efficiency and excessive computational resource consumption caused by remodeling each frame separately.

[0064] The image processing method provided in this disclosure first preprocesses the image to be processed, which contains temporal information, to obtain the spatiotemporal features of multiple spatial points related to the image. Then, a trained neural network model can predict the probability of each spatial point belonging to different motion domains and its motion features in each motion domain based on the spatiotemporal features. The probabilities and motion features are then combined to obtain the motion combination features of the spatial points. Finally, rendering is performed based on a specified target viewpoint, the spatial coordinates of the spatial points, and the motion combination features to obtain a redrawn image of the image to be processed from the target viewpoint. Therefore, this method can predict and extract features from spatial points in an image to be processed that carries temporal information, obtaining the motion combination features of each spatial point. This allows for rendering of spatial points belonging to different motion domain categories based on the target viewpoint during rendering, improving processing efficiency and saving computational resources.

[0065] In some embodiments, the step S205 of "determining the probability of each spatial point belonging to each motion domain based on the spatiotemporal characteristics of each spatial point" may include: processing the spatiotemporal characteristics of each spatial point through a motion domain classification model to obtain the probability that each spatial point belongs to a static domain, a rigid body motion domain, and a non-rigid body motion domain.

[0066] The motion domain classification model can be implemented by a fully connected neural network. The output layer can be three normalized neurons, representing the probability that the point belongs to the static class, rigid body motion class, and non-rigid body motion class, respectively, thereby predicting the probability of the motion domain category to which each spatial point belongs.

[0067] In some practical applications, the images to be processed can be multiple images captured consecutively or multiple video frames from a video. The motion domain classification model is trained on scene videos of any scenario. The motion domain classification model can first learn from the differences between video frames in the scene video to obtain the ability to identify the motion domain to which each pixel belongs based on multiple video frames. Then, the trained motion domain classification model can be used to process multiple images to be processed, identifying the motion domain to which each spatial point in each image belongs based on the differences between these multiple images, that is, the probability that each spatial point belongs to the static domain, rigid body motion domain, and non-rigid body motion domain.

[0068] The training process for the motion domain classification model can be as follows: First, one or more videos of a known scene can be acquired, where multiple videos can be videos from different perspectives. Then, several video frames with time information are selected as sample data, and the actual motion domains (including static domains, rigid body motion domains, and non-rigid body motion domains) of the objects to which each pixel in these video frames belongs are labeled based on the video itself. These labeled actual motion domains can serve as sample labels for the sample data. The video frames can be preprocessed as described in the aforementioned embodiments to obtain the spatiotemporal features of the spatial points of the sample data. Then, the spatiotemporal features of the sample data are input into the motion domain classification model to be trained, and the predicted motion domains (including static domains, rigid body motion domains, and non-rigid body motion domains) for each spatial point are output. Then, the actual motion domain of the spatial point corresponding to the pixel is determined based on the actual motion domain of the original pixel. For each spatial point, a category loss function is constructed using its actual motion domain and predicted motion domain. Then, the motion domain classification model can be trained based on the category loss function of all spatial points to obtain a trained motion domain classification model, which can be applied to the above embodiments.

[0069] In some practical applications, motion domain classification models can also be trained on the scene video corresponding to the image to be processed. During application, the motion domain classification model can predict the spatial points of the image to be processed within the same scene, obtaining the probability that each spatial point belongs to the static domain, rigid body motion domain, or non-rigid body motion domain.

[0070] Furthermore, if the image to be processed is any frame in the scene video, the time corresponding to that frame in the scene video can be directly used as the time information of the image to be processed. If the image to be processed is a single photo taken in the same scene, the time information of the image to be processed can be assigned a default value (such as 0), or it can be compared with the scene video used for training, and the time information of the closest video frame can be used as the time information of the image to be processed.

[0071] In some embodiments, step S205, "determining the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point," may include: processing the spatiotemporal characteristics of each spatial point through a static feature extraction model to obtain the static domain motion characteristics of each spatial point; processing the spatiotemporal characteristics of each spatial point through a displacement prediction model to obtain the post-displacement spatiotemporal characteristics of each spatial point, and processing the post-displacement spatiotemporal characteristics of each spatial point through a static feature extraction model to obtain the rigid body motion domain motion characteristics of each spatial point; and processing the spatiotemporal characteristics of each spatial point through a dynamic feature extraction model to obtain the non-rigid body motion domain motion characteristics of each spatial point.

[0072] The static feature extraction model can be implemented using any mainstream convolutional neural network or fully connected neural network, and this disclosure does not impose any limitations on it. The displacement prediction model can also be trained based on videos of known scenes as training samples, so that the displacement prediction model can estimate the position offset of the object to which different pixels belong over time, and thus estimate the offset of spatial points.

[0073] Figure 3 A flowchart illustrating rendering in an image processing method according to an embodiment of this disclosure is shown, such as... Figure 3 As shown, in some embodiments, step S209 may further include the following steps.

[0074] Step S301: Process the target viewpoint, spatial coordinates of each spatial point, and motion combination features through a neural radiation field network model to obtain the color value and density value at each spatial point.

[0075] The neural radiation field network model is trained using videos of known scenes as training samples, such as the scene video corresponding to the image to be processed, or scene videos of any scene.

[0076] Step S303: Render the image based on the color and density values ​​at each spatial point to obtain a redrawn image of the image to be processed from the target viewpoint.

[0077] Figure 4 A flowchart illustrating an image processing method for obtaining color and density values ​​at various spatial points, as shown in an embodiment of this disclosure, is presented. Figure 4As shown, in some embodiments, step S301 may further include the following steps.

[0078] Step S401: Determine the two-dimensional coordinates of each spatial point based on the spatial coordinates of each spatial point and the target viewpoint;

[0079] Step S403: Based on the two-dimensional coordinates corresponding to each spatial point, use the bilinear interpolation algorithm to process the motion combination features of each spatial point and obtain the difference features of the two-dimensional coordinates corresponding to each spatial point.

[0080] Step S405: Encode the spatial coordinates of each spatial point to obtain the spatial coordinate code;

[0081] Step S407: Process the difference features of the two-dimensional coordinates corresponding to each spatial point and the spatial coordinate encoding of each spatial point through the neural radiation field network model to obtain the color value and density value at each spatial point.

[0082] In some embodiments, step S303 may further include: for each three-dimensional virtual ray, integrating the color values ​​and density values ​​at all spatial points sampled from the three-dimensional virtual ray to obtain the redraw color value of the pixel corresponding to the three-dimensional virtual ray; and rendering based on the redraw color values ​​of the pixels corresponding to all three-dimensional virtual rays to obtain a redrawn image.

[0083] In this context, each 3D virtual ray in the 3D space corresponds to a point in the 2D image (i.e., the redrawn image) from the target's perspective. The rendering process can be performed point by point, meaning that integration can be performed along the spatial points sampled along the ray. The specific formula is as follows:

[0084] in,

[0085] Where C is the RGB value of the pixel corresponding to a 3D virtual ray, r represents the ray, d is the ray direction vector, lowercase c represents the RGB value of the spatial point, and σ is the density value. n and t nf T(t) calculates the cumulative projection rate of light up to a point on the ray, where T is the nearest and farthest point on the ray, respectively, through integration.

[0086] Figure 5 A schematic flowchart of an image processing method according to an embodiment of this disclosure is shown below. (Refer to...) Figure 5 As shown, in some practical applications, image processing methods include the following processes.

[0087] First, the video stream to be processed is acquired, and video frames are selected as images to be processed. These images are then input into the SFM algorithm module 501 to predict the camera pose. Based on the pose of the images to be processed, a 3D virtual ray is generated along the direction of each pixel in the image to be processed, centered on the observer. Several spatial points are sampled along each ray.

[0088] Secondly, the temporal information of the image to be processed, as well as the coordinates and orientation information of spatial points, can be input to form the spatiotemporal features of the spatial points. These spatiotemporal features are then input into the category probability prediction module 502 to estimate the probability of each spatial point belonging to its category (including static domain, rigid body motion domain, and non-rigid body motion domain). The category probability prediction module 502 can be implemented by a general convolutional network or a fully connected neural network. The last layer can have three neurons, and the three output values ​​can be normalized, representing the probabilities of belonging to the static domain, rigid body motion domain, and non-rigid body motion domain, respectively. The spatiotemporal features of spatial points can be input into all three defined domains to calculate the feature values ​​for each category. Specifically, for the static class, the first feature extraction network 503 directly extracts the feature values ​​(i.e., motion features in the static domain); for the rigid body motion class, the displacement estimation network 504 is first used to estimate the displacement, which is then added to the original position as a new coordinate and input into the first feature extraction network 503 for feature extraction to obtain the motion features in the rigid body motion domain. For non-rigid body motion classes, we directly treat them as new objects and use the second feature extraction network 505 to predict new feature vectors (i.e., motion features in the non-rigid body motion domain). Finally, we use the feature combination module 506 to perform feature combination, that is, to linearly combine the probability and motion feature results together to obtain motion combination features.

[0089] Then, the spatial points can be transformed from three dimensions to two-dimensional pixel coordinates based on the camera pose and intrinsic parameters. The specific process is as follows: for each spatial point, its spatial coordinates are transformed from the world coordinate system to the camera coordinate system to obtain the first intermediate coordinates; the first intermediate coordinates are transformed from the camera coordinate system to the image coordinate system to obtain the second intermediate coordinates; the second intermediate coordinates are transformed from the image coordinate system to the pixel coordinate system to obtain the two-dimensional coordinates corresponding to each spatial point.

[0090] Next, bilinear interpolation is performed on the motion combination features of each spatial point to obtain the difference features at the corresponding two-dimensional coordinates. Simultaneously, each spatial coordinate can be encoded to obtain a spatial coordinate code for vector alignment, using high-frequency information to represent the original spatial coordinate information. Then, the difference features and spatial coordinate codes can be input into the neural radiation field network module 507 to output the color and density values ​​at that point.

[0091] Finally, the 3D volume rendering module 508 can be used to perform volume rendering operations along the sampled spatial points of each 3D virtual ray. Specifically, if each ray has n sampling points, the neural rendering network needs to be queried n times. The values ​​of these n points are calculated to obtain the color value of a point on the redrawn image, thus realizing the rendering of a point. If the pixel size of the redrawn image is (H, W), then repeating H×W times will generate the redrawn image.

[0092] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0093] Figure 6 A block diagram of an image processing apparatus 600 according to an embodiment of the present disclosure is shown; as follows: Figure 6 As shown, the system includes: an acquisition module 601, used to acquire the image to be processed carrying time information and the target viewpoint; a construction module 602, used to construct multiple spatial points of the image to be processed under the original viewpoint of the image to be processed, and to determine the spatiotemporal features of each spatial point based on the time information and the spatial coordinates of each spatial point; a feature extraction module 603, used to determine the probability of each spatial point belonging to each motion domain based on the spatiotemporal features of each spatial point, and to determine the motion features of each spatial point under each motion domain based on the spatiotemporal features of each spatial point; wherein, the motion domain includes: static domain, rigid body motion domain and non-rigid body motion domain; a feature combination module 604, used to determine the motion combination features of each spatial point based on the probability and motion features; and a rendering module 605, used to render based on the target viewpoint, the spatial coordinates of each spatial point and the motion combination features of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0094] The image processing apparatus provided in this embodiment can first preprocess the image to be processed, which carries temporal information, to obtain the spatiotemporal features of multiple spatial points related to the image. Then, a trained neural network model can predict the probability of each spatial point belonging to different motion domains and its motion features in each motion domain based on the spatiotemporal features. The probabilities and motion features are then combined to obtain the motion combination features of the spatial points. Finally, rendering can be performed based on a specified target viewpoint, the spatial coordinates of the spatial points, and the motion combination features to obtain a redrawn image of the image to be processed from the target viewpoint. Therefore, this method can predict and extract features from spatial points in an image to be processed that carries temporal information, obtaining the motion combination features of each spatial point. This allows for rendering of spatial points belonging to different motion domain categories based on the target viewpoint during rendering, improving processing efficiency and saving computational resources.

[0095] In some embodiments, the construction module 602 constructs multiple spatial points of the image to be processed from the original viewpoint of the image to be processed, including: processing the image to be processed by a motion recovery structure algorithm to obtain a camera pose corresponding to the image to be processed, and using the camera pose as the original viewpoint of the image to be processed; constructing a three-dimensional virtual ray based on the original viewpoint and the pixels in the image to be processed; and sampling on each three-dimensional virtual ray to obtain multiple spatial points.

[0096] In some embodiments, the feature extraction module 603 determines the probability that each spatial point belongs to each motion domain based on the spatiotemporal characteristics of each spatial point, including: processing the spatiotemporal characteristics of each spatial point through a motion domain classification model to obtain the probability that each spatial point belongs to a static domain, a rigid body motion domain, and a non-rigid body motion domain.

[0097] In some embodiments, the feature extraction module 603 determines the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point, including: processing the spatiotemporal characteristics of each spatial point through a static feature extraction model to obtain the static domain motion characteristics of each spatial point; processing the spatiotemporal characteristics of each spatial point through a displacement prediction model to obtain the post-displacement spatiotemporal characteristics of each spatial point, and processing the post-displacement spatiotemporal characteristics of each spatial point through a static feature extraction model to obtain the rigid body motion domain motion characteristics of each spatial point; and processing the spatiotemporal characteristics of each spatial point through a dynamic feature extraction model to obtain the non-rigid body motion domain motion characteristics of each spatial point.

[0098] In some embodiments, the feature combination module 604 determines the motion combination features of each spatial point based on probability and motion features, including: determining the feature components of each spatial point in the static domain based on the static domain motion features of each spatial point and the probability that each spatial point belongs to the static domain; determining the feature components of each spatial point in the rigid body motion domain based on the rigid body motion domain motion features of each spatial point and the probability that each spatial point belongs to the rigid body motion domain; determining the feature components of each spatial point in the non-rigid body motion domain based on the non-rigid body motion domain motion features of each spatial point and the probability that each spatial point belongs to the non-rigid body motion domain; and determining the motion combination features of each spatial point based on the feature components of each spatial point in the static domain, rigid body motion domain, and non-rigid body motion domain, respectively.

[0099] In some embodiments, the rendering module 605 renders the image of the image to be processed under the target viewpoint based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point, and obtains a redrawn image of the image to be processed under the target viewpoint. This includes: processing the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features through a neural radiation field network model to obtain the color value and density value at each spatial point; and rendering the image based on the color value and density value at each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint.

[0100] In one embodiment of this disclosure, the rendering module 605 processes the target viewpoint, spatial coordinates of each spatial point, and motion combination features through a neural radiation field network model to obtain color and density values ​​at each spatial point. This includes: determining the two-dimensional coordinates corresponding to each spatial point based on the spatial coordinates of each spatial point and the target viewpoint; processing the motion combination features of each spatial point using a bilinear interpolation algorithm based on the two-dimensional coordinates of each spatial point to obtain the difference features of the two-dimensional coordinates corresponding to each spatial point; encoding the spatial coordinates of each spatial point to obtain spatial coordinate codes; and processing the difference features of the two-dimensional coordinates corresponding to each spatial point and the spatial coordinate codes of each spatial point through the neural radiation field network model to obtain color and density values ​​at each spatial point.

[0101] In some embodiments, the rendering module 605 renders based on the color and density values ​​at each spatial point to obtain a redrawn image of the image to be processed from the target viewpoint, including: for each three-dimensional virtual ray, integrating based on the color and density values ​​at all spatial points sampled from the three-dimensional virtual ray to obtain the redrawn color value of the pixel corresponding to the three-dimensional virtual ray; and rendering based on the redrawn color values ​​of the pixels corresponding to all three-dimensional virtual rays to obtain a redrawn image.

[0102] Figure 6 Other aspects of the embodiments can be referred to in the other embodiments described above, and will not be repeated here.

[0103] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0104] Figure 7 This diagram illustrates a structural block diagram of an image processing computer device according to an embodiment of the present disclosure. It should be noted that the illustrated electronic device is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention.

[0105] The following reference Figure 7 To describe an electronic device 700 according to this embodiment of the present invention. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0106] like Figure 7As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).

[0107] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 710 can perform actions such as... Figure 2 The method shown.

[0108] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.

[0109] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0110] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0111] Electronic device 700 can also communicate with one or more external devices 800 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0113] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0117] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0118] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0119] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0120] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0121] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire the image to be processed and the target viewpoint carrying time information; Multiple spatial points of the image to be processed are constructed under the original viewpoint of the image to be processed, and the spatiotemporal characteristics of each spatial point are determined according to the time information and the spatial coordinates of each spatial point. The probability of each spatial point belonging to each motion domain is determined based on the spatiotemporal characteristics of each spatial point, and the motion characteristics of each spatial point in each motion domain are determined based on the spatiotemporal characteristics of each spatial point; wherein, the motion domain includes: static domain, rigid body motion domain and non-rigid body motion domain. Based on the probability and the motion characteristics, the feature components of each spatial point in each motion domain are determined, and then the motion combination characteristics of each spatial point are determined. Rendering is performed based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint. Among them, determining the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point includes: The displacement prediction model is used to process the spatiotemporal features of each spatial point to obtain the spatiotemporal features after displacement of each spatial point. The displacement prediction model is used to estimate the position offset of the object to which different pixels belong over time. The static feature extraction model is used to process the spatiotemporal features after displacement of each spatial point to obtain the rigid body motion domain motion features of each spatial point. By processing the spatiotemporal features of each spatial point through a dynamic feature extraction model, the non-rigid motion domain motion features of each spatial point are obtained.

2. The method according to claim 1, characterized in that, Multiple spatial points of the image to be processed are constructed from the original viewpoint of the image to be processed, including: The image to be processed is processed by the structure of motion recovery algorithm to obtain the camera pose corresponding to the image to be processed, and the camera pose is used as the original viewpoint of the image to be processed. A three-dimensional virtual ray is constructed based on the original viewpoint and the pixels in the image to be processed; Sampling is performed on each of the three-dimensional virtual rays to obtain the multiple spatial points.

3. The method according to claim 1, characterized in that, Determining the probability that each spatial point belongs to a specific motion domain based on the spatiotemporal characteristics of each spatial point includes: By processing the spatiotemporal characteristics of each spatial point using a motion domain classification model, the probability that each spatial point belongs to a static domain, a rigid body motion domain, or a non-rigid body motion domain is obtained.

4. The method according to claim 1, characterized in that, Determining the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point also includes: The static feature extraction model is used to process the spatiotemporal features of each spatial point to obtain the static domain motion features of each spatial point.

5. The method according to claim 4, characterized in that, Based on the probability and the motion characteristics, the feature components of each spatial point in each motion domain are determined, and then the motion combination characteristics of each spatial point are determined, including: The feature components of each spatial point in the static domain are determined based on the static domain motion characteristics of each spatial point and the probability that each spatial point belongs to the static domain. The characteristic components of each spatial point under the rigid body motion domain are determined based on the rigid body motion domain motion characteristics of each spatial point and the probability that each spatial point belongs to the rigid body motion domain. The characteristic components of each spatial point in the non-rigid motion domain are determined based on the motion characteristics of each spatial point in the non-rigid motion domain and the probability that each spatial point belongs to the non-rigid motion domain. The motion combination characteristics of each spatial point are determined based on the characteristic components of each spatial point in the static domain, rigid body motion domain, and non-rigid body motion domain.

6. The method according to claim 2, characterized in that, Rendering is performed based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint, including: The target viewpoint, the spatial coordinates of each spatial point, and the motion combination features are processed by a neural radiation field network model to obtain the color value and density value at each spatial point. Rendering is performed based on the color and density values ​​at each spatial point to obtain a redrawn image of the image to be processed from the target viewpoint.

7. The method according to claim 6, characterized in that, By processing the target viewpoint, the spatial coordinates of each spatial point, and motion combination features using a neural radiation field network model, the color and density values ​​at each spatial point are obtained, including: The two-dimensional coordinates of each spatial point are determined based on the spatial coordinates of each spatial point and the target viewpoint. Based on the two-dimensional coordinates corresponding to each spatial point, the motion combination features of each spatial point are processed using a bilinear interpolation algorithm to obtain the difference features of the two-dimensional coordinates corresponding to each spatial point. The spatial coordinates of each spatial point are encoded to obtain spatial coordinate codes; The color value and density value at each spatial point are obtained by processing the difference features of the two-dimensional coordinates corresponding to each spatial point and the spatial coordinate encoding of each spatial point through the neural radiation field network model.

8. The method according to claim 6, characterized in that, Rendering is performed based on the color and density values ​​at each spatial point to obtain a redrawn image of the image to be processed under the target viewpoint, including: For each three-dimensional virtual ray, the redraw color value of the corresponding pixel is obtained by integrating the color and density values ​​of all spatial points sampled from the three-dimensional virtual ray. The redrawn image is obtained by rendering based on the redrawn color values ​​of the corresponding pixels of all three-dimensional virtual rays.

9. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the image to be processed and the target viewpoint carrying time information; The construction module is used to construct multiple spatial points of the image to be processed from the original viewpoint, and to determine the spatiotemporal characteristics of each spatial point based on the time information and the spatial coordinates of each spatial point. The feature extraction module is used to determine the probability that each spatial point belongs to each motion domain based on the spatiotemporal characteristics of each spatial point, and to determine the motion characteristics of each spatial point in each motion domain based on the spatiotemporal characteristics of each spatial point; wherein, the motion domain includes: static domain, rigid body motion domain and non-rigid body motion domain. The feature combination module determines the feature components of each spatial point in each motion domain based on the probability and the motion features, and then determines the motion combination features of each spatial point. The rendering module is used to render the image based on the target viewpoint, the spatial coordinates of each spatial point, and the motion combination features of each spatial point, to obtain a redrawn image of the image to be processed under the target viewpoint. The feature extraction module determines the motion features of each spatial point in each motion domain based on the spatiotemporal features of each spatial point, including: The displacement prediction model is used to process the spatiotemporal features of each spatial point to obtain the spatiotemporal features after displacement of each spatial point. The displacement prediction model is used to estimate the position offset of the object to which different pixels belong over time. The static feature extraction model is used to process the spatiotemporal features after displacement of each spatial point to obtain the rigid body motion domain motion features of each spatial point. By processing the spatiotemporal features of each spatial point through a dynamic feature extraction model, the non-rigid motion domain motion features of each spatial point are obtained.

10. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the image processing method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image processing method as described in any one of claims 1 to 8.

Images