Image transmission for 3D scene reconstruction

Abstract

The present disclosure provides a method and apparatus for image transmission for three-dimensional (3D) scene reconstruction. A target image may be obtained. Geometric features and appearance features may be separated from the target image. An intermediate image may be reconstructed based on the geometric features and reference appearance features. A difference between the intermediate image and the target image may be determined. In response to determining that the difference is below a predetermined threshold, the geometric features may be sent to a receiving device for 3D scene reconstruction.

Description

Background Art

[0001] 3D scene reconstruction refers to the process of establishing a 3D mathematical model suitable for computer representation and processing of the objective world scene. It is a key technology for establishing a virtual reality that expresses the objective world in a computer. For example, in image-based 3D scene reconstruction, multiple scene images taken from different angles can be used to reconstruct 3D information and reproduce the 3D scene through a predetermined algorithm. 3D scene reconstruction has been widely used in industrial measurement, architectural design, medical imaging, 3D animation games, virtual reality (VR), etc. Summary of the invention

[0002] This Summary is provided to introduce a group of concepts that will be further described in the following Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure provide a method and apparatus for image transmission for 3D scene reconstruction. A target image may be obtained. Geometric features and appearance features may be separated from the target image. An intermediate image may be reconstructed based on the geometric features and reference appearance features. A difference between the intermediate image and the target image may be determined. In response to determining that the difference is below a predetermined threshold, the geometric features may be sent to a receiving device for 3D scene reconstruction.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 An exemplary process for 3D scene reconstruction according to an embodiment of the present disclosure is shown.

[0007] Figure 2 An exemplary process for image transmission and image reconstruction according to an embodiment of the present disclosure is shown.

[0008] Figure 3 is a schematic diagram showing exemplary positions and connection relationships of key points of a human body according to an embodiment of the present disclosure.

[0009] Figure 4 An exemplary process for training a separation model according to an embodiment of the present disclosure is shown.

[0010] Figure 5 An exemplary process for image projection according to an embodiment of the present disclosure is shown.

[0011] Figure 6 is a flowchart of an exemplary method for image transmission for 3D scene reconstruction according to an embodiment of the present disclosure.

[0012] Figure 7 An exemplary apparatus for image transmission for 3D scene reconstruction according to an embodiment of the present disclosure is shown.

[0013] Figure 8 An exemplary apparatus for image transmission for 3D scene reconstruction according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0014] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that the discussion of these embodiments is only used to enable those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0015] There are some existing technologies for performing image-based 3D scene reconstruction. These technologies collect multiple original images taken by multiple pre-deployed cameras through a server, and then send the collected multiple original images to a terminal device for reconstructing the 3D scene. Generally, in order to obtain a better 3D scene reconstruction effect, a large number of cameras with high imaging quality need to be deployed. Correspondingly, the amount of data carried by the original images taken by these cameras is also huge. Therefore, the process of sending these original images to the terminal device requires more bandwidth resources.

[0016] The embodiments of the present disclosure provide an improved method for image transmission for 3D scene reconstruction. The method may involve a process of sending information of an image for 3D scene reconstruction from a server to a receiving device that performs 3D scene reconstruction. The receiving device may be, for example, a terminal device located locally to a user and capable of reconstructing a 3D scene, such as a device based on VR technology, such as VR glasses, VR helmets, etc.

[0017] In one aspect, the embodiments of the present disclosure propose to separate geometric features and appearance features from a target image from an original image, and determine how to transmit the information of the target image to a terminal device according to whether the image reconstructed based on the geometric features and the reference appearance features meets the quality requirements. In this article, the target image can be an image extracted from the foreground image of the original image and associated with the target object, wherein the target object can broadly refer to various entities such as people and objects existing in the objective world. Taking the 3D scene reconstruction based on the live broadcast of a basketball game as an example, the target object can be a basketball player on the court, a basketball used for the game, etc. The geometric feature can refer to a set of information separated from the target image for describing the shape, posture, action, etc. of the target object, which can be represented by a small amount of data. The appearance feature can refer to a set of information separated from the target image for describing the texture, skin, color, brightness, etc. of the target object, which may need to be represented by a relatively large amount of data. The reference appearance feature can refer to an appearance feature stored at a server and / or a terminal device and associated with a specific target object. When the image reconstructed based on the geometric features and the reference appearance features associated with the same target object meets the quality requirements, only the geometric features can be sent to the terminal device; when the image reconstructed based on the geometric features and the reference appearance features associated with the same target object does not meet the quality requirements, the reference appearance features can be updated at least based on the appearance features, and the geometric features and the updated reference appearance features can be sent to the terminal device. The reference appearance features describe the texture, skin, color, brightness, etc. of the target object, which usually change little during a certain period of time, so the image reconstructed based on the geometric features and the reference appearance features can meet the quality requirements most of the time. Therefore, in the case where the terminal device has obtained and stored the reference appearance features of the target object, in most of the time, only the geometric features of the target object with a small amount of data need to be sent to the terminal device, thereby significantly reducing the amount of data transmitted, increasing the transmission speed and reducing the transmission delay. This is particularly beneficial in the case of 3D scene reconstruction based on live broadcast.

[0018] In another aspect, the embodiments of the present disclosure propose that after reconstructing the 3D scene, the 3D scene can be projected as a two-dimensional (2D) image corresponding to the perspective of the target person. In this article, the target person can be a person existing in the 3D scene. Taking the 3D scene based on the live broadcast of sports events as an example, the target person can be an athlete on the field. The target person can wear a portable device that can obtain his perspective in real time. When the user selects the target person through a terminal device associated with it, the terminal device can obtain the perspective associated with the target person in real time, and project the 3D scene in real time as a 2D image corresponding to the perspective. In this way, users can watch the game or performance from the perspective of the target person at close range, thereby obtaining a richer and more vivid viewing experience.

[0019] It should be understood that although the foregoing discussion and the following discussion may involve examples of image transmission in a 3D scene reconstruction scenario, the embodiments of the present disclosure are not limited thereto, but image transmission may be performed in a similar manner in other scenarios such as live video broadcasting.

[0020] Figure 1 An exemplary process 100 for 3D scene reconstruction according to an embodiment of the present disclosure is shown. The process 100 may reconstruct the 3D scene using at least a set of original images taken by a set of cameras. The process 100 may be performed collaboratively by a server and a terminal device.

[0021] At 110, a group of original images 112 taken by a group of cameras can be collected by a server. The group of cameras can be pre-deployed in an actual scene. Taking a sports game scene as an example, multiple cameras can be deployed at different locations such as a stadium, an auditorium, and a passage so that these cameras can capture images from different shooting angles. In one embodiment, the group of original images can be taken by the group of cameras at the same time. Accordingly, the group of original images corresponding to the same time can be used to reconstruct the 3D scene at the time through process 100. It should be understood that the group of original images can be taken by the group of cameras in real time, so that process 100 can be executed for applications involving, for example, live streaming, or the group of original images can be taken in advance by the group of cameras, so that process 100 can be executed for applications involving, for example, playing recorded content.

[0022] At 120, one or more target images may be obtained from each of the collected set of original images 112 by the server, thereby obtaining a set of target images 122. For example, a foreground image may be segmented from the original image by known methods such as foreground and background segmentation and matting, and one or more target images may be extracted from the foreground image. The foreground image may be an image about one or more target objects. For example, one or more target objects may be detected from the foreground image by a known object detection method, and for each of the one or more target objects detected, a portion corresponding to the target object may be extracted from the foreground image as a target image associated with the target object. In addition, when extracting the target image from the foreground image, the position of the target image in the foreground image may also be obtained. Preferably, the background image of each original image in the set of original images 112 may be determined in advance, and the background image may be sent to the terminal device in advance, for example, the background image may be determined and sent before the 3D scene reconstruction, thereby reducing the bandwidth usage during the 3D scene reconstruction. Usually, during a certain period of time, such as during a live broadcast, the camera that captures the original image is fixed at a specific position, and parameters such as direction and field of view are unchanged. Correspondingly, the background images of the multiple original images captured by the camera during this time period may also have small changes. The original images captured by the camera may be collected in advance, and the background images may be extracted from the pre-collected original images by methods such as foreground-background segmentation and cutout as the background images of the original images captured by the camera later. Taking the 3D scene reconstruction based on live sports events as an example, the original images used to determine the background images may be collected before the game. Alternatively, a video captured by the camera may be collected in advance, and the unchanged parts may be identified from the pre-collected video as the background images of the original images captured by the camera later. Continuing to take the 3D scene reconstruction based on live sports events as an example, the video used to determine the background image may be collected during the warm-up activities before the game. The determined background image may be used to synthesize the image for 3D scene reconstruction later.

[0023] After the set of target images 122 is obtained, an image transmission and image reconstruction process may be performed on each target image in the set of target images 122, so that at least a portion of information of the target image is sent from the server to the terminal device, and an image for 3D scene reconstruction is reconstructed at least based on the received information at the terminal device. The image transmission and image reconstruction process may be performed through steps 130 to 150.

[0024] At 130, feature separation may be performed on each target image in the obtained set of target images 122 by the server. For example, geometric features and appearance features may be separated from each target image. Figure 2 To illustrate an exemplary process for feature separation.

[0025] At 140, at least a portion of information of each target image may be sent to the terminal device through the server. The geometric features separated from the target image may be deterministically sent to the terminal device, and the appearance features may be selectively used to update the reference appearance features associated with the target image and the same target object stored at the server. When the reference appearance features are updated, the updated reference appearance features may be sent to the terminal device together with the geometric features. Figure 2 To illustrate an exemplary process for feature sending.

[0026] At 150, image reconstruction may be performed by the terminal device. In the case where the server only sends the geometric features of the target image, the terminal device may receive the geometric features and perform image reconstruction based on the geometric features and the reference appearance features stored therein and associated with the target image and the same target object. In the case where the server sends the geometric features and the updated reference appearance features, the terminal device may receive the geometric features and the updated reference appearance features and perform image reconstruction based on the geometric features and the updated reference appearance features. Figure 2 1 to illustrate an exemplary process for image reconstruction. In this article, the image obtained by image reconstruction at the terminal device may be referred to as a reconstructed image. Steps 130 to 150 may be performed for each target image in a set of target images 122, thereby obtaining a set of reconstructed images 152.

[0027] Since the reconstructed image is reconstructed based on the geometric features separated from the target image, or the geometric features and the reference appearance features, the reconstructed image only contains the information in the target image. As mentioned above, the target image is extracted from the foreground image segmented from the original image taken by the camera, so the target image only contains the information in the foreground image. Correspondingly, the reconstructed image also only contains the information in the foreground image. At 160, the terminal device can synthesize the reconstructed image as a foreground image with its corresponding background image. For example, one or more reconstructed images corresponding to one or more target images extracted from the original image can be synthesized as foreground images with the background image corresponding to the original image. For each reconstructed image, the position of the reconstructed image in the foreground image can be determined based on the position of the target image corresponding to the reconstructed image in the foreground image. In this article, the image obtained by image synthesis can be referred to as a synthesized image. Step 160 can be performed for each reconstructed image in a set of reconstructed images 152, thereby obtaining a set of synthesized images 162.

[0028] At 170, 3D scene reconstruction may be performed based at least on the set of synthetic images 162 to obtain a 3D scene 172. For example, the 3D scene may be reconstructed based on the set of synthetic images 162 and camera parameters of a set of cameras that capture a set of original images 112. The camera parameters of the camera may include, for example, spatial position coordinates of the camera in real space, the direction or orientation of the camera, field of view (FOV) parameters of the camera, and the like.

[0029] It should be understood that Figure 1 The process 100 in the embodiment is merely an example of a process for 3D scene reconstruction. Depending on the actual application requirements, the process for 3D scene reconstruction may include any other steps, and may include more or fewer steps. For example, instead of predetermining and sending a background image, a background image may be periodically determined during 3D scene reconstruction, and the determined background image may be periodically sent to a terminal device for synthesizing a synthetic image for 3D scene reconstruction. In addition, the specific order or hierarchy of the steps in the process 100 is merely exemplary, and the process for 3D scene reconstruction may be performed in an order different from the order described.

[0030] Figure 2 An exemplary process 200 for image transmission and image reconstruction according to an embodiment of the present disclosure is shown. The process 200 may be performed by a server and a terminal device in collaboration, and may correspond to Figure 1 Steps 130 to 150 in, for example, the process of separating the model 210 may correspond to Figure 1 Step 130, steps 220-270 may correspond to Figure 1 Step 140 in , and step 280 may correspond to Figure 1 Step 150 in the process 200. Through the process 200, the information of the target image 202 can be sent from the server to the terminal device, and a reconstructed image 282 corresponding to the target image 202 can be reconstructed at the terminal device. The target image 202 can correspond to Figure 1 , and the reconstructed image 282 may correspond to a target image in a set of target images 122 Figure 1 A reconstructed image in the set of reconstructed images 152 in , corresponding to the target image 202 .

[0031] The target image 202 may be provided to a separation model 210 at the server. The separation model 210 may be a neural network-based model, such as a residual network (ResNet)-based model. The separation model 210 may separate geometric features 212 and appearance features 214 from the target image 202. Figure 4 To illustrate an exemplary process for training a separation model.

[0032] Taking the case where the target object associated with the target image 202 is a person as an example, the geometric features 212 separated from the target image 202 may be an information set for characterizing the skeletal structure of the human body, which may include, for example, information related to the positions of various key points of the human body in the target image and / or the connection relationship between the key points. Figure 3 300 is a schematic diagram showing exemplary positions and connection relationships of key points of a human body according to an embodiment of the present disclosure. FIG300 shows 18 key points extracted from the skeletal structure of a human body, including: head 302, neck 304, right shoulder 306, right elbow 308, right wrist 310, right hand 312, left shoulder 314, left elbow 316, left wrist 318, left hand 320, right hip 322, right knee 324, right ankle 326, right foot 328, left hip 330, left knee 332, left ankle 334 and left foot 336. In addition, FIG300 also shows the connection relationships between these key points through lines, for example, there is a connection relationship between the head 302 and the neck 304, there is a connection relationship between the neck 304 and the right hip 322, and so on. It should be understood that Figure 3The number and position of the key points shown in and the connection relationship between the key points are all exemplary. Depending on the specific application scenario and requirements, different numbers of key points at different positions and the connection relationship between the key points can be marked. In addition, it should be understood that for other types of target objects, any other number of key points at any other positions and their connection relationships that help characterize the shape, posture, action, etc. of the target object can be marked. The geometric feature 212 can be represented by a small amount of data. For example, in the case where the geometric feature 212 includes information about the positions of 18 key points of the human body in the target image, the geometric feature 212 can be a vector with a dimension of 2×18, where "2" indicates that the position of the key point in the target image is represented by a two-dimensional coordinate.

[0033] Return to reference Figure 2 , continuing to take the example that the target object associated with the target image 202 is a person, the surface features 214 separated from the target image 202 may be a set of information for characterizing the person's skin, skin color, clothing, etc. Compared with the geometric features 212, the surface features 214 may need to be represented by a relatively large amount of data.

[0034] The server may store reference appearance features 216 associated with the target image 202 and the same target object. The reference appearance features 216 may be associated with a specific camera. For example, the reference appearance features 216 may be associated with a camera that captures the original image corresponding to the target image 202. In this case, the reference appearance features 216 may be appearance features separated from a previous target image from the specific camera. Alternatively, the reference appearance features 216 may be associated with a group of cameras, which may be, for example, cameras that captured the original image with the same target object at the same time. In this case, the reference appearance features 216 may be appearance features generated based on a group of appearance features separated from a previous group of target images from the group of cameras. For example, a comprehensive appearance feature may be generated by weighted averaging the group of appearance features, and the comprehensive appearance feature may be used as a reference appearance feature. The reference appearance features associated with a group of cameras may include more complete and stable appearance information of the target object.

[0035] According to an embodiment of the present disclosure, how to transmit the information of the target image 202 to the terminal device can be determined according to whether the image reconstructed based on the geometric features 212 and the reference appearance features 216 meets the quality requirements. When the image reconstructed based on the geometric features 212 and the reference appearance features 216 meets the quality requirements, only the geometric features 212 can be sent to the terminal device; and when the image reconstructed based on the geometric features 212 and the reference appearance features 216 does not meet the quality requirements, the reference appearance features 216 can be updated based on at least the appearance features 214, and the geometric features 212 and the updated reference appearance features can be sent to the terminal device.

[0036] At 220, image reconstruction can be performed by the server based on the geometric features 212 and the reference appearance features 216 to obtain an intermediate image 222. In this article, the image reconstructed at the server based on the geometric features and the reference appearance features can be referred to as an intermediate image. The image reconstruction at 220 can be performed, for example, by known differentiable rendering. Differentiable rendering can calculate the derivative of the entire simulation for input features, such as the geometric features 212 and the reference appearance features 216, and combined with a gradient-based optimization algorithm can help solve inverse rendering problems related to light and the like.

[0037] After reconstructing the intermediate image 222, it can be determined whether the quality of the intermediate image 222 meets the predetermined requirement. In one embodiment, it can be determined whether the quality of the intermediate image 222 meets the predetermined requirement by determining whether the difference between the intermediate image 222 and the target image 202 is lower than a predetermined threshold.

[0038] At 230, the difference between the intermediate image 222 and the target image 202 may be determined to obtain a difference 232. In one embodiment, the difference 232 may be determined by determining the difference between the intermediate image 222 and the target image 202 pixel by pixel. For example, in the case where the pixels are represented using an RGB (Red; Green; Blue) color space, each pixel of the intermediate image 222 and the target image 202 may be represented as a three-dimensional vector having three components corresponding to R value, G value, and B value, respectively. A sub-difference between each pixel in a set of pixels of the intermediate image 222 and a corresponding pixel in a set of pixels of the target image 202 may be first calculated to obtain a set of sub-differences. For example, the sub-difference between the two pixels may be calculated by calculating the Euclidean distance between the two pixels. Subsequently, the difference 232 may be calculated based on the set of sub-differences. For example, the difference 232 may be calculated as an average value of the set of sub-differences. It should be understood that a group of pixels of the intermediate image 222 and a group of pixels of the target image 202 used to calculate the sub-difference may be all pixels of the intermediate image 222 and all pixels of the target image 202, respectively. Alternatively, a group of pixels of the intermediate image 222 and a group of pixels of the target image 202 used to calculate the sub-difference may be a portion of pixels selected from the intermediate image 222 and a portion of pixels selected from the target image 202, respectively. In addition, it should be understood that the above-mentioned method for determining the difference between the intermediate image and the target image is only exemplary. According to actual application requirements, the difference between the intermediate image and the target image may also be determined in other ways.

[0039] After determining the difference 232 , at 240 , it may be determined whether the difference 232 is below a predetermined threshold.

[0040] If it is determined at 240 that the difference 232 is below a predetermined threshold, the process 200 may proceed to 250, where only the geometric features 212 separated from the target image 202 may be sent to the terminal device. As previously described, the intermediate image 222 is reconstructed based on the geometric features 212 separated from the target image 202 and the reference appearance features 216 stored by the server. The reference appearance features 216 may be sent in advance and stored in the terminal device. For example, the reference appearance features 216 may be sent to the terminal device before the server obtains the target image 202. Therefore, in this case, it may not be necessary to send the appearance features 214 to the terminal device. After receiving the geometric features 212, at 280, the terminal device may perform image reconstruction based on the received geometric features 212 and the stored reference appearance features 216 to obtain a reconstructed image 282.

[0041] If it is determined at 240 that the difference 232 is not less than the predetermined threshold, the process 200 may proceed to 260, where the reference appearance feature 216 may be updated to obtain an updated reference appearance feature 262. In the case where the reference appearance feature 216 is associated with a specific camera, the reference appearance feature 216 may be updated to the appearance feature 214 separated from the target image 202, that is, the updated reference appearance feature 262 may be the appearance feature 214. In the case where the reference appearance feature 216 is associated with a group of cameras, a group of target images associated with the target image 202 and the same target object may be obtained, and the group of target images may be from a group of original images taken by the group of cameras at the same time. The group of target images may include the target image 202. In one embodiment, it may be determined whether two target images are associated with the same target object based on the position of the target object associated therewith. A comprehensive appearance feature may be generated based on a group of appearance features separated from the group of target images. For example, the comprehensive appearance feature may be generated by weighted averaging the group of appearance features. Then, the reference apparent feature 216 may be updated to the comprehensive apparent feature, that is, the updated reference apparent feature 262 may be the comprehensive apparent feature. At 270, the geometric features 212 separated from the target image 202 and the updated reference apparent feature 262 may be sent to the terminal device. The process 200 may proceed to 280, where the terminal device may perform image reconstruction based on the received geometric features 212 and the updated reference apparent feature 262 after receiving the geometric features 212 and the updated reference apparent feature 262 to obtain a reconstructed image 282. In addition, the reference apparent feature 216 at the terminal device may also be updated to the updated reference apparent feature 262.

[0042] In process 200, geometric features 212 are deterministically sent to the terminal device, and only when the image reconstructed based on geometric features 212 and reference appearance features 216 does not meet the quality requirements, the reference appearance features 262 updated based on at least appearance features 214 will be sent to the terminal device. Reference appearance features 216 describe the texture, skin, color, brightness, etc. of the target object, which usually change little during a certain period of time. Taking the scene of live sports events as an example, during a certain period of time, the skin, clothing, etc. of the athletes on the field change little. Therefore, the image reconstructed based on geometric features 212 and reference appearance features 216 can meet the quality requirements most of the time. Accordingly, most of the time, only geometric features 212 with a small amount of data need to be sent to the terminal device, thereby significantly reducing the amount of data transmitted, increasing the transmission speed and reducing the transmission delay. This is particularly beneficial in the case of 3D scene reconstruction based on live broadcasts, especially based on live sports events with more target objects.

[0043] It should be understood that Figure 2 The process 200 in the figure is merely an example of a process for image transmission and image reconstruction. Depending on the actual application requirements, the process for image transmission and image reconstruction may include any other steps, and may include more or fewer steps. For example, the process for determining whether the quality of the intermediate image 222 meets the predetermined requirements may also be performed by a trained neural network-based model. In addition, the specific order or hierarchy of the steps in the process 200 is merely exemplary, and the process for image transmission and image reconstruction may be performed in an order different from the described order.

[0044] Figure 4 An exemplary process 400 for training a separation model according to an embodiment of the present disclosure is shown. The process 400 may utilize a training image set 402 to train a separation model 410, so that when actually deployed, the trained separation model 410 may separate geometric features and appearance features from an input image. The separation model 410 may correspond to Figure 2 Each training image in the training image set 402 may be a target image associated with only one target object. The following takes the training image 404 in the training image set 402 as an example to illustrate an exemplary process for training the separation model 410.

[0045] The training image 404 may be provided to a separation model 410. The separation model 410 may separate the training geometric features 412 and the training appearance features 414 from the training image 404. At 420, image reconstruction may be performed based on the training geometric features 412 and the training appearance features 414 to obtain a training intermediate image 422. Figure 2 The image reconstruction at 420 may be performed in a manner similar to the image reconstruction at 220 in FIG. At 430, a difference between the training intermediate image 422 and the training image 404 may be determined to obtain a difference 432. Figure 2 The difference determination at 430 is performed in a similar manner to the difference determination at 230 in FIG. The separation model 410 can be optimized by minimizing the difference 432. The process 400 can be performed for each training image in the training image set 402, so that the separation model 410 can be continuously optimized. It should be understood that Figure 4 The process 400 in FIG. 4 is merely an example of a process for training a separation model. According to actual application requirements, the process for training a separation model may include any other steps, and may include more or fewer steps.

[0046] pass Figure 1According to the process 100 in the embodiment of the present disclosure, a 3D scene can be reconstructed. According to an embodiment of the present disclosure, the reconstructed 3D scene can be further used to implement image projection, so as to present a projection image corresponding to a target perspective associated with a target person to a user. In this article, the target perspective may refer to a perspective associated with a target person. Figure 5 An exemplary process 500 for image projection according to an embodiment of the present disclosure is shown. Process 500 may be performed by a terminal device associated with a user. Through process 500, a projection image corresponding to a target viewing angle may be projected using a 3D scene, and the projection image may be presented to the user.

[0047] At 510, a target person may be determined. Taking a 3D scene based on a live broadcast of a sports event as an example, the target person may be an athlete, referee, etc. on the field; taking a 3D scene based on a live broadcast of a concert as an example, the target person may be a singer, host, etc. on the stage. The target person may be, for example, selected by a user through a user interface of a terminal device, or automatically detected based on the user's behavior.

[0048] At 520, a target perspective 522 associated with the target person may be obtained. The target perspective 522 may be expressed in a manner similar to the camera parameters, for example, it may be expressed by at least one of spatial position coordinates, direction, and field of view parameters. The target person may wear a portable device that can obtain its perspective in real time. For example, for scenes with strong or intense confrontation, such as basketball games, sprint races, etc., the target person may carry a lightweight device that can obtain its perspective, such as a bracelet, on the hand or other body part or clothes. In addition, for scenes with weak or gentle confrontation, such as canoeing competitions, concerts, etc., the target person may also wear a device such as a miniature camera on a helmet or headband that can obtain its perspective and additionally collect images and sounds. These portable devices may send the obtained target perspective to the server in real time. The terminal device may obtain the target perspective 522 associated with the target person from the server. For example, the terminal device may send the determined identifier of the target person to the server. In response to receiving the identifier of the target person, the server may send the target perspective 522 associated with the target person to the terminal device in real time.

[0049] At 530, an image projection process may be performed. For example, the reconstructed 3D scene 532 may be used by a trained scene reconstruction network to generate a projection image 534 corresponding to the target view 522. The projection image 534 may then be presented to the user.

[0050] In process 500, when the user selects a target person through a terminal device associated with the target person, the terminal device can obtain the viewing angle associated with the target person in real time, and project the 3D scene into a 2D image corresponding to the viewing angle in real time. In this way, the user can watch the game or performance from the perspective of the target person at close range, thereby obtaining a richer and more vivid viewing experience. For example, in the case of 3D scene reconstruction based on live sports events, the user can watch the game from the perspective of the athlete on the field at close range, for example, the expression and action of the athlete near the athlete can be seen, and the rapid movement of the athlete can even be felt by watching the rapidly switching pictures, thereby experiencing the intensity and excitement of the game firsthand.

[0051] It should be understood that Figure 5 The process 500 in is only an example of a process for image projection. Depending on the actual application requirements, the process for image projection may include any other steps, and may include more or fewer steps. For example, after generating a projection image 534 corresponding to the target viewing angle 522, an image enhancement process may be performed on the projection image 534 to improve the image quality, for example, by an image enhancement network based on a generative adversarial network (GAN). In addition, it should be understood that in the case where the target person wears a device capable of additionally collecting images and sounds, the collected images and sounds can participate in the 3D scene reconstruction to obtain a more complete and rich 3D scene. In addition, the user can also send bullet screens through the terminal device associated with it to interact with other viewers.

[0052] Figure 6 is a flow chart of an exemplary method 600 for image transmission for 3D scene reconstruction according to an embodiment of the present disclosure.

[0053] At 610, a target image can be obtained.

[0054] At 620 , geometric features and appearance features may be separated from the target image.

[0055] At 630, an intermediate image may be reconstructed based on the geometric features and the reference appearance features.

[0056] At 640, a difference between the intermediate image and the target image may be determined.

[0057] At 650 , in response to determining that the difference is below a predetermined threshold, the geometric features may be sent to a receiving device for 3D scene reconstruction.

[0058] In one embodiment, the target image and the reference appearance feature may be associated with the same target object.

[0059] In one implementation, obtaining the target image may include: receiving an original image captured by a camera; and extracting the target image from the original image.

[0060] The extracting the target image from the original image may include: segmenting a foreground image from the original image; and extracting the target image from the foreground image.

[0061] The method 600 may further include: sending a predetermined background image to the receiving device.

[0062] In one embodiment, the separation of geometric features and appearance features from the target image may be performed by a separation model based on a neural network.

[0063] The separation model can be trained by the following operations: separating training geometric features and training appearance features from a training image by means of the separation model; reconstructing a training intermediate image based on the training geometric features and the training appearance features; determining the difference between the training intermediate image and the training image; and optimizing the separation model by minimizing the difference between the training intermediate image and the training image.

[0064] In one embodiment, determining the difference between the intermediate image and the target image may include: calculating sub-differences between each pixel in a set of pixels of the intermediate image and a corresponding pixel in a set of pixels of the target image to obtain a set of sub-differences; and calculating the difference based on the set of sub-differences.

[0065] In one embodiment, method 600 may further include, in response to determining that the difference is not lower than the predetermined threshold: updating the reference appearance feature; and sending the geometric feature and the updated reference appearance feature to the receiving device for 3D scene reconstruction.

[0066] The updating of the reference appearance feature may include: updating the reference appearance feature to the appearance feature.

[0067] The target image may be from a group of target images. The group of target images may be associated with the same target object and may be from a group of original images captured at the same time. The updating of the reference appearance feature may include: generating a comprehensive appearance feature based on a group of appearance features separated from the group of target images; and updating the reference appearance feature to the comprehensive appearance feature.

[0068] In one embodiment, the reconstructing the intermediate image may be performed by differentiable rendering.

[0069] In one implementation, the method 600 may further include: before obtaining the target image, sending the reference appearance feature to the receiving device.

[0070] In one implementation, the 3D scene reconstruction may be performed using at least an image reconstructed based at least on the geometric features.

[0071] It should be understood that the method 600 may also include any steps / processes of image transmission for 3D scene reconstruction according to the above-mentioned embodiments of the present disclosure.

[0072] Figure 7 An exemplary apparatus 700 for image transmission for 3D scene reconstruction according to an embodiment of the present disclosure is shown.

[0073] The apparatus 700 may include: a target image acquisition module 710 for acquiring a target image; a feature separation module 720 for separating geometric features and appearance features from the target image; an intermediate image reconstruction module 730 for reconstructing an intermediate image based on the geometric features and reference appearance features; a difference determination module 740 for determining the difference between the intermediate image and the target image; and a feature transmission module 750 for transmitting the geometric features to a receiving device for 3D scene reconstruction in response to determining that the difference is lower than a predetermined threshold. In addition, the apparatus 700 may also include any other module configured for image transmission for 3D scene reconstruction according to the above-mentioned embodiments of the present disclosure.

[0074] Figure 8 An exemplary apparatus 800 for image transmission for 3D scene reconstruction according to an embodiment of the present disclosure is shown.

[0075] The apparatus 800 may include: at least one processor 810; and a memory 820 storing computer executable instructions. When the computer executable instructions are executed, the at least one processor 810 may: obtain a target image, separate geometric features and appearance features from the target image, reconstruct an intermediate image based on the geometric features and reference appearance features, determine a difference between the intermediate image and the target image, and in response to determining that the difference is below a predetermined threshold, send the geometric features to a receiving device for 3D scene reconstruction.

[0076] In one embodiment, the separation of geometric features and appearance features from the target image may be performed by a separation model based on a neural network.

[0077] The separation model can be trained by the following operations: separating training geometric features and training appearance features from a training image by means of the separation model; reconstructing a training intermediate image based on the training geometric features and the training appearance features; determining the difference between the training intermediate image and the training image; and optimizing the separation model by minimizing the difference between the training intermediate image and the training image.

[0078] In one embodiment, the computer executable instructions, when executed, may also cause the at least one processor 810, in response to determining that the difference is not less than the predetermined threshold: to update the reference appearance features, and to send the geometric features and the updated reference appearance features to the receiving device for 3D scene reconstruction.

[0079] The target image may be from a group of target images. The group of target images may be associated with the same target object and may be from a group of original images captured at the same time. The updating of the reference appearance feature may include: generating a comprehensive appearance feature based on a group of appearance features separated from the group of target images; and updating the reference appearance feature to the comprehensive appearance feature.

[0080] It should be understood that the processor 810 may also execute any other steps / processes of the method for image transmission for 3D scene reconstruction according to the above-mentioned embodiment of the present disclosure.

[0081] The embodiments of the present disclosure provide a computer program product for image transmission for 3D scene reconstruction, including a computer program, which is executed by at least one processor to: obtain a target image; separate geometric features and appearance features from the target image; reconstruct an intermediate image based on the geometric features and reference appearance features; determine the difference between the intermediate image and the target image; and in response to determining that the difference is lower than a predetermined threshold, send the geometric features to a receiving device for 3D scene reconstruction. In addition, the computer program can also be executed to implement any other steps / processes of the method for image transmission for 3D scene reconstruction according to the above-mentioned embodiments of the present disclosure.

[0082] The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions, which, when executed, cause one or more processors to perform any operation of the method for image transmission for 3D scene reconstruction according to the embodiments of the present disclosure as described above.

[0083] It should be appreciated that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operations in the method or the order of these operations, but should cover all other equivalent changes under the same or similar concept.

[0084] It should also be appreciated that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together in function.

[0085] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gated logic unit, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using software executed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0086] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the multiple aspects provided in the present disclosure, the memory can also be located inside the processor, such as a cache or a register.

[0087] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described in this disclosure that are known or to be known to those of ordinary skill in the art are expressly incorporated herein and covered by the claims.

Claims

1. A method for image transmission for three-dimensional (3D) scene reconstruction, comprising: Obtaining a target image, wherein the target image is an image extracted from a foreground image of an original image and associated with a target object; Separating geometric features and appearance features from the target image; reconstructing an intermediate image based on the geometric features and a reference appearance feature, wherein the reference appearance feature is an appearance feature associated with a target object; determining a difference between the intermediate image and the target image; and In response to determining that the difference is below a predetermined threshold, the geometric features are sent to a receiving device for 3D scene reconstruction.

2. The method according to claim 1, wherein: The target image and the reference appearance feature are associated with the same target object.

3. The method according to claim 1, wherein: The obtaining of the target image comprises: receiving a raw image captured by a camera; and The target image is extracted from the original image.

4. The method according to claim 3, wherein: The extracting the target image from the original image comprises: Segmenting a foreground image from the original image; and The target image is extracted from the foreground image.

5. The method according to claim 4, further comprising: A predetermined background image is sent to the receiving device.

6. The method according to claim 1, wherein: The separation of geometric features and appearance features from the target image is performed by a separation model based on a neural network.

7. The method according to claim 6, wherein: The separation model is trained by: Separating training geometric features and training appearance features from the training image by using the separation model; reconstructing a training intermediate image based on the training geometric features and the training appearance features; determining a difference between the training intermediate image and the training image; as well as The separation model is optimized by minimizing the difference between the training intermediate images and the training images.

8. The method according to claim 1, wherein: Determining the difference between the intermediate image and the target image comprises: calculating a sub-difference between each pixel in a set of pixels of the intermediate image and a corresponding pixel in a set of pixels of the target image to obtain a set of sub-differences; and The difference is calculated based on the set of sub-differences.

9. The method of claim 1 , further comprising, in response to determining that the difference is not below the predetermined threshold: updating the reference appearance characteristics; and The geometric features and the updated reference appearance features are sent to the receiving device for 3D scene reconstruction.

10. The method according to claim 9, wherein: The updating of the reference appearance features comprises: The reference appearance feature is updated to the appearance feature.

11. The method according to claim 9, wherein: The target image comes from a group of target images, the group of target images are associated with the same target object and come from a group of original images taken at the same time, and updating the reference appearance feature includes: generating a composite appearance feature based on a set of appearance features separated from the set of target images; and The reference appearance feature is updated to the comprehensive appearance feature.

12. The method according to claim 1, wherein: The reconstructing of the intermediate image is performed by differentiable rendering.

13. The method according to claim 1, further comprising: Before obtaining the target image, the reference appearance feature is sent to the receiving device.

14. The method according to claim 1, wherein: The 3D scene reconstruction is performed using at least an image reconstructed based at least on the geometric features.

15. An apparatus for image transmission for three-dimensional (3D) scene reconstruction, comprising: at least one processor; as well as a memory storing computer executable instructions that, when executed, cause the at least one processor to: obtaining a target image, wherein the target image is an image associated with a target object extracted from a foreground image of an original image, Separating geometric features and appearance features from the target image, reconstructing an intermediate image based on the geometric features and a reference appearance feature, wherein the reference appearance feature is an appearance feature associated with the target object, determining a difference between the intermediate image and the target image, and In response to determining that the difference is below a predetermined threshold, the geometric features are sent to a receiving device for 3D scene reconstruction.

16. The device according to claim 15, wherein: The separation of geometric features and appearance features from the target image is performed by a separation model based on a neural network.

17. The device according to claim 16, wherein: The separation model is trained by: Separating training geometric features and training appearance features from the training image by using the separation model; reconstructing a training intermediate image based on the training geometric features and the training appearance features; determining a difference between the training intermediate image and the training image; as well as The separation model is optimized by minimizing the difference between the training intermediate images and the training images.

18. The device according to claim 15, wherein: The computer executable instructions, when executed, further cause the at least one processor, in response to determining that the difference is not below the predetermined threshold: updating the reference appearance characteristics, and The geometric features and the updated reference appearance features are sent to the receiving device for 3D scene reconstruction.

19. The device according to claim 18, wherein: The target image comes from a group of target images, the group of target images are associated with the same target object and come from a group of original images taken at the same time, and updating the reference appearance feature includes: generating a composite appearance feature based on a set of appearance features separated from the set of target images; and The reference appearance feature is updated to the comprehensive appearance feature.

20. A computer program product for image transmission of three-dimensional (3D) scene reconstruction, comprising a computer program, the computer program being executed by at least one processor for: Get the target image, where The target image is an image extracted from the foreground image of the original image and associated with the target object; Separating geometric features and appearance features from the target image; reconstructing an intermediate image based on the geometric features and a reference appearance feature, wherein the reference appearance feature is an appearance feature associated with a target object; determining a difference between the intermediate image and the target image; and In response to determining that the difference is below a predetermined threshold, the geometric features are sent to a receiving device for 3D scene reconstruction.

Images