Image synthesis method and image synthesis system

Abstract

The invention provides an image synthesis solution and aims to correct the proportional relation between a person and a 3D virtual scene, and the correction mode is that the position of a virtual camera in a virtual world is adjusted to be matched with the position of a camera used for shooting the person in the real world.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to an image synthesis method and an image synthesis system. Background Art

[0002] There are many video-related software applications on the market for different purposes, such as video conference software such as Zoom, Microsoft Teams, and Google Meet, and live streaming software such as Twitch, YouTube Live, and Open Broadcaster Software (OBS). In addition to the basic image transmission function, these software have also developed many auxiliary image processing applications, such as background blur, autofocus, face detection, background replacement, etc. Among them, the traditional common background replacement function uses 2D images as the replacement target. Its advantage is that it allows users to choose their favorite background images to achieve the purpose of customization. In addition, it can also block the user's current real-life environment to maintain personal privacy. However, its disadvantage is that the character itself is a 3D entity and therefore has a poor sense of integration with the 2D background image. In addition, the 2D background image cannot change dynamically (for example, it cannot move the perspective or change the relative position of the character in the background), and it looks unchanged.

[0003] Considering the above advantages and disadvantages, an application with a 3D virtual scene as the background is expected to have market demand, because it can better integrate characters into the background and present richer and more interesting visual effects than an application with a 2D picture as the background. It can even allow users to customize virtual scenes, such as scenes in their favorite games, greatly improving the user experience.

[0004] The above-mentioned applications with 3D virtual scenes as backgrounds may involve technologies such as real-time image background removal, virtual scene creation, conversion from 3D scenes to 2D images, superposition of character images and virtual scenes, etc. One of the technical difficulties encountered is the problem of size adaptation between characters and scenes. If this problem is ignored or not properly handled, the following will occur: Figure 1A and Figure 1B The situation shown. Figure 1A and Figure 1B The following are two examples of synthetic images in which the proportions between the characters and the scene are out of balance. Figure 1A The figure 10A in the figure is too large relative to the background scene. Figure 1B The proportion of the person 10B in the image is too small relative to the background scene. These two synthetic images are very abrupt for the user.

[0005] Therefore, a solution for image synthesis is needed that can overcome the above technical difficulties. Summary of the invention

[0006] The embodiment of the present disclosure provides an image synthesis method implemented by a computer system. The method includes: obtaining a character image of a character photographed by a photographic device; generating a 3D virtual scene based on multiple virtual scene resources, and setting a reference object with the same real size as the character in the 3D virtual scene; setting a first virtual camera so that the first virtual camera faces the reference object in the 3D virtual scene; determining an ideal position of the first virtual camera based on a first distance between the photographic device and the character when photographing the character image; moving the first virtual camera to an ideal position, and using the first virtual camera located at the ideal position to project the 3D virtual scene onto a virtual scene layer; using an image segmentation model to separate a character layer from the character image; superimposing the character layer onto the virtual scene layer to generate a pair of superimposed layers; using a second virtual camera to project the pair of superimposed layers onto a 2D image; and rendering the 2D image to obtain a synthesized image.

[0007] In one embodiment, the method further comprises calculating a first visual size of the person in the person image using an object recognition model; and calculating a real size based on the first visual size, the first distance, and a field of view (FoV) parameter of the camera device.

[0008] In one embodiment, after determining the ideal position of the first virtual camera and before moving the first virtual camera to the ideal position, the method further includes: projecting the 3D virtual scene onto the virtual scene image; using a raycasting algorithm to calculate the second visual size of the reference object in the virtual scene image; and determining the correction amount required to move the first virtual camera relative to the ideal position based on the first visual size, the second visual size and the first distance. The step of moving the first virtual camera to the ideal position further includes moving the first virtual camera according to the correction amount.

[0009] In one embodiment, the step of determining the correction amount required to move the first virtual camera relative to the ideal position further includes determining the correction amount required to move the first virtual camera relative to the ideal position based on the first visual size, the second visual size, the first distance and the offset.

[0010] In one embodiment, the method further comprises obtaining the first distance from a photographing device, wherein the photographing device is a depth camera.

[0011] In one embodiment, the method further includes estimating the first distance using a depth estimation model based on the image of the person.

[0012] In one embodiment, the method further includes: using a gesture recognition model to recognize a gesture in a person image; determining whether the gesture is mapped to a specified operation; and if it is determined that the gesture is mapped to a specified operation, adjusting the ideal position of the first virtual camera according to the specified operation.

[0013] In one embodiment, the virtual scene resources include mesh resources, texture resources, shader resources, and material resources.

[0014] In one embodiment, the computer system includes a photographing device.

[0015] In one embodiment, the computer system is connected to a mobile device, and the mobile device includes a camera.

[0016] The embodiment of the present disclosure further provides an image synthesis system, which includes a storage device and a processing device. The processing device loads a program from the storage device to perform the following steps: obtain a character image of a character photographed by a photographic device; generate a 3D virtual scene based on multiple virtual scene resources, and set a reference object with the same real size as the character in the 3D virtual scene; set a first virtual camera so that the first virtual camera faces the reference object in the 3D virtual scene; determine the ideal position of the first virtual camera based on a first distance between the photographic device and the character when shooting the character image; move the first virtual camera to the ideal position, and use the first virtual camera located at the ideal position to project the 3D virtual scene onto a virtual scene layer; use an image segmentation model to separate a character layer from the character image; superimpose the character layer onto the virtual scene layer to generate a pair of superimposed layers; use a second virtual camera to project the pair of superimposed layers onto a 2D image; and render the 2D image to obtain a synthesized image.

[0017] The image synthesis solution provided by the present disclosure adjusts the position of the virtual camera and ensures that it matches the position of the camera in the real world, thereby achieving proportion correction to overcome the size adaptation problem between the character and the 3D virtual scene, so that the synthesized image presents a more natural and coordinated visual effect. In this way, the characters in the synthesized image will no longer appear to be too large or too small, and the coordination and naturalness of the image are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present disclosure will be better understood from the following illustrative embodiments with accompanying drawings. In addition, it should be understood that in the flowchart of the present disclosure, the execution order of each block may be changed, and / or some blocks may be changed, deleted or merged.

[0019] Figure 1AA pattern of a synthetic image showing a person being oversized relative to the background scene.

[0020] Figure 1B A pattern of a synthetic image showing a person being too small in proportion to the background scene.

[0021] Figure 2A The diagram is a schematic diagram illustrating a situation in which a person's image is captured by a camera in the real world according to an embodiment of the present disclosure.

[0022] Figure 2B It is a schematic diagram illustrating a scenario of using a virtual camera to capture a 3D virtual scene in a virtual world according to an embodiment of the present disclosure.

[0023] Figure 3 The present invention is a flowchart of an image synthesis method according to an embodiment of the present invention.

[0024] Figure 4A It is a flow chart showing the calculation of the real size according to the preferred embodiment of the present disclosure.

[0025] Figure 4B It is a schematic diagram illustrating multiple values ​​required for calculating the actual size according to a preferred embodiment of the present disclosure.

[0026] Figure 5 The present invention is a flowchart illustrating a procedure for determining a correction amount of a first virtual camera according to an embodiment of the present invention.

[0027] Fig. 6A The present invention is a system block diagram of an image synthesis system according to an embodiment of the present invention.

[0028] Figure 6B is a system block diagram of an image synthesis system according to another embodiment of the present disclosure.

[0029] The reference numerals are described as follows:

[0030] 10A: People

[0031] 10B: People

[0032] 200: Characters

[0033] 201: Camera

[0034] 202: Image of people

[0035] D1: First distance

[0036] 210: Reference Object

[0037] 211: Virtual Camera

[0038] 212: 3D virtual scene

[0039] D2: Second distance

[0040] 300: Image synthesis method

[0041] S301-S309: Steps

[0042] S401-S402: Steps

[0043] 410: Characters

[0044] 411: Photographic Installation

[0045] 412: Image of people

[0046] S501-S503: Steps

[0047] 600A: Image synthesis system

[0048] 601: Photographic Installation

[0049] 602: Storage device

[0050] 603: Processing device

[0051] 604: Display device

[0052] 611: Central Processing Unit (CPU)

[0053] 612: Graphics Processing Unit (GPU)

[0054] 613: Neural Network Processing Unit (NPU)

[0055] 600B: Image synthesis system

[0056] 605: Mobile Devices DETAILED DESCRIPTION

[0057] The following description lists various embodiments of the present invention, but is not intended to limit the content of the present invention. The actual scope of the invention is defined by the claims.

[0058] In the various embodiments listed below, the same reference numerals will be used to represent the same or similar elements or components.

[0059] In this specification and claims, serial numbers, such as "first", "second", etc., are only for convenience of description and have no sequential relationship with each other.

[0060] It should be clarified that the term "size" is often used to roughly describe the size of an object in space, but the "dimension" mentioned in this article specifically refers to the measurement of an object in a specific direction, such as length, width, height, depth or diameter.

[0061] The following description of the embodiments of the apparatus or system is also applicable to the embodiments of the method, and vice versa.

[0062] In 3D imaging technology, a virtual camera system is usually used to capture and present 3D scenes. The virtual camera system can simulate the camera or eyes in traditional photography and convert the 3D scene into a 2D image. In this way, the audience can watch the 3D scene in 2D form and even interact with it. Technically speaking, the virtual camera in the virtual camera system does not really shoot the scene in the real world like a normal camera, but uses 3D projection technology to map the scene captured from the virtual world to a 2D image.

[0063] The 3D projection technology generally includes orthographic projection and perspective projection. In orthographic projection, the size of an object in an image is not affected by the distance between the camera and the object. No matter how far the object is from the camera, its size in the image remains constant. In perspective projection, the distance between the camera and the object affects the size of the object in the image. Specifically, the closer the object is to the camera, the larger the object will appear in the image. In order to reflect the perspective changes caused by the distance of objects in the real world, perspective projection is usually used, whether it is a general camera or a virtual camera.

[0064] therefore, Figure 1A and Figure 1B The problem of the inappropriate proportion between the characters and the scene is caused by the imbalance in the shooting distance between the ordinary camera in the real world and the virtual camera in the virtual world. Figure 1A For example, the proportion of the person 10A relative to the background scene is too large because the camera shooting the person is relatively close to the person, while the virtual camera captures the virtual scene at a relatively far distance. Figure 1B Generally speaking, the proportion of the character 10B relative to the background scene is too small because the camera shooting the character is relatively far away from the character, while the virtual camera captures the virtual scene at a relatively close distance.

[0065] The image synthesis solution proposed in the present disclosure is intended to correct the proportional relationship between the character and the 3D virtual scene by adjusting the position of the virtual camera in the virtual world to match the position of the camera in the real world. The principle used is that if two target objects of the same actual size have the same visual size in the images captured by two cameras (whether ordinary cameras or virtual cameras), the distance between the two cameras and the target objects they shoot must be the same. On the other hand, if the distance between the two cameras and the target objects they shoot is the same, and the actual sizes of the two target objects are also the same, the two target objects will also have the same visual size in the images captured by the two cameras.

[0066] The following will refer to Figure 2A and Figure 2B The above-mentioned image synthesis solution is described.

[0067] Figure 2A 2 is a schematic diagram illustrating a situation in which a camera 201 captures a person image 202 of a person 200 in the real world according to an embodiment of the present disclosure. Figure 2A As shown, the distance between the camera 201 and the person 200 when shooting the person image 202 is referred to as the "first distance" hereinafter, which is D1. The size of the person 200 in the real world is referred to as the "real size" hereinafter, which is W centimeters. The size of the person 200 in the person image 202 shot by the camera 201 is referred to as the "first visual size" hereinafter, which is X pixels. It should be understood that although Figure 2A In the figure, the real size W cm of the person 200 seems to be the same as the first visual size X pixels, but the two are completely different concepts and should not be considered as comparable values. The real size is the value of the person 200 in a specific direction in the real world, such as the width of the head of the person 200. The first visual size is the number of pixels occupied by the person 200 (such as the width of its head) in the person image 202 or on the screen.

[0068] Figure 2B FIG. 2 is a schematic diagram illustrating a scenario in which a virtual camera 211 captures a 3D virtual scene 212 in a virtual world according to an embodiment of the present disclosure. Figure 2BAs shown, a reference object 210 is configured in the 3D virtual scene 212. The objects in the 3D virtual scene 212 can be drawn by 3D drawing tools, such as Autodesk Maya, Blender, Autodesk 3ds Max, Cinema 4D, etc. These drawing tools have the function of simulating the scale of the real world, which can ensure that the objects and scenes in the 3D virtual scene 212 have the same size and proportion as the real world. Therefore, in the embodiment of the present disclosure, the reference object 210 can be set to have the same real size as the character 200 in the real world (such as the width of the reference object 210 itself), that is, W centimeters. In addition, the virtual camera 211 can be set to face the reference object 210.

[0069] Since the reference object 210 and the person 200 have the same real size, if the distance between the virtual camera 211 and the reference object 210, hereinafter referred to as the "second distance", is equal to the first distance between the camera 201 and the person 200, then the size of the reference object 210 in the image projected by the virtual camera 211 from the 3D virtual scene 212, hereinafter referred to as the "second visual size", will also be equal to the first visual size of the person 200 in the person image 202. At this time, the proportion between the person 200 and the 3D virtual scene 212 is harmonious, and the result of the image synthesis will not appear as Figure 1A or Figure 1B situation.

[0070] However, in Figure 2B In the example of , it is assumed that the second distance D2 between the virtual camera 211 and the reference object 210 is greater than the first distance D1 between the camera 201 and the person 200. In other words, the shooting distance of the virtual camera 211 is farther than the shooting distance of the camera 201. Therefore, the second visual size Y pixels of the reference object 210 will be smaller than the first visual size X pixels of the person 200. At this time, the proportion of the person 200 relative to the 3D virtual scene 212 is too large, and the result of its image synthesis may appear similar to Figure 1A situation.

[0071] According to an embodiment of the present disclosure, the above Figure 2B The problem of the second distance between the virtual camera 211 and the reference object 210 can be solved by shortening the second distance between the virtual camera 211 and the reference object 210 to be equal to or substantially equal to the first distance D1. As for the first distance D1, the depth of each pixel of the person image 202 can be estimated by using a depth estimation model, or by using a depth camera as the camera 201 to measure the depth of the person 200.

[0072] Figure 3 FIG. 3 is a flow chart of an image synthesis method 300 according to an embodiment of the present disclosure. Figure 3 As shown, the method 300 may include steps S301 - S309 .

[0073] In step S301, a person image captured by a photographic device is obtained. Figure 2A For example, a person image 202 of a person 200 photographed by a camera 201 is obtained.

[0074] In step S302, a 3D virtual scene is generated based on a plurality of virtual scene resources, and a reference object having the same real size as the character is set in the 3D virtual scene. Figure 2B As an example, a 3D virtual scene 212 is generated based on multiple virtual scene resources, and the 3D virtual scene 212 is configured with Figure 2A The character 200 in FIG. 1 has the same real-world size as the reference object 210 of W centimeters.

[0075] In one embodiment, step S302 may be implemented by calling an application programming interface (API) provided by a graphics engine. The graphics engine may be, for example, Unity, Unreal Engine, OpenGL, DirectX, Vulkan, Metal, etc., but the present disclosure is not limited thereto.

[0076] In one embodiment, the virtual scene resources used when generating the 3D virtual scene in step S302 may include mesh resources, texture resources, shader resources and material resources. Mesh resources are composed of multiple vertices, edges and faces, and are used to define the size, shape and topology of 3D objects (including reference objects). Texture resources are 2D images or patterns applied to the surface of 3D objects, usually in the form of image files (such as JPEG, PNG or TGA), which include information such as color texture, normal vector texture, environment mapping, etc., which are used to add color, texture and visual effects to the surface of objects. Shader resources are a program used to control the effects of changes in light illumination, shadows, reflections, refractions, etc. on the surface of objects and their visual effects. Material resources are a collection of physical properties of materials (such as metal, plastic, wood, etc.) used by objects in a virtual scene, which are used to define the optical properties of objects, such as color, reflection, transparency, etc.

[0077] In step S303, a first virtual camera is set to face a reference object in the 3D virtual scene. Figure 2B For example, a virtual camera 211 is set to face a reference object 210 in a 3D virtual scene 212 .

[0078] In step S304, an ideal position of a first virtual camera is determined based on a first distance between the camera and the person when the camera is capturing the person's image. Figure 2A and Figure 2B For example, based on the first distance D1 between the camera 201 and the person 200 when shooting the person image 202 , the ideal position of the virtual camera 211 is determined.

[0079] In one embodiment, the ideal position is equal to the position at the first distance from the reference object. In another embodiment, it is desired that the person and the reference object are not at exactly the same position, but have a reasonable degree of offset, so the ideal position can be set at the position at the first distance from the person plus an offset. The offset can be a preset value or can be set by the user, but the present disclosure is not limited to this.

[0080] In step S305, the first virtual camera is moved to an ideal position, and the 3D virtual scene is projected onto the virtual scene layer using the first virtual camera located at the ideal position. In other words, based on the parameters associated with the first virtual camera, the 3D virtual scene captured by the first virtual camera from the virtual world is mapped onto the 2D virtual scene layer using 3D projection technology.

[0081] In step S306, the image segmentation model is used to separate the character layer from the character image. The image segmentation model can be implemented by various existing machine learning algorithms for segmenting different objects (such as characters) in the image, such as U-Net, DeepLab, Mask R-CNN, HRNet, Enet... etc., but the present disclosure is not limited to this. The training process of the image segmentation model includes the acquisition of label data (such as manually marking the segmentation results, or collecting open source label data), the selection of loss functions, and the configuration of optimization algorithms, etc., and various existing supervised learning methods can be used, but the present disclosure is not limited to this. In addition, the image segmentation model can be trained on the local side, or it can be trained on other computer devices (such as servers) first and then obtain the trained image segmentation model via the network (such as downloading from the cloud), storage media (such as external hard disk) or other communication interfaces (such as USB), but the present disclosure is not limited to this.

[0082] In step S307, the character layer is superimposed on the virtual scene layer to generate a pair of superimposed layers. In the process of superimposing the character layer and the virtual scene layer, a variety of image synthesis techniques may be involved, such as alpha blending, mask blending, raycasting, etc., to ensure that the synthesized visual effect is realistic and natural, with good overall consistency.

[0083] In step S308, the second virtual camera is used to project the pair of overlay layers onto the 2D image. In other words, based on the parameters associated with the second virtual camera, the 3D projection technology is used to map the pair of overlay layers captured by the second virtual camera onto the 2D image.

[0084] In step S309, the 2D image is rendered to obtain a synthetic image. This step may further involve processes such as ray tracing, shadow generation, ambient lighting, material assignment, etc. The present disclosure does not limit the specific implementation of the rendering technology, and different rendering methods and parameter configurations may be selected according to practical needs.

[0085] In one embodiment, the real size (that is, the size of the person in the real world) required to generate the reference object in the 3D virtual scene in step S302 can be a value input by the user. In another embodiment, the average head width of humans, such as 16.5 cm, can be applied as the real size. Since the average head width varies with gender, for example, the average head width of adult males is approximately between 16.5-18.5 cm, and the average head width of adult females is approximately between 15-16.5 cm, a gender recognition model can also be used to determine the gender of the person in the character image to determine the real size. However, in addition to gender, the actual head width may also vary due to complex factors such as race, geographical location and individual differences. Therefore, the above approach cannot guarantee the accuracy of the real size, which may affect the image synthesis result.

[0086] In a preferred embodiment, the actual size of the person can be further calculated based on the values ​​obtained from the person's image. Figure 4A and Figure 4B This preferred embodiment is described.

[0087] Figure 4A FIG. 1 is a flow chart showing the calculation of the actual size according to a preferred embodiment of the present disclosure. Figure 4A As shown, the calculation of the real size includes steps S401 and S402. Figure 4A , Figure 4B This is a schematic diagram showing multiple values ​​required for calculating the actual size according to a preferred embodiment of the present disclosure. Figure 4A and Figure 4B , to better understand this preferred embodiment.

[0088] In step S401, based on the person image 412, the object recognition model is used to calculate the first visual size of the person 410 in the person image 412. Figure 4B In the example of FIG. 1 , the first visual size is X pixels.

[0089] The inference process of the object recognition model can be roughly divided into two stages: "feature extraction" and "object classification and positioning". In the feature extraction stage, the object recognition model can extract the features of the person image 412. The features can be in the form of feature vectors, feature tensors or feature maps, representing the attributes or characteristics of the person image 412. The features of the person image 412 extracted in the feature extraction stage will be used as the input of the object classification and positioning stage. Next, in the object classification and positioning stage, object classification (classification) and localization (localization) are performed based on the extracted features to identify the person 410 and its position and range in the person image 412. Information about the position and range of the person 410 can be described by a bounding box. The bounding box refers to a rectangular box in the person image 412 that can just surround the person 410. Therefore, the size (such as length or width) of the bounding box is also the first visual size of the person 410 in the person image 412.

[0090] The object recognition model can be implemented based on one or more machine learning algorithms and their combination. For example, the object recognition model as a whole can be implemented using Region Convolutional Neural Networks (RCNN) or its variants (such as Fast R-CNN, Faster R-CNN, Mask R-CNN), or the YOLO (You Only Look Once) series of algorithms. In addition to using the convolution layers and pooling layers of the Convolutional Neural Network (CNN) to implement the feature extraction stage in its inference process, it can also be implemented using non-neural network technologies such as the Viola-Jones object detection framework, Scale-invariant feature transform (SIFT) or Histogram of oriented gradient (HOG). The object classification and positioning stage can be implemented using fully connected layers of a convolutional neural network, or existing machine learning algorithms such as a support vector machine (SVM) or a joint Bayesian.

[0091] In one embodiment, the labeled data in the training data set used as the ground truth in the training phase of the object recognition model can be collected by manually labeling the characters in multiple images of characters. In another embodiment, the labeled data can be collected from open source data sets, such as the Pascal VOC data set and the Common Objects in Context (CoCo) data set. In a further embodiment, a Generative Adversarial Network (GAN) can be used to expand the amount of labeled data.

[0092] In one embodiment, during the training phase of the object recognition model, a loss function such as mean square error (MSE), mean absolute error (MAE) or cross-entropy may be used to calculate a loss value representing the difference between the output of the object recognition model and the labeled data. Furthermore, an optimizer may be used to recursively adjust the parameters of the object recognition model (e.g., the weights of each layer of the neural network) so that the loss value is minimized, thereby optimizing the object recognition model. The optimizer may be implemented using algorithms such as gradient descent (GD), stochastic gradient descent (SGD) or adaptive moment estimation (Adam). Through a training process such as repeated result feedback and parameter update, the loss value is gradually reduced until the loss value converges to a minimum value.

[0093] In addition, the object recognition model can be trained locally, or it can be first trained on other computer devices (such as a server) and then the trained object recognition model can be obtained via a network (such as downloading from the cloud), a storage medium (such as an external hard drive), or other communication interfaces (such as a USB), but the present disclosure is not limited to this.

[0094] In step S402, the real size is calculated based on the first visual size, the first distance, and the field of view (FoV) parameter of the photographing device 411. Figure 4B In the example, the first visual size is X pixels, the first distance is D1, and the real size to be calculated is W centimeters. In addition, Figure 4B It is also drawn that the maximum viewing angle of the photographic device 411 is θ, the angle formed by the rays emitted by the photographic device 411 to the person 410 is ρ, and the maximum width that the photographic device 411 can capture is M pixels, wherein the maximum viewing angle θ and the maximum width M pixels can be included in the field of view parameters of the photographic device 411, or can be derived from other field of view parameters of the photographic device 411.

[0095] Depend on Figure 4B It can be seen that the mathematical relationship between the true size W and the angle ρ can be written as <Formula 1>, as follows:

[0096] <Formula 1>

[0097] W=tanρ×D1×2

[0098] Next, since the viewing angle is proportional to the projection amount, the mathematical relationship between the angle ρ, the maximum viewing angle θ, the first visual size X, and the maximum width M can be written as <Formula 2>, as follows:

[0099] <Formula 2>

[0100]

[0101] Finally, by combining <Formula 1> and <Formula 2>, we can get <Formula 3>, as follows:

[0102] <Formula 3>

[0103]

[0104] In one embodiment, Figure 3 Between step S304 and step S305, the process of determining the correction amount of the first virtual camera is further included. Figure 5 Describe the detailed steps.

[0105] Figure 5 FIG. 1 is a flow chart illustrating a procedure for determining a correction amount of a first virtual camera according to an embodiment of the present disclosure. Figure 5 As shown, the procedure for determining the correction amount of the first virtual camera may include steps S501-S503.

[0106] In step S501, the 3D virtual scene is projected onto the virtual scene image using the first virtual camera. In other words, based on the parameters associated with the first virtual camera, the 3D virtual scene captured by the first virtual camera from the virtual world is mapped onto the 2D virtual scene image using 3D projection technology. It should be understood that since step S501 is before step S305, the first virtual camera has not yet moved to the ideal position, so the parameters associated with the first virtual camera will be different from those when step S305 is executed, and the projected virtual scene image will also be different from the virtual scene layer projected by step S305.

[0107] In step S502, a raycasting algorithm is used to calculate a second visual size of the reference object in the virtual scene image. Figure 2B For example, the second visual size is Y pixels.

[0108] The ray casting algorithm simulates the line of sight of the first virtual camera, that is, the light emitted from the first virtual camera, and then checks whether the light intersects with the reference object to determine whether the reference object is in the line of sight of the first virtual camera. When the light intersects with the object, the projection position and size of the reference object in the virtual camera image can be calculated, which is the second visual size.

[0109] In step S503, based on the first visual size, the second visual size and the first distance, the correction amount required for the first virtual camera to move relative to the ideal position is determined. Figure 2A and Figure 2B For example, the first visual size is X pixels, the second visual size is Y pixels, the first distance is D1, and the correction amount to be determined is D1-D2. A positive correction amount means that the shooting distance of the first virtual camera should be lengthened, and a negative correction amount means that the shooting distance of the first virtual camera should be shortened.

[0110] The mathematical relationship between the first visual size X, the second visual size Y, the first distance D1 and the second distance D2 can be expressed as <Formula 4>, as follows:

[0111] <Formula 4>

[0112]

[0113] Based on <Formula 4>, the correction amount D1-D2 can be further written as <Formula 5>, as follows:

[0114] <Formula 5>

[0115]

[0116] After the correction amount determination process is completed, the first virtual camera can be moved according to the correction amount in step S305 to move the first virtual camera to an ideal position. Figure 2B For example, the virtual camera 211 may be moved according to the correction amount D1-D2, so as to move the virtual camera 211 to an ideal position, that is, a position at a shooting distance of D1.

[0117] In one embodiment, it is hoped that the person and the reference object are not in exactly the same position, but have a reasonable degree of offset. Therefore, the aforementioned correction amount should also take the offset value into consideration. More specifically, the result of D1-D2 calculated by <Formula 5> needs to be added with the offset as the correction amount. The offset can be a preset value or can be set by the user, but the present disclosure is not limited to this.

[0118] In one embodiment, when photographing an image of a person, the first distance between the photographing device and the person is, for example, Figure 2A or Figure 4B The first distance D1 in the figure can be obtained from the photographic device itself, and the photographic device itself is a depth camera, such as a time-of-flight (ToF) camera, a structured light camera, a stereovision camera, etc., which can perceive the depth information of an object (including a person).

[0119] In one embodiment, the first distance between the photographic device and the person when shooting the person image can be estimated based on the person image using a depth estimation model. The depth estimation model can be implemented using an algorithm based on a convolutional neural network, but the present disclosure is not limited to this. The training process of the depth estimation model includes the acquisition of label data (for example, manually marking the depth, or collecting open source label data), the selection of loss functions, and the configuration of optimization algorithms, etc. Various existing supervised learning methods can be used, but the present disclosure is not limited to this. In addition, the depth estimation model can be trained on the local side, or it can be trained on other computer devices (such as servers) and then the trained depth estimation model can be obtained through the network (for example, downloading from the cloud), storage media (such as an external hard disk) or other communication interfaces (such as USB), but the present disclosure is not limited to this.

[0120] In one embodiment, the image synthesis method provided by the present disclosure further includes using a gesture recognition model to recognize gestures in a person image, and then determining whether the gesture is mapped to a specified operation. If it is determined that the gesture is mapped to a specified operation, the ideal position of the first virtual camera is adjusted according to the specified operation. For example, a thumbs-up gesture to the left may represent that the user (i.e., the person object photographed by the photographic device) wants to move the first virtual camera to the left (in other words, the ideal position is moved to the left), and a thumbs-up gesture to the right may represent that the user wants to move the first virtual camera to the right (in other words, the ideal position is moved to the right). The mapping relationship between gestures and operations such as these can be pre-specified and recorded in a mapping table. In response to the user making a gesture recorded in the mapping table, the first virtual camera can be operated accordingly.

[0121] The gesture recognition model can be implemented using various existing machine learning algorithms, such as convolutional neural networks, recurrent neural networks (RNN), long short-term memory networks (LSTM), support vector machines, decision trees, and random forests, but the present disclosure is not limited to this. The training process of the gesture recognition model includes the acquisition of label data (for example, manually marking gestures, or collecting open source label data), the selection of loss functions, and the configuration of optimization algorithms, etc., and various existing supervised learning methods can be used, but the present disclosure is not limited to this. In addition, the gesture recognition model can be trained on the local side, or it can be trained on other computer devices (such as servers) and then the trained gesture recognition model can be obtained through the network (for example, downloading from the cloud), storage media (such as external hard disk) or other communication interfaces (such as USB), but the present disclosure is not limited to this.

[0122] Fig. 6A FIG. 6 is a system block diagram of an image synthesis system 600A according to an embodiment of the present disclosure. Fig. 6A As shown, system 600A may include a photographing device 601, a storage device 602, a processing device 603, and a display device 604. System 600A itself may be a personal computer (such as a desktop computer or a laptop computer) or a server computer running an operating system (such as Windows, Mac OS, Linux, UNIX, etc.), or a mobile device such as a tablet computer or a smart phone, but the present disclosure is not limited thereto.

[0123] The photographic device 601 may include a photographic lens for photographing images, and the photographic lens may include a general optical lens or an infrared lens. The present disclosure does not limit the type and number of the photographic lens. The photographic lens may be movable (e.g., rotatable to photograph images at different angles) or immovable (e.g., immovable and only capable of photographing images at fixed angles). In one embodiment, the photographic device 601 may be a depth camera, such as a time-of-flight camera, a structured light camera, a stereoscopic vision camera, or the like, which can sense the depth information of an object (including a person), and can provide a first distance between the person and the person when photographing an image of the person.

[0124] The storage device 602 may include a volatile memory (such as a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM)) and / or any one or more devices containing non-volatile memory (such as a read only memory, an electrically-erasable programmable read-only memory (EEPROM), a flash memory, a non-volatile random access memory (NVRAM)) (such as a hard disk (HDD), a solid state drive (SSD) or an optical disk) and a combination thereof, but the present disclosure is not limited thereto. In various embodiments of the present disclosure, the storage device 602 is used to store a program for implementing the aforementioned image synthesis method 300 and its various embodiments, as well as data required or generated during the operation of the program, such as the aforementioned character image, virtual scene resources, 3D virtual scene, virtual scene layer, character layer, overlay layer, synthesized image, and various machine learning models that may be used.

[0125] The processing device 603 may include any one or more general or special processors and combinations thereof for executing instructions. The processing device 603 loads the aforementioned program from the storage device 602 to execute the image synthesis method 300 and its various embodiments. The processing device 603 may include a central processing unit (CPU) 611 and a graphics processing unit (GPU) 612. The GPU 612 is an electronic circuit specially designed to perform computer graphics calculations and image processing, so it is more efficient in computer graphics calculations and image processing than the general CPU 611. Therefore, in various embodiments of the present disclosure, appropriate tasks can be assigned according to the characteristics of the CPU 611 and the GPU 612, for example, tasks for obtaining data or communicating with other devices are assigned to the CPU 611, and tasks related to computer graphics calculations and image processing are assigned to the GPU 612.

[0126] In one embodiment, the processing device 603 may further include a neural network processing unit (NPU) 613 optimized for deep learning. Compared with the GPU 612, the NPU 613 may have a greater computing performance advantage in operating the aforementioned image segmentation model, object recognition model, depth estimation model and / or gesture recognition model. Therefore, in this embodiment, the tasks involving the use of these machine learning models may be assigned to the NPU 613.

[0127] The processing device 603 may be coupled to the display device 604 to transmit the synthesized image obtained by executing the image synthesis method 300 to the display device 604, so as to display the synthesized image on the display device 604. The display device 604 may be any device for displaying visible information, such as an LCD display, an LED display, an OLED display, or a plasma display, but the present disclosure is not limited thereto.

[0128] Figure 6B FIG. 6 is a system block diagram of an image synthesis system 600B according to another embodiment of the present disclosure. Compared with the system 600A, the system 600B also includes the same storage device 602, processing device 603 and display device 604, so the hardware components are not repeated. The difference between the system 600B and the system 600A is that the system 600B does not need to include the camera device 601, but is connected to an external mobile device 605.

[0129] The mobile device 605 can be any type of smart phone or tablet computer equipped with a camera and thus having a camera function. In this embodiment, the image of the person is taken by the mobile device 605 and provided to the system 600B.

[0130] The image synthesis solution provided by the present disclosure adjusts the position of the virtual camera and ensures that it matches the position of the camera in the real world, thereby achieving proportion correction to overcome the size adaptation problem between the character and the 3D virtual scene, so that the synthesized image presents a more natural and coordinated visual effect. In this way, the characters in the synthesized image will no longer appear to be too large or too small, and the coordination and naturalness of the image are significantly improved.

[0131] The above paragraphs are described using multiple aspects. Obviously, the teachings of this article can be implemented in multiple ways, and any specific architecture or function disclosed in the examples is only a representative case. According to the teachings of this article, it should be understood in the art that each aspect disclosed in this article can be implemented independently, or two or more aspects can be combined to implement.

[0132] Although the present disclosure has been described above with reference to the embodiments, it is not intended to limit the present disclosure. Anyone skilled in the art may make slight changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the invention shall be determined by the following claims.

Claims

1. An image synthesis method, implemented by a computer system, comprising: Obtaining an image of a person photographed by a photographic device; Based on a plurality of virtual scene resources, a 3D virtual scene is generated, and a reference object is set in the 3D virtual scene, wherein the reference object has the same real size as the character; Setting a first virtual camera, so that the first virtual camera faces the reference object in the 3D virtual scene; Determining an ideal position of the first virtual camera based on a first distance between the photographic device and the person when photographing the person; Moving the first virtual camera to the ideal position, and projecting the 3D virtual scene onto a virtual scene layer using the first virtual camera located at the ideal position; Separating a person layer from the person image using an image segmentation model; Overlaying the character layer onto the virtual scene layer to generate a pair of overlay layers; projecting the pair of overlay layers onto a 2D image using a second virtual camera; as well as The 2D image is rendered to obtain a composite image.

2. The image synthesis method as claimed in claim 1, further comprising: Based on the person image, using an object recognition model to calculate a first visual size of the person in the person image; as well as The real size is calculated based on the first visual size, the first distance, and the field of view parameter of the photographic device.

3. The image synthesis method as claimed in claim 2, wherein after determining the ideal position of the first virtual camera and before moving the first virtual camera to the ideal position, the method further comprises: Projecting the 3D virtual scene onto a virtual scene image; Calculating a second visual size of the reference object in the virtual scene image using a ray casting algorithm; and Determine a correction amount that the first virtual camera needs to move relative to the ideal position based on the first visual size, the second visual size and the first distance; The step of moving the first virtual camera to the ideal position further includes: The first virtual camera is moved according to the correction amount.

4. The image synthesis method as claimed in claim 3, wherein determining the correction amount required for the first virtual camera to move relative to the ideal position further comprises: The correction amount required for the first virtual camera to move relative to the ideal position is determined based on the first visual size, the second visual size, the first distance and an offset.

5. The image synthesis method according to any one of claims 1 to 4, further comprising: The first distance is obtained from the photographic device, wherein the photographic device is a depth camera.

6. The image synthesis method according to any one of claims 1 to 4, further comprising: Based on the person image, a depth estimation model is used to estimate the first distance.

7. The image synthesis method as claimed in claim 1, further comprising: Recognize a gesture in the person image using a gesture recognition model; Determine whether the gesture is mapped to a specified operation; as well as If it is determined that the gesture is mapped to the designated operation, the ideal position of the first virtual camera is adjusted according to the designated operation. 8 . The image synthesis method as claimed in claim 1 , wherein the plurality of virtual scene resources include a mesh resource, a texture resource, a shader resource and a material resource.

9. The image synthesis method as claimed in claim 1, wherein the computer system comprises the photographic device. 10 . The image synthesis method as claimed in claim 1 , wherein the computer system is connected to a mobile device, and the mobile device comprises the photographic device.

11. An image synthesis system, comprising a storage device and a processing device, wherein the processing device loads a program from the storage device to perform the following steps: Obtaining an image of a person photographed by a photographic device; Based on a plurality of virtual scene resources, a 3D virtual scene is generated, and a reference object is set in the 3D virtual scene, wherein the reference object has the same real size as the character; Setting a first virtual camera, so that the first virtual camera faces the reference object in the 3D virtual scene; Determining an ideal position of the first virtual camera based on a first distance between the photographic device and the person when photographing the person; Moving the first virtual camera to the ideal position, and projecting the 3D virtual scene onto a virtual scene layer using the first virtual camera located at the ideal position; Separating a person layer from the person image using an image segmentation model; Overlaying the character layer onto the virtual scene layer to generate a pair of overlay layers; projecting the pair of overlay layers onto a 2D image using a second virtual camera; as well as The 2D image is rendered to obtain a composite image.

12. The image synthesis system as claimed in claim 11, wherein the processing device further performs the following steps: Based on the person image, using an object recognition model to calculate a first visual size of the person in the person image; and The real size is calculated based on the first visual size, the first distance, and the field of view parameter of the photographic device.

13. The image synthesis system as claimed in claim 12, wherein the processing device further performs the following steps after determining the ideal position of the first virtual camera and before moving the first virtual camera to the ideal position: Projecting the 3D virtual scene onto a virtual scene image; Calculating a second visual size of the reference object in the virtual scene image using a ray casting algorithm; and Determine a correction amount that the first virtual camera needs to move relative to the ideal position based on the first visual size, the second visual size and the first distance; The processing device further moves the first virtual camera according to the correction amount.

14. The image synthesis system of claim 13, wherein the processing device further performs the following steps to determine the correction amount that the first virtual camera needs to move relative to the ideal position: The correction amount required for the first virtual camera to move relative to the ideal position is determined based on the first visual size, the second visual size, the first distance and an offset. 15 . The image synthesis system as claimed in any one of claims 11 to 14 , wherein the processing device further obtains the first distance from the photographic device, wherein the photographic device is a depth camera. 16 . The image synthesis system as claimed in any one of claims 11 to 14 , wherein the processing device further estimates the first distance using a depth estimation model based on the person image.

17. The image synthesis system as claimed in claim 11, wherein the processing device further performs the following steps: Recognize a gesture in the person image using a gesture recognition model; Determining whether the gesture is mapped to a specified operation; and If it is determined that the gesture is mapped to the designated operation, the ideal position of the first virtual camera is adjusted according to the designated operation. 18 . The image synthesis system as claimed in claim 11 , wherein the plurality of virtual scene resources include a mesh resource, a texture resource, a shader resource, and a material resource.

19. The image synthesis system as claimed in claim 11, wherein the image synthesis system further comprises the photographic device.

20. The image synthesis system as claimed in claim 11, wherein the image synthesis system is connected to a mobile device, and the mobile device includes the photographic device.

Images