Image generation method, related device and computer storage medium

Abstract

The invention provides an image generation method, a related device and a computer storage medium, and the method comprises the steps: generating material region information according to material shooting data and a picture sequence, and projecting the material region information to a 3D virtual character, thereby obtaining a target 3D virtual character; wherein when the target 3D virtual character moves along with the real character, the material area on the target 3D virtual character also moves along with the real character; then, analyzing the target 3D virtual character in the second picture stream in real time to obtain an analysis result; wherein the analysis result comprises the material performance of each region of the target 3D virtual character; and feeding back the material performance of all the areas to a position corresponding to the real figure image in the first picture stream to obtain a target real figure image. Therefore, the target real figure image can feed back the light change and the environment change of the virtual world in real time, so that the image is more real and the fusion degree is higher.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to an image generation method, related devices and computer storage media. Background Art

[0002] Currently, in the virtual shooting process, the image of the real person is presented in the virtual 3D world on a plane. The material used in this plane has a unified single material attribute, which results in the real person image being unable to respond to changes in lighting and environment in the virtual world. This makes the picture very separate from the real person image, and the user experience feels very unreal. Summary of the invention

[0003] In view of this, the present application provides an image generation method, related devices and computer storage media, so that the image of a real person can feedback the lighting changes and environmental changes in the virtual world in real time, making the picture more realistic and more integrated.

[0004] The first aspect of the present application provides an image generation method, comprising:

[0005] Acquire shooting data of at least one multi-eye depth camera; wherein the shooting data includes object depth information and a picture sequence;

[0006] Using the object depth information to perform 3D modeling on real people in the image sequence to obtain a 3D virtual person;

[0007] Generate material area information according to the material shooting data and the picture sequence; wherein the material shooting data is obtained by circling and shooting a real person in the picture sequence with a multi-eye depth camera under calibrated lighting;

[0008] Projecting the material region information onto the 3D virtual character to obtain a target 3D virtual character; wherein when the target 3D virtual character moves with the real person, the material region on the target 3D virtual character also moves along;

[0009] The first virtual camera is used to capture the image of the real person in the virtual scene to obtain a first picture stream, and the second virtual camera is used to capture the target 3D virtual person in the virtual scene to obtain a second picture stream;

[0010] Analyze the target 3D virtual character in the second picture stream in real time to obtain an analysis result; wherein the analysis result includes a material performance of each area of ​​the target 3D virtual character;

[0011] The material expressions of all areas are fed back to the positions corresponding to the real person images in the first picture stream to obtain the target real person images.

[0012] Optionally, generating material area information according to the material shooting data and the picture sequence includes:

[0013] The image sequence is input into an object recognition model, and a boundary map of the object is output; wherein the object recognition model is obtained by training a deep learning model with a training sample set; the training sample set includes at least one training sample; the training sample includes a training sample image sequence and a real boundary map of the object;

[0014] For each region in the boundary map of the object, material recognition is performed on the region according to the material shooting data to obtain a material recognition result of the region;

[0015] The material area information is generated based on the material recognition results of all areas.

[0016] Optionally, for each region in the boundary map of the object, performing material recognition on the region according to the material shooting data to obtain a material recognition result of the region includes:

[0017] For each area in the boundary map of the object, obtaining front and back lighting frame images of the area in the material shooting data; wherein the front and back lighting frame images include non-light frame images and light frame images;

[0018] Light and shadow reflection analysis is performed based on the dark frame image, the light frame image and the set value to obtain the material recognition result of the area.

[0019] Optionally, the real-time analysis of the target 3D virtual character in the second picture stream to obtain the analysis result includes:

[0020] Performing color separation on different colors of the target 3D virtual character in the second picture stream in real time to obtain a color separation result;

[0021] For each color, determine the material attribute of the color according to the corresponding relationship between the color and the material attribute;

[0022] For each region, the material performance of the region is determined according to the material properties of all colors in the region.

[0023] A second aspect of the present application provides an image generation device, comprising:

[0024] A first acquisition unit, configured to acquire photographing data of at least one multi-view depth camera; wherein the photographing data includes object depth information and a picture sequence;

[0025] A 3D modeling unit, used to perform 3D modeling on real people in the picture sequence using the object depth information to obtain a 3D virtual person;

[0026] A first generating unit is used to generate material area information according to material shooting data and the picture sequence; wherein the material shooting data is obtained by circling a real person in the picture sequence with a multi-eye depth camera under a calibrated light;

[0027] A projection unit is used to project the material area information onto the 3D virtual character to obtain a target 3D virtual character; wherein, when the target 3D virtual character moves with the real person, the material area on the target 3D virtual character will also move along;

[0028] A capture unit, used to capture the image of a real person in a virtual scene through a first virtual camera to obtain a first picture stream, and to capture the target 3D virtual person in the virtual scene through a second virtual camera to obtain a second picture stream;

[0029] A material analysis unit, configured to analyze the target 3D virtual character in the second picture stream in real time to obtain an analysis result; wherein the analysis result includes a material performance of each area of ​​the target 3D virtual character;

[0030] The feedback unit is used to feed back the material performance of all areas to the position corresponding to the real person image in the first picture stream to obtain the target real person image.

[0031] Optionally, the first generating unit includes:

[0032] A boundary analysis unit, configured to input the image sequence into an object recognition model and output a boundary map of the object; wherein the object recognition model is obtained by training a deep learning model with a training sample set; the training sample set includes at least one training sample; the training sample includes a training sample image sequence and a real boundary map of the object;

[0033] an identification unit, configured to perform material identification on each region in the boundary map of the object according to the material shooting data, and obtain a material identification result of the region;

[0034] The generation subunit is used to generate material area information according to the material recognition results of all areas.

[0035] Optionally, the identification unit includes:

[0036] A second acquisition unit is used to acquire, for each area in the boundary map of the object, front and back illumination frame images of the area in the material shooting data; wherein the front and back illumination frame images include non-light frame images and light frame images;

[0037] The light and shadow reflection analysis unit is used to perform light and shadow reflection analysis according to the dark frame image, the light frame image and the set value to obtain the material recognition result of the area.

[0038] Optionally, the material analysis unit includes:

[0039] A color separation unit, used for performing color separation on different colors of the target 3D virtual character in the second picture stream in real time to obtain a color separation result;

[0040] A material attribute determination unit, configured to determine the material attribute of each color according to the corresponding relationship between the color and the material attribute;

[0041] The material expression determination unit is used to determine the material expression of each region according to the material properties of all colors in the region.

[0042] A third aspect of the present application provides an electronic device, including:

[0043] one or more processors;

[0044] a storage device having one or more programs stored thereon;

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described in any one of the first aspects.

[0046] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the image generation method as described in any one of the first aspects is implemented.

[0047] As can be seen from the above scheme, the present application provides an image generation method, related device and computer storage medium, which generates material area information according to material shooting data and picture sequence, and then projects the material area information onto a 3D virtual character to obtain a target 3D virtual character; wherein, when the target 3D virtual character moves with the real person, the material area on the target 3D virtual character will also follow the movement; then, the target 3D virtual character in the second screen stream is analyzed in real time to obtain an analysis result; wherein, the analysis result includes the material performance of each area of ​​the target 3D virtual character; the material performance of all areas is fed back to the position corresponding to the real person image in the first screen stream to obtain the target real person image. Thus, the target real person image can feedback the lighting changes and environmental changes of the virtual world in real time, making the image more realistic and more integrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A specific flow chart of an image generation method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the arrangement position and practical application principle of a multi-eye depth camera provided in another embodiment of the present application;

[0051] Figure 3 A schematic diagram of generating a target 3D virtual character provided by another embodiment of the present application;

[0052] Figure 4 A flowchart for generating material area information provided by another embodiment of the present application;

[0053] Figure 5 A flowchart of obtaining a material recognition result of a region provided in another embodiment of the present application;

[0054] Figure 6 A schematic diagram of a material detection model provided in another embodiment of the present application;

[0055] Figure 7 A schematic diagram of generating a target real person image in an actual application scenario provided by another embodiment of the present application;

[0056] Figure 8 A flowchart of a method for analyzing a target 3D virtual character in a second picture stream provided by another embodiment of the present application;

[0057] Fig. 9 A schematic diagram of an image generating device provided by another embodiment of the present application;

[0058] Fig.10 A schematic diagram of an electronic device for implementing an image generating method provided in another embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0061] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0062] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0063] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0064] The present application embodiment provides an image generation method, such as Figure 1 As shown, the specific steps include:

[0065] S101. Acquire shooting data of at least one multi-eye depth camera.

[0066] The shooting data includes object depth information and picture sequences.

[0067] In the specific implementation process of the present application, at least one multi-eye depth camera can be installed above the camera, such as Figure 2 The 00 point is the basic point in front of the camera for synchronization. The priorities of other positions are 01, 02, and 03. Among them, 01 is directly behind the person, and 00, the person, and 01 are on the same line. The positions of 02 and 03 are confirmed by the angle. The angle of 00, 01, and 02 is 45 degrees.

[0068] Specifically, the multi-eye depth camera can be kept mounted directly above the imaging component of the camera (not the camera lens), and the distance between the depth camera and the imaging component of the camera can be measured and calculated to calculate the virtual imaging offset value to adjust the final image.

[0069] S102: Perform 3D modeling on real people in the image sequence using object depth information to obtain 3D virtual people.

[0070] In the specific implementation process of this application, the three-dimensional space coordinates of each point in the real person image in the picture sequence are constructed according to the object depth information, and finally a 3D virtual person is generated, such as Figure 3 T1-3 and T2-3 in.

[0071] S103: Generate material area information according to the material shooting data and the picture sequence.

[0072] Among them, the material shooting data is obtained by shooting the real people in the picture sequence in circles with a multi-eye depth camera under calibrated lighting.

[0073] In the specific implementation process of this application, the calibrated light can be but is not limited to a basic power of 200 watts, an intensity of 50%, a color temperature of 6300K, etc., which is not limited here.

[0074] Optionally, in another embodiment of the present application, an implementation of step S103 is as follows: Figure 4 As shown, including:

[0075] S401: Input the image sequence into the object recognition model, and output the boundary map of the object.

[0076] Among them, the object recognition model is obtained by training the deep learning model with a training sample set; the training sample set includes at least one training sample; the training sample includes a training sample image sequence and a real boundary map of the object.

[0077] Specifically, the object recognition model can be constructed by, but is not limited to, constructing a training sample set; inputting the training sample image sequence into the deep learning model, and outputting a predicted boundary map of the object; using the error between the predicted boundary map of the object and the actual boundary map of the object to adjust the parameters in the machine learning model until the error between the predicted boundary map of the object and the actual boundary map of the object meets a preset convergence condition, and using the deep learning model as the object recognition model.

[0078] The preset convergence condition is preset and changed by a technician or authorized staff and is not limited here.

[0079] It should be noted that, in the specific implementation process of the present application, it is not limited to using the preset convergence conditions to train the model, and the model can also be trained using a preset number of iterations, which is also not limited here.

[0080] In another embodiment of the present application, the learning model can also be used to learn to identify what type of clothing an object is. There is no limitation here, and it can be understood that after there is a large type of clothing, it will be more convenient to subsequently determine the material and boundaries, similar to step-by-step reasoning, such as: jacket, robe, shirt, sweater, etc.

[0081] S402 : For each region in the boundary map of the object, perform material recognition on the region according to the material shooting data to obtain a material recognition result of the region.

[0082] Since the material characteristics of an object may vary in depth, color, refraction and reflection of light, etc., the above characteristics may be identified and analyzed to obtain a material recognition result.

[0083] Optionally, in another embodiment of the present application, an implementation of step S402 is as follows: Figure 5 As shown, including:

[0084] S501 . For each region in the boundary map of the object, obtain front and back lighting frames of the region in the material shooting data.

[0085] The front and rear lighting frame images include dark frame images and light frame images.

[0086] In the specific implementation process of this application, if Figure 2 As shown, the person can be fixed in the range of R0, R1-R2 and waved in a circle once, R1 is greater than 100cm, and R2 is less than 200cm. During this process, the depth camera will shoot and record the object at a shooting speed of 120 times per second, and obtain the front and back lighting frames. The depth camera can record the depth change and light and shadow change of the object, identify the lightless frame image and the light frame image, and use it to analyze the different material properties of the object.

[0087] S502: Perform light and shadow reflection analysis according to the dark frame image, the light frame image and the set value to obtain a material recognition result of the area.

[0088] In the actual application of this application, the material may be but is not limited to metal, leather, cloth, silk, skin, hair, etc., which is not limited here. The setting values ​​include but are not limited to highlight value, reflection value, transmission value, object messiness value, smoothness, skin color value, pearl sphere diameter, dark reflection shadow intensity, etc. The specific values ​​and ranges are set and changed in advance by technicians, experts, etc., which are not limited here.

[0089] Since this is a small-scale object recognition of people's clothing and accessories, not a material recognition of all objects in the world, the setting value can narrow the scope and specify certain fixed materials as guessing objects to improve accuracy. Figure 6 Set the highlight value of the uniform object to n, the reflection value to m, the transmission value to s, the object messiness value to i, the smoothness to p, the skin color value to k, the pearl sphere diameter to r, and the dark reflection shadow intensity to d.

[0090] For example: according to the characteristics of metal objects, the setting value is mainly analyzed for surface smoothness, roughness and reflection characteristics, and the light and shadow reflection analysis is performed based on the front and back lightless frames Z1 and the lighted frames Z100. When the highlight value is greater than n, it is n5, the smoothness is p5, and the reflection value reaches m5, and the analysis of color and sphere diameter attributes is excluded, the object is determined to be metal, otherwise it is determined to be other materials.

[0091] It should be noted that n is just a value range for highlights. A single n cannot determine whether it is a metal. Multiple attributes are needed to confirm it as a metal. It is understandable that n5 is a highlight range currently set, and it is not ruled out that n6 will be added according to subsequent optimization plans.

[0092] Color is excluded in order to clearly define the skin. For this purpose, the color gamut RGB range is set to enhance the recognition of the skin. The analysis of the sphere diameter attribute is to enhance the recognition of some spheres with obvious characteristics such as jewelry, such as pearls, pendants, etc.

[0093] For example, according to the characteristics of leather material: weak reflection, certain projection, average surface smoothness, and texture. Therefore, when analyzing light and shadow reflection based on the front and back light frames, the dark frame Z1 and the light frame Z100, when the highlight value is greater than n, which is n3, the smoothness is p2-p3, the reflection value reaches m3-m4, and the intensity of the dark reflection shadow is d2-d3, and the analysis of the color and sphere diameter attributes is excluded, the object is determined to be leather, otherwise it is determined to be other materials.

[0094] For example: according to the characteristics of cloth material, when analyzing the light reflection of the dark frame Z1 and the light frame Z100 according to the front and back lighting frames, when the highlight value is greater than n, which is n0, the smoothness is p0, the reflection value reaches m0, the shadow intensity of the dark reflection is d4, and the object disorder value is i2-i3, excluding the analysis of the color and spherical shape diameter attributes, the object is determined to be cloth, otherwise it is determined to be other materials.

[0095] For example, according to the characteristics of silk material, when analyzing the light reflection of the dark frame Z1 and the light frame Z100 based on the front and back lighting frames, when the highlight value is greater than n, which is n3, the smoothness is p5, the reflection value reaches m3, the shadow intensity of the dark reflection is d2-d3, and the object disorder value is i0, excluding the analysis of the color and spherical shape diameter attributes, the object is determined to be silk, otherwise it is determined to be other materials.

[0096] After excluding the clothing area, we further judge whether the material of the object is skin. According to the characteristics of skin material, when the light reflection analysis is performed based on the front and back light frames, the lightless frame Z1 and the lighted frame Z100, when the highlight value is greater than n, which is n2, the smoothness is p2, the reflection value reaches m2, the dark reflection shadow intensity is d1-d2, the object disorder value is i0, the color k is within the range, and there is no analysis of the spherical shape and diameter attributes. The object is judged to be skin, otherwise it is judged to be other materials.

[0097] After excluding clothing and skin areas, we further judge whether the material of the object is hair. When we analyze the light and shadow reflection of the dark frame Z1 and the light frame Z100 based on the front and back lighting frames, when the highlight value is greater than n, which is n4, the smoothness is p4, the reflection value reaches m4, the shadow intensity of the dark reflection is d3-d4, the object's messy characteristic value is i6, and there is no analysis of the spherical shape and diameter attribute, the object is judged to be hair, otherwise it is judged to be other materials.

[0098] S403: Generate material region information according to the material recognition results of all regions.

[0099] Continuing with the above example, according to the material recognition results of all regions, the complete material region information is generated according to the material and the preset color corresponding to the material (different material regions of the object are assigned different colors, that is, the correspondence between the preset colors and material properties. It can be understood that the color corresponding to the material is not necessarily consistent with the color of the area where the original image is located), such as Figure 3 T1-2, T2-2.

[0100] S104: Project the material region information onto the 3D virtual character to obtain a target 3D virtual character.

[0101] When the target 3D virtual character moves with the real person, the material area on the target 3D virtual character will also move along.

[0102] Since the angle of the selected image sequence is consistent with the angle of the 3D virtual character, Figure 3 With T1-1 and T2-1 as the main framework, T1-2 and T2-2 can be respectively projected onto T1-3 and T2-3 to complete the texture production, thus obtaining the target 3D virtual character T3 with material ID texture.

[0103] S105 . Capture the image of the real person in the virtual scene by using the first virtual camera to obtain a first picture stream, and capture the target 3D virtual person in the virtual scene by using the second virtual camera to obtain a second picture stream.

[0104] First, two virtual cameras are created in the virtual scene. Figure 7 The first virtual camera c0 is responsible for capturing the real person and background images and for the final output of the images. Figure 2 The position simulation of the real camera 00 in . Figure 7 The second virtual camera c1 is mounted under c0, keeps all properties of c0 consistent, and the distance is zero. It only changes the captured image content, and the captured content only captures the 3D virtual character. T5 is the real person image, and T5-1 is the 3D virtual character imaging following the motion image.

[0105] By setting the distance and screen position between the 3D virtual character and c1 to be equal to the distance and screen position between the real character's screen plane and c0, when the real character moves, the 3D virtual character will follow the movement, and the material area in the screen will always be integrated with the different materials of the real character.

[0106] S106: Analyze the target 3D virtual character in the second picture stream in real time to obtain an analysis result.

[0107] The analysis results include the material performance of each area of ​​the target 3D virtual character.

[0108] It is understandable that since the lighting, weather, and environment in the virtual scene will change at any time, different brightness will be generated, and the material performance of the target 3D virtual character will also change accordingly. Therefore, it is necessary to analyze the material performance of each area of ​​the 3D virtual character in real time, so as to feed back the material performance of all areas to the position corresponding to the real character image in the first picture stream, and obtain the target real character image, so that the overall picture is more realistic and more integrated.

[0109] Optionally, in another embodiment of the present application, an implementation of step S106 is as follows: Figure 8 As shown, including:

[0110] S801. Perform color separation on different colors of the target 3D virtual character in the second picture stream in real time to obtain a color separation result.

[0111] S802: For each color, determine the material attribute of the color according to the corresponding relationship between the color and the material attribute.

[0112] S803: For each region, determine the material performance of the region according to the material properties of all colors in the region.

[0113] In the specific implementation process of this application, for each color in the color separation result, the material attribute of the color is determined according to the corresponding relationship between the color and the material attribute, the overall material attribute is superimposed and connected, and the overall area range of the picture is set to , different color areas correspond to , , And so on, the overall material properties of the set screen are , different material properties are , , Set the material to Physically-Based Rendering (PBR), and the material of each region is pbr1, pbr2, pbr3, and so on.

[0114] Color acquisition: In the early stage, color presets were made for different materials (the corresponding relationship between color and material properties). When the color RGB value of the current area meets the specified material, the color will be preset and matched. After extracting the current color range, fill it with the value 1, and other areas are filled with black and the value 0.

[0115] Specifically, set p to be any pixel in the region, and P to be the set of all pixels in the region. Then PBR can be achieved by The cumulative calculation is:

[0116] .

[0117] S107 , feeding back the material expressions of all regions to the positions corresponding to the real person images in the first picture stream, to obtain the target real person images.

[0118] As can be seen from the above scheme, the present application provides an image generation method, which generates material area information according to material shooting data and picture sequence, and then projects the material area information onto a 3D virtual character to obtain a target 3D virtual character; wherein, when the target 3D virtual character moves with the real person, the material area on the target 3D virtual character will also follow the movement; then, the target 3D virtual character in the second screen stream is analyzed in real time to obtain an analysis result; wherein, the analysis result includes the material performance of each area of ​​the target 3D virtual character; the material performance of all areas is fed back to the position corresponding to the real person image in the first screen stream to obtain the target real person image. Thus, the target real person image can feedback the lighting changes and environmental changes of the virtual world in real time, making the image more realistic and more integrated.

[0119] The present application embodiment provides an image generation device, such as Fig. 9 As shown, specifically including:

[0120] The first acquisition unit 901 is used to acquire shooting data of at least one multi-view depth camera.

[0121] The shooting data includes object depth information and picture sequences.

[0122] The 3D modeling unit 902 is used to perform 3D modeling on real people in the picture sequence by using object depth information to obtain a 3D virtual person.

[0123] The first generating unit 903 is used to generate material area information according to the material shooting data and the picture sequence.

[0124] Among them, the material shooting data is obtained by shooting the real people in the picture sequence in circles with a multi-eye depth camera under calibrated lighting.

[0125] Optionally, in another embodiment of the present application, an implementation of the first generating unit 903 includes:

[0126] The boundary analysis unit is used to input the image sequence into the object recognition model and output the boundary map of the object.

[0127] Among them, the object recognition model is obtained by training the deep learning model with a training sample set; the training sample set includes at least one training sample; the training sample includes a training sample image sequence and a real boundary map of the object.

[0128] The recognition unit is used to perform material recognition on each region in the boundary map of the object according to the material shooting data to obtain a material recognition result of the region.

[0129] The generation subunit is used to generate material area information according to the material recognition results of all areas.

[0130] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 4 As shown, no further description is given here.

[0131] Optionally, in another embodiment of the present application, an implementation of the identification unit includes:

[0132] The second acquisition unit is used to acquire, for each area in the boundary map of the object, front and back lighting frames of the area in the material shooting data.

[0133] The front and rear lighting frame images include dark frame images and light frame images.

[0134] The light and shadow reflection analysis unit is used to perform light and shadow reflection analysis based on the dark frame image, the light frame image and the set value to obtain the material recognition result of the area.

[0135] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 5 As shown, no further description is given here.

[0136] The projection unit 904 is used to project the material region information onto the 3D virtual character to obtain a target 3D virtual character.

[0137] When the target 3D virtual character moves with the real person, the material area on the target 3D virtual character will also move along.

[0138] The capturing unit 905 is used to capture the image of the real person in the virtual scene through the first virtual camera to obtain a first picture stream, and to capture the target 3D virtual person in the virtual scene through the second virtual camera to obtain a second picture stream.

[0139] The material analysis unit 906 is used to analyze the target 3D virtual character in the second picture stream in real time to obtain an analysis result.

[0140] The analysis results include the material performance of each area of ​​the target 3D virtual character.

[0141] Optionally, in another embodiment of the present application, an implementation of the material analysis unit 906 includes:

[0142] The color separation unit is used to perform color separation on different colors of the target 3D virtual character in the second picture stream in real time to obtain a color separation result.

[0143] The material attribute determination unit is used to determine the material attribute of each color according to the corresponding relationship between the color and the material attribute.

[0144] The material expression determination unit is used to determine the material expression of each region according to the material properties of all colors in the region.

[0145] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 8 As shown, no further description is given here.

[0146] The feedback unit 907 is used to feed back the material expressions of all regions to the position corresponding to the real person image in the first picture stream to obtain the target real person image.

[0147] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0148] As can be seen from the above scheme, the present application provides an image generation device, wherein the first acquisition unit 901 acquires the shooting data of at least one multi-eye depth camera, the 3D modeling unit 902 uses the object depth information to perform 3D modeling on the real person in the picture sequence to obtain a 3D virtual person, and then the first generation unit 903 generates material area information according to the material shooting data and the picture sequence. The projection unit 904 projects the material area information onto the 3D virtual person to obtain the target 3D virtual person; wherein, when the target 3D virtual person moves with the real person, the material area on the target 3D virtual person will also move with the movement; the capture unit 905 captures the real person image in the virtual scene through the first virtual camera to obtain the first picture stream, and captures the target 3D virtual person in the virtual scene through the second virtual camera to obtain the second picture stream, and then the material analysis unit 906 analyzes the target 3D virtual person in the second picture stream in real time to obtain the analysis result; wherein, the analysis result includes the material performance of each area of ​​the target 3D virtual person; the feedback unit 907 feeds back the material performance of all areas to the position corresponding to the real person image in the first picture stream to obtain the target real person image. This allows the target real person image to reflect the lighting and environmental changes in the virtual world in real time, making the picture more realistic and integrated.

[0149] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] Another embodiment of the present application provides an electronic device, such as Fig.10 As shown, including:

[0151] One or more processors 1001.

[0152] The storage device 1002 stores one or more programs.

[0153] When the one or more programs are executed by the one or more processors 1001, the one or more processors 1001 implement the image generation method as described in any one of the above embodiments.

[0154] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the image generation method as described in any one of the above embodiments is implemented.

[0155] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0157] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0158] Another embodiment of the present application provides a computer program product. When the computer program product is executed, it is used to execute the above-mentioned image generation method.

[0159] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0160] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0161] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0162] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. An image generation method, characterized in that: include: Acquire shooting data of at least one multi-eye depth camera; wherein the shooting data includes object depth information and a picture sequence; Using the object depth information to perform 3D modeling on real people in the image sequence to obtain a 3D virtual person; Generate material area information according to the material shooting data and the picture sequence; wherein the material shooting data is obtained by circling and shooting a real person in the picture sequence with a multi-eye depth camera under calibrated lighting; Projecting the material region information onto the 3D virtual character to obtain a target 3D virtual character; wherein when the target 3D virtual character moves with the real person, the material region on the target 3D virtual character also moves along; The first virtual camera is used to capture the image of the real person in the virtual scene to obtain a first picture stream, and the second virtual camera is used to capture the target 3D virtual person in the virtual scene to obtain a second picture stream; Analyze the target 3D virtual character in the second picture stream in real time to obtain an analysis result; wherein the analysis result includes a material performance of each area of ​​the target 3D virtual character; The material expressions of all areas are fed back to the positions corresponding to the real person images in the first picture stream to obtain the target real person images.

2. The image generation method according to claim 1, characterized in that: The step of generating material region information according to the material shooting data and the picture sequence includes: Inputting the image sequence into an object recognition model, and outputting a boundary map of the object; For each region in the boundary map of the object, material recognition is performed on the region according to the material shooting data to obtain a material recognition result of the region; The material area information is generated based on the material recognition results of all areas.

3. The image generation method according to claim 2, characterized in that: For each region in the boundary map of the object, performing material recognition on the region according to the material shooting data to obtain a material recognition result of the region includes: For each area in the boundary map of the object, obtaining front and back lighting frame images of the area in the material shooting data; wherein the front and back lighting frame images include non-light frame images and light frame images; Light and shadow reflection analysis is performed based on the dark frame image, the light frame image and the set value to obtain the material recognition result of the area.

4. The image generation method according to claim 1, characterized in that: The real-time analysis of the target 3D virtual character in the second picture stream to obtain the analysis result includes: Performing color separation on different colors of the target 3D virtual character in the second picture stream in real time to obtain a color separation result; For each color, determine the material attribute of the color according to the corresponding relationship between the color and the material attribute; For each region, the material performance of the region is determined according to the material properties of all colors in the region.

5. An image generating device, characterized in that: include: A first acquisition unit, configured to acquire photographing data of at least one multi-view depth camera; wherein the photographing data includes object depth information and a picture sequence; A 3D modeling unit, used to perform 3D modeling on real people in the picture sequence using the object depth information to obtain a 3D virtual person; A first generating unit is used to generate material area information according to material shooting data and the picture sequence; wherein the material shooting data is obtained by circling a real person in the picture sequence with a multi-eye depth camera under a calibrated light; A projection unit is used to project the material area information onto the 3D virtual character to obtain a target 3D virtual character; wherein, when the target 3D virtual character moves with the real person, the material area on the target 3D virtual character will also move along; A capture unit, used to capture the image of a real person in a virtual scene through a first virtual camera to obtain a first picture stream, and to capture the target 3D virtual person in the virtual scene through a second virtual camera to obtain a second picture stream; A material analysis unit, configured to analyze the target 3D virtual character in the second picture stream in real time to obtain an analysis result; wherein the analysis result includes a material performance of each area of ​​the target 3D virtual character; The feedback unit is used to feed back the material performance of all areas to the position corresponding to the real person image in the first picture stream to obtain the target real person image.

6. The image generation device according to claim 5, characterized in that: The first generating unit comprises: A boundary analysis unit, used to input the image sequence into an object recognition model and output a boundary map of the object; an identification unit, configured to perform material identification on each region in the boundary map of the object according to the material shooting data, and obtain a material identification result of the region; The generation subunit is used to generate material area information according to the material recognition results of all areas.

7. The image generation device according to claim 6, characterized in that: The identification unit comprises: A second acquisition unit is used to acquire, for each area in the boundary map of the object, front and back illumination frame images of the area in the material shooting data; wherein the front and back illumination frame images include non-light frame images and light frame images; The light and shadow reflection analysis unit is used to perform light and shadow reflection analysis according to the dark frame image, the light frame image and the set value to obtain the material recognition result of the area.

8. The image generation device according to claim 5, characterized in that: The material analysis unit comprises: A color separation unit, used for performing color separation on different colors of the target 3D virtual character in the second picture stream in real time to obtain a color separation result; A material attribute determination unit, configured to determine the material attribute of each color according to the corresponding relationship between the color and the material attribute; The material expression determination unit is used to determine the material expression of each region according to the material properties of all colors in the region.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method according to any one of claims 1 to 4.

10. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the image generation method as described in any one of claims 1 to 4 is implemented.

Images