Image processing method, electronic equipment and storage medium

Abstract

The invention provides an image processing method, electronic equipment and a storage medium. The method comprises the following steps: determining a ground area of a virtual scene graph; acquiring a first camera parameter corresponding to the target position under a first visual angle; determining a first figure graph from a preset database based on the first camera parameter; according to a second camera parameter corresponding to the first figure graph, first pixel distribution of the first figure graph and a third camera parameter corresponding to the virtual scene graph under a third visual angle, a transformation matrix is transformed from the second visual angle to the third visual angle, and a second figure graph with second pixel distribution is obtained; based on the depth information of the second figure graph, adding the second figure graph to the target position to obtain a first composite graph; and obtaining a second composite image based on the first composite image and the image generation model. The method can improve the synthesis effect of the synthesized picture.

Description

Technical Field

[0001] The present application relates to the field of images, and in particular to an image processing method, an electronic device and a storage medium. Background Art

[0002] With the development of image processing technology, image synthesis technology can effectively process image information and synthesize at least two images. For example, a person in an image is added to an existing virtual scene image. At present, a common practice is to paste the mask image corresponding to the person in an image into the existing virtual scene image. However, this method cannot guarantee the normal perspective relationship of the person, and the synthesis effect of the pasted person and the virtual scene image is poor. Summary of the invention

[0003] The embodiments of the present application disclose an image processing method, an electronic device and a storage medium, which solve the technical problem of poor synthesis effect of synthetic pictures.

[0004] The present application provides an image processing method, the method comprising: determining a ground area of ​​a virtual scene graph, the ground area comprising a target position; obtaining a first camera parameter corresponding to the target position under a first viewing angle; determining a first character image from a preset database based on the first camera parameter, the first character image being generated by processing a target rendering image based on a preset image generation model, the target rendering image being a two-dimensional image obtained by converting a three-dimensional character model under a second viewing angle; obtaining a second character image with a second pixel distribution according to a second camera parameter corresponding to the first character image, a first pixel distribution of the first character image, a third camera parameter corresponding to the virtual scene graph under a third viewing angle, and a transformation matrix from the second viewing angle to the third viewing angle; adding the second character image to the target position based on depth information of the second character image to obtain a first composite image; and obtaining a second composite image based on the first composite image and the image generation model.

[0005] In some embodiments of the present application, obtaining the second composite image based on the first composite image and the image generation model includes: based on a first control condition, processing the composite rendering image through the image generation model to obtain the second composite image, the first control condition including a composite depth map, a composite segmentation map and a composite key point map corresponding to the first composite image.

[0006] In some embodiments of the present application, based on the depth information of the second character image, the second character image is added to the target position to obtain a first composite image, including: determining an occlusion area of ​​the second character image, the occlusion area being an overlapping area between the second character image and the third character image in the virtual scene image; based on the overlapping area, if the depth information of the second character image is greater than the depth information of the third character image, and the layer of the second character image is located above the layer of the third character image, moving the layer of the second character image downward so that the layer of the second character image in the first composite image is located below the layer of the third character image; or, if the depth information of the second character image is less than the depth information of the third character image, and the layer of the second character image is located below the layer of the third character image, moving the layer of the second character image upward so that the layer of the second character image in the first composite image is located above the layer of the third character image.

[0007] In some embodiments of the present application, determining the ground area of ​​the virtual scene graph includes: determining the ground area according to scene information corresponding to the virtual scene graph.

[0008] In some embodiments of the present application, the scene information includes ground parameters, and determining the ground area according to the scene information in the virtual scene graph includes: determining the ground area based on the virtual scene graph carrying the ground parameters.

[0009] In some embodiments of the present application, the scene information includes a depth map, and determining the ground area according to the scene information in the virtual scene map includes: calculating point cloud data based on the depth map corresponding to the virtual scene map; and determining the ground area according to the point cloud data.

[0010] In some embodiments of the present application, determining the ground area of ​​the virtual scene graph includes: calculating camera extrinsic parameters based on the vanishing points in the virtual scene graph, the camera extrinsic parameters including the rotation matrix and translation vector of the camera that captures the virtual scene graph relative to the ground area; determining a ground equation based on the camera extrinsic parameters and preset ground features; calculating ground parameters based on the ground equation; and determining the ground area based on the ground parameters.

[0011] In some embodiments of the present application, the method also includes constructing the database, including: obtaining a model rendering image, a model depth map, a model segmentation map, and a model key point map of a preset three-dimensional model at multiple preset perspectives, and recording the camera parameters corresponding to each preset perspective, and the camera position corresponding to each preset perspective; based on a second control condition and a preset style prompt word, processing the model rendering image through the image generation model to obtain model character images of multiple styles, the second control condition including the model depth map, the model segmentation map, and the model key point map; constructing the database according to the model rendering image, the model depth map, the depth information corresponding to the model depth map, the model segmentation map, the model key point map, the model character image, the camera parameters and the camera position.

[0012] The present application also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the image processing method when executing a computer program stored in the memory.

[0013] The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the image processing method is implemented.

[0014] In the image processing method provided in the present application, a ground area of ​​a virtual scene graph is determined, the ground area includes a target position, a first camera parameter corresponding to the target position under a first perspective is obtained, a first character image is determined from a preset database using the first camera parameter, and a first character image having a similar visual angle to the target position is found, wherein the first character image is generated after being processed by an image generation model, which can improve the subsequent fusion effect with the virtual scene graph.

[0015] The first character image under the second perspective is converted into the second character image under the third perspective by using the second camera parameters corresponding to the first character image, the first pixel distribution of the first character image, the third camera parameters corresponding to the virtual scene image under the third perspective, and the transformation matrix from the second perspective to the third perspective, so that the second character image and the virtual scene image have the same perspective. Then, based on the depth information of the second character image, the second character image is added to the target position to obtain the first composite image, which can ensure the rationality of the perspective relationship between the second character image and the virtual scene image or other character images in the virtual scene image. Based on the first composite image and the image generation model, the second composite image is obtained, and the image generation model can be used to repair the possible defects in the first composite image, which can improve the fusion efficiency and ensure the fusion effect to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0017] Figure 2 It is a flowchart of the image processing method provided in an embodiment of the present application.

[0018] Figure 3 It is a virtual scene graph provided in an embodiment of the present application.

[0019] Figure 4 This is a fusion effect diagram provided in an embodiment of the present application.

[0020] Figure 5 This is a flow chart for determining the ground area provided in an embodiment of the present application.

[0021] Figure 6 It is a flowchart of building a database provided in an embodiment of the present application.

[0022] Figure 7 It is a schematic diagram of the perspective conversion provided in the embodiment of the present application.

[0023] Figure 8 It is a schematic diagram of the virtual camera provided in the embodiment of the present application within the imaging range.

[0024] Fig. 9 It is a schematic diagram of the imaging corresponding to the three-dimensional model provided in the embodiment of the present application. DETAILED DESCRIPTION

[0025] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0026] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0027] With the development of image processing technology, image synthesis technology can effectively process image information and synthesize at least two images. For example, a person in an image is added to an existing virtual scene image. At present, a common practice is to paste the mask image corresponding to the person in an image into the existing virtual scene image. However, this method cannot guarantee the normal perspective relationship of the person, and the synthesis effect of the pasted person and the virtual scene image is poor.

[0028] In order to solve the technical problem of poor synthesis effect of synthetic pictures, the embodiments of the present application provide an image processing method, an electronic device and a storage medium, which obtains a first character image by processing a target rendering image, and then converts the first character image with a first pixel distribution into a second character image with a second pixel distribution, so that the second character image and the virtual scene image have the same third perspective. In addition, based on the depth information of the second character image, the second character image is added to the virtual scene image to obtain a first synthetic image, and the rationality of the perspective relationship of the characters in the first synthetic image can be ensured through the depth information. Finally, the second synthetic image obtained based on the first synthetic image and the image generation model can effectively improve the fusion effect of the second character image and the virtual scene image, and repair the problem of missing characters that may appear in the first synthetic image. The structure of the electronic device to which the image processing method of the present application is applied is first described below.

[0029] Figure 1 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application is applied to an electronic device 10. The electronic device 10 may be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, or other electronic device with a display screen. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0030] like Figure 1 As shown, the electronic device 10 may include a display screen 100, a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the display screen 100, the communication interface 101, the memory 102, and the I / O interface 104 through the bus 105.

[0031] The display screen 100 may be a touch screen, which is an inductive touch-sensitive liquid crystal display device. Alternatively, the display screen 100 may also be a non-touch screen. The display screen 100 may be used to display a virtual scene graph.

[0032] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB), Controller Area Network (CAN), etc. The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), etc.

[0033] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0034] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0035] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0036] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 103, the image processing method executed on the electronic device 10 can be implemented.

[0037] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0038] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0039] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned image processing method.

[0040] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0041] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0042] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0043] Figure 2 is a flowchart of an image processing method provided in an embodiment of the present application, which is applied to an electronic device (such as Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0044] Step S201, determining the ground area of ​​the virtual scene graph.

[0045] In some embodiments of the present application, the electronic device includes a virtual camera, and a similar shooting effect as that of a real camera is simulated by using the virtual camera, thereby creating a realistic virtual scene graph in a virtual environment. The virtual scene graph is a three-dimensional space virtual scene graph or a two-dimensional virtual scene graph corresponding to a digital scene outlined by a computer through digital communication technology. The virtual scene graph may include virtual buildings, virtual ground, virtual scenery, virtual characters, etc.

[0046] In some embodiments of the present application, a virtual scene graph that does not include a virtual character can be constructed so that an image corresponding to the virtual character can be added to a target position in the ground area of ​​the virtual scene graph later. The ground area can be the area in the virtual scene graph where the image corresponding to the virtual character is added, and the target position can be the position of the image corresponding to the virtual character in the ground area. In one example, Figure 3 As shown, a virtual scene graph is obtained, a ground area is extracted from the virtual scene graph, and a target position where an image corresponding to a virtual character can be added is rendered in the ground area.

[0047] In addition, if there are already virtual characters in the virtual scene graph, the virtual characters in the virtual scene graph can be removed through image processing tools, so that the virtual scene graph does not contain images corresponding to virtual characters that do not belong to the preset database, and the virtual scene graph is used as the background image for subsequent additions of images corresponding to virtual characters.

[0048] In some embodiments of the present application, in order to add a virtual character, it is necessary to first determine the ground area in the virtual scene graph. Since different virtual scene graphs come from different sources, the virtual scene graph used may include corresponding scene information, or may not include scene information. Therefore, when using a virtual scene graph, different methods need to be used to determine the ground area. Specifically, if the virtual scene graph carries scene information, the ground area can be determined by the scene information, wherein the scene information may include ground parameters, a depth map corresponding to the virtual scene graph, and binocular information; if the virtual scene graph does not carry scene information, the ground area can be determined by the vanishing point in the virtual scene graph and preset ground features. The above are only examples, and other methods can also be used to determine the ground area, which is not limited by the present application. In addition, the specific implementation methods of the above examples can be referred to as follows: Figure 5 The embodiment shown.

[0049] Step S202, obtaining first camera parameters corresponding to the target position under the first viewing angle.

[0050] In some embodiments of the present application, the virtual camera shoots the virtual scene image at a normal viewing angle. In order to add an image corresponding to a virtual character at a target position, the virtual camera can be rotated from the normal viewing angle to a first viewing angle, where the first viewing angle is the normal viewing angle when the virtual camera shoots the target position. The first camera parameters of the virtual camera at the first viewing angle are obtained, and the first camera parameters include camera intrinsic parameters and camera extrinsic parameters.

[0051] Step S203: determining a first person image from a preset database based on the first camera parameter.

[0052] In some embodiments of the present application, the image corresponding to the virtual character to be added is recorded as the first character image. The first character image is stored in a preset database. In the preset database, the target rendering image is processed based on a preset image generation model to generate the first character image, and the target rendering image is a two-dimensional image obtained by converting a three-dimensional character model (such as a virtual character) from a second perspective. Among them, the image generation model can be a stable diffusion model (StableDiffusion, SD). In addition, the construction of the preset database can refer to the following Figure 6 The embodiment shown.

[0053] In some embodiments of the present application, in order to increase the rationality of the first person image added at the target position, the neighboring camera parameters closest to the first camera parameters are searched in a preset database, so that the image corresponding to the neighboring camera parameters is used as the first person image.

[0054] Step S204, obtaining a second character image with a second pixel distribution according to the second camera parameters corresponding to the first character image, the first pixel distribution of the first character image, the third camera parameters corresponding to the virtual scene image under the third viewing angle, and the transformation matrix from the second viewing angle to the third viewing angle.

[0055] In some embodiments of the present application, the second camera parameter is a camera parameter when the virtual camera shoots the first character image at a normal viewing angle. The third viewing angle is a normal viewing angle when the virtual camera shoots the virtual scene, and the third camera parameter is a camera parameter when the virtual camera shoots the virtual scene at the third viewing angle.

[0056] In some embodiments of the present application, after determining the first person image, the virtual camera can be rotated from the first perspective to the third perspective, so that the virtual camera shoots the virtual scene image at a normal perspective. In addition, the virtual camera takes the third perspective as the normal perspective, and the first person image takes the second perspective as the normal perspective, and the first person image can be converted into the second person image at the third perspective. The conversion of perspective will change the distribution of pixels, which can be expressed by the following formula: ; In the formula, represents a second pixel distribution of a second character image, represents a first pixel distribution of a first character image, represents the third camera parameter, represents the second camera parameters, and They respectively represent the rotation and translation in the transformation matrix from the second perspective to the third perspective.

[0057] Step S205 , based on the depth information of the second character image, add the second character image at the target position to obtain a first composite image.

[0058] In some embodiments of the present application, after adding the second character image at the target position, an initial composite image may be generated, and a third character image may exist in the initial composite image. If the third character image is added to the virtual scene image before the second character image, and the target position is located next to the position where the third character image is added, there may be a situation where the second character image and the third character image have an occluded area.

[0059] In some embodiments of the present application, if it is determined that there is an occluded area, the depth information of the second character image is compared with the depth information of the third character image. Usually, the layer with smaller depth information is located above the layer with larger depth information. If the depth information of the second character image is greater than the depth information of the third character image, and the layer of the second character image is located above the layer of the third character image, it means that the layer positions corresponding to the second character image and the third character image need to be changed. Move the layer of the second character image downward so that the layer of the second character image is located below the layer of the third character image, and the initial composite image of the changed layer is used as the first composite image.

[0060] If the depth information of the second character image is less than the depth information of the third character image, and the layer of the second character image is located below the layer of the third character image, it means that the layer positions corresponding to the second character image and the third character image need to be changed. The layer of the second character image is moved upward so that the layer of the second character image is located above the layer of the third character image, and the initial composite image of the changed layer is used as the first composite image.

[0061] If the depth information of the second character image is greater than the depth information of the third character image, and the layer of the second character image is located below the layer of the third character image, it means that the occlusion relationship of the occlusion area is reasonable, and there is no need to change the layers of the second character image and the third character image. Alternatively, if the depth information of the second character image is less than the depth information of the third character image, and the layer of the second character image is located above the layer of the third character image, it means that the occlusion relationship of the occlusion area is reasonable, and there is no need to change the layers of the second character image and the third character image. The initial composite image that does not require changing the layer is used as the first composite image.

[0062] Step S206: obtaining a second composite image based on the first composite image and the image generation model.

[0063] In some embodiments of the present application, the first synthetic image is rendered to obtain a synthetic rendering image, a synthetic depth map, a synthetic segmentation map, and a synthetic key point map, wherein rendering refers to converting two-dimensional or three-dimensional graphic data (e.g., lines, shapes, colors, etc.) into an image presented on a display screen of an electronic device. In the rendering process, attention is paid to the processing of the color, brightness, contrast, material, and texture of the image. The synthetic rendering image is a color image generated after adjusting the geometric data, texture data, and illumination data of the first synthetic image. The synthetic depth map is a gray image containing the depth information of each pixel in the first synthetic image. The synthetic segmentation map is a contour map obtained after segmenting the first synthetic image. The synthetic key point map is an image that marks the key points in the first synthetic image.

[0064] In some embodiments of the present application, there may be partial missing characters in the first composite image. In order to avoid such situations affecting the fusion effect, the first composite image is processed using an image generation model. The composite rendering image, the composite depth map, the composite segmentation map, and the composite key point map are input into the image generation model, and the composite depth map, the composite segmentation map, and the composite key point map are used as the first control condition of the image generation model. Based on the control conditions, the image generation model processes the composite rendering image to obtain a second composite image. The image generation model is used to repair any incomplete human body that may appear, and the edges of the characters in the second composite image processed by the image generation model can be more naturally integrated with the virtual scene map. The following is combined with Figure 4 The fusion effect of the above-mentioned embodiment is described.

[0065] like Figure 4 Figure (1) shows the fusion effect of related technologies. Figure 4 Figure (2) is a fusion effect diagram of an embodiment of the present application. Figure 4 Figure (1) directly adds the rendered image to the virtual scene image, resulting in an unnatural perspective effect of the character. The first character image of the present application is an image generated after being processed by an image generation model, which can increase the character details and realism. On this basis, the first character image is processed and added to the virtual scene image, and the second composite image is obtained through the image generation model, which increases the fusion effect of the character edge and the virtual scene image. Compared with Figure 4 Figure (1), such as Figure 4 The fusion effect of (2) is better, and the edge of the character is better.

[0066] In the above embodiment, a ground area of ​​the virtual scene graph is determined, the ground area includes a target position, a first camera parameter corresponding to the target position under a first viewing angle is obtained, a first character image is determined from a preset database using the first camera parameter, and a first character image having a similar visual angle to the target position is found, wherein the first character image is generated after being processed by an image generation model, which can improve the subsequent fusion effect with the virtual scene graph.

[0067] The first character image under the second perspective is converted into the second character image under the third perspective by using the second camera parameters corresponding to the first character image, the first pixel distribution of the first character image, the third camera parameters corresponding to the virtual scene image under the third perspective, and the transformation matrix from the second perspective to the third perspective, so that the second character image and the virtual scene image have the same perspective. Then, based on the depth information of the second character image, the second character image is added to the target position to obtain the first composite image, which can ensure the rationality of the perspective relationship between the second character image and the virtual scene image or other character images in the virtual scene image. Based on the first composite image and the image generation model, the second composite image is obtained, and the image generation model can be used to repair the possible defects in the first composite image, which can improve the fusion efficiency and ensure the fusion effect to a certain extent.

[0068] Figure 5 is a flow chart for determining the ground area provided in the embodiment of the present application. Figure 5 As shown, the method of determining the ground area may include the following steps. According to different requirements, the order of the steps in the flow chart may be changed, and some steps may be omitted.

[0069] Step S501, determining whether there is scene information corresponding to the virtual scene graph.

[0070] In some embodiments of the present application, the scene information includes any label, data or illustration that can describe the virtual scene graph. Since different virtual scene graphs come from different sources, the virtual scene graph used may include corresponding scene information or may not include scene information. Therefore, when using a virtual scene graph, it is possible to first determine whether the virtual scene graph carries scene information, and then determine the ground area based on the scene information, or determine the ground area based on a method that does not carry scene information.

[0071] If there is scene information corresponding to the virtual scene graph, execute step S502. If there is no scene information corresponding to the virtual scene graph, jump to execute step S508.

[0072] Step S502: determine whether the scene information includes ground parameters.

[0073] In some embodiments of the present application, ground parameters are used to describe ground characteristics and positions, and may include height information, texture features, geometric shapes, and coordinate positions of the ground. The ground parameters may be obtained by a preset ground segmentation algorithm. If the scene information includes ground parameters, step S503 is executed. If the scene information does not include ground parameters, step S504 is executed.

[0074] Step S503: determining the ground area based on the virtual scene graph carrying the ground parameters.

[0075] In some embodiments of the present application, the ground area can be marked in the virtual scene graph by using ground parameters.

[0076] Step S504: determine whether the scene information includes a depth map.

[0077] In some embodiments of the present application, the depth map is a gray image containing depth information of each pixel in the virtual scene image. Since the depth map provides depth information of each pixel, the ground area can be effectively determined through the depth map.

[0078] If the scene information does not include a depth map, execute step S506. If the scene information includes a depth map, execute step S505.

[0079] Step S505, calculating point cloud data based on the depth map corresponding to the virtual scene map, so as to determine the ground area according to the point cloud data.

[0080] In some embodiments of the present application, in order to obtain three-dimensional point cloud data based on a two-dimensional depth map, it is necessary to obtain the camera intrinsic parameters and camera extrinsic parameters of the virtual camera that shoots the virtual scene map. Among them, the camera intrinsic parameters and camera extrinsic parameters can be obtained from the scene information. If the scene information does not contain camera intrinsic parameters, the Transformers model can be used to learn the image features in the virtual scene map, and the geometric transformation of the image can be realized through the spatial transformation network, so as to realize automatic calibration of the camera and predict the camera intrinsic parameters when shooting the virtual scene map.

[0081] According to the camera intrinsic parameters, the points corresponding to the depth map can be converted from the image coordinate system to the camera coordinate system. According to the camera extrinsic parameters, the points in the camera coordinate system are converted to the points in the world coordinate system. All the points in the world coordinate system are collected to obtain point cloud data. Point cloud data is used to describe the surface features of the virtual scene graph, so that the ground area can be determined based on the surface features.

[0082] Step S506, determining whether the scene information includes binocular information.

[0083] In some embodiments of the present application, if the scene information includes binocular information, it means that the virtual scene graph is obtained by shooting with a binocular camera, wherein the binocular camera is stereo corrected. If the scene information does not include binocular information, execute step S508. If the scene information includes binocular information, execute step S507.

[0084] Step S507: Calculate the depth map corresponding to the virtual scene map.

[0085] In some embodiments of the present application, a depth map of a virtual scene map can be generated by calculating the disparity of the virtual scene map. Specifically, assuming that the binocular camera includes a first camera and a second camera, the virtual scene map includes a first map and a second map. The first map is acquired by the first camera, and the second map is acquired by the second camera. A method based on deep learning or a block matching method can be used to calculate the position difference of corresponding pixel points between the first map and the second map to obtain a disparity map. A depth map is generated based on the disparity map and the binocular camera parameters corresponding to the binocular camera.

[0086] In addition, in order to ensure the quality of the depth map, some post-processing may be performed on the depth map, such as image denoising, interpolation, edge optimization, etc.

[0087] Step S508, calculating the camera extrinsic parameters according to the vanishing points in the virtual scene graph.

[0088] In some embodiments of the present application, the vanishing point refers to the point where parallel lines converge in the distance in perspective projection. Linear algebra methods can be used to predict the vanishing point. For example, for each straight line in the virtual scene graph, the direction vector of each straight line is calculated. The target direction of the direction vector is calculated based on principal component analysis and other methods, and the target direction points to the vanishing point. The above is just an example, and there is no limitation on the calculation method of the vanishing point. The straight line detection method based on Hough transform can also be used for calculation.

[0089] When the vanishing point is obtained, the camera extrinsic parameters of the virtual camera relative to the ground area can be calculated according to the vanishing point. Specifically, the camera extrinsic parameters of the virtual camera are obtained according to the vanishing point, the optical center of the virtual camera and the shooting position of the virtual camera.

[0090] Step S509, determining the ground equation according to the camera external parameters and the preset ground features.

[0091] In some embodiments of the present application, the ground feature is a geometric feature describing the ground. A plurality of image points matching the ground feature are obtained from the virtual scene graph. Based on the camera external parameters, the plurality of image points are converted from the camera coordinate system to the world coordinate system. The ground equation is determined based on the points converted to the world coordinate system.

[0092] Step S510, calculating ground parameters according to ground equations.

[0093] In some embodiments of the present application, ground parameters, such as the normal vector and height of the ground, etc., may be calculated based on the ground equation.

[0094] Step S511, determining the ground area according to ground parameters.

[0095] In some embodiments of the present application, the ground area is marked in the virtual scene graph according to ground parameters such as the normal vector and height of the ground.

[0096] In the above embodiment, when the virtual scene graph carries scene information, the ground area can be determined based on the scene information. When the virtual scene graph does not carry scene information, the location of the ground area can also be predicted by the estimated vanishing point. By effectively determining the ground area, the rationality of adding the image corresponding to the virtual character in the virtual scene graph is increased.

[0097] Figure 6 This is a flowchart of building a database provided in the embodiment of the present application. Figure 6 As shown, the following steps are included.

[0098] Step S601, obtaining model rendering images, model depth images, model segmentation images, and model key point images of a preset three-dimensional model at multiple preset viewing angles, and recording camera parameters corresponding to each preset viewing angle, and camera positions corresponding to each preset viewing angle.

[0099] In some embodiments of the present application, the three-dimensional model may be a virtual character model constructed using three-dimensional reconstruction technology. The three-dimensional model is rendered using a virtual camera, and the rendering process involves all possible distances and angles between the virtual camera and the three-dimensional model. Redundant information will appear in the rendering process. For example, the imaging effect obtained when the virtual camera moves 1 meter toward the three-dimensional model is consistent with the imaging effect obtained when the three-dimensional model moves 1 meter toward the virtual camera. After removing the redundancy, there are multiple degrees of freedom in the rendering process, such as the X-coordinate of the three-dimensional model, the rotation matrix of the three-dimensional model around the Z-axis, and the height of the virtual camera. Among them, the direction of the Z-axis is perpendicular to the ground, and the X-coordinate is the projection point of the positive direction of the virtual camera on the ground. Figure 7 As shown, based on the degrees of freedom shown above, the rotation of the virtual camera around the optical center only changes the pixel distribution of the image, for example, the pixel distribution of the image when moving from imaging plane 1 to imaging plane 2. Therefore, Figure 2 In step S204 shown, the second viewing angle may be converted to a third viewing angle, so that the three-dimensional model is changed from the first pixel distribution to the second pixel distribution, satisfying the viewing effect of viewing the three-dimensional model at the third viewing angle.

[0100] In order to traverse all degrees of freedom during the rendering process, the position of the virtual camera can be fixed and the position of the 3D model can be moved to achieve the purpose of capturing data at different distances and angles.

[0101] In one example, all possible camera heights of the virtual camera are traversed, and a camera height is determined and fixed, assuming it is the camera height of (0,0,h). Move the 3D model so that it moves on the ground along the central axis of the virtual camera. Each time it moves to a feasible position, the virtual camera rotates at a fixed angle until it rotates 360°, and then moves the 3D model to the next feasible position. The fixed angle can be 10°, and the viewing angle generated after each rotation at a fixed angle is used as a preset viewing angle. The 3D model is rendered every 10° rotation. In addition, in order to avoid the following Figure 8 In the case where the human body is incomplete, when the three-dimensional model is moved to a feasible position, the internal parameters of the virtual camera can be adjusted so that the three-dimensional model can be completely located within the shooting range of the virtual camera without any additional spare parts within the shooting range.

[0102] In order to traverse all degrees of freedom during the rendering process, the position of the 3D model can also be fixed and the position of the virtual camera can be moved to achieve the purpose of capturing data at different distances and angles.

[0103] In one example, the 3D model is fixed at a certain position, and the virtual camera rotates around the position of the 3D model. Each rotation is performed at a fixed angle (e.g., 10°), and after rotating 360°, the height of the virtual camera or the radius around the 3D model is changed, and the results of each rotation at a fixed angle are recorded again. Figure 8 In the case of an incomplete human body shown, the internal parameters of the virtual camera can be adjusted by moving a fixed angle each time, so that the three-dimensional model can be completely located within the shooting range of the virtual camera without any additional spare parts within the shooting range.

[0104] In some embodiments of the present application, the three-dimensional model is rendered at multiple preset viewing angles to obtain a model rendering image, a model depth image, a model segmentation image, and a model key point image. The camera parameters corresponding to each preset viewing angle and the camera position corresponding to each preset viewing angle are recorded. The camera parameters may include camera intrinsic parameters and camera extrinsic parameters. The camera position may include a height position and a horizontal position.

[0105] Step S602, based on the second control condition and the preset style prompt words, the model rendering image is processed by the image generation model to obtain model character images of various styles.

[0106] In some embodiments of the present application, since the character model in the model rendering is not realistic enough, if the model rendering is directly added to the virtual scene graph later, it will affect the fusion effect of the image. Therefore, the model rendering can be processed using an image generation model to increase the detail information and realism of the character model in the model rendering. In addition, the virtual scene graph may have different styles. In order to further improve the fit between the virtual scene graph and the character model, a variety of different style prompts can be set for the image generation model. The style prompt is used to guide the image generation model to generate model character graphs of different styles when processing the model rendering.

[0107] In some embodiments of the present application, the model rendering, model depth map, model segmentation map and model key point map are input into the image generation model, and the model depth map, model segmentation map and model key point map are used as the second control condition of the image generation model. The image generation model can generate model character images of various styles based on the preset style prompt words and the second control condition. Fig. 9 As shown in the figure, the model character image obtained by the image generation model is more expressive than the model rendering image, and the model character image can show clearer contours and character details.

[0108] Step S603, constructing a database according to the model rendering image, the model depth image, the depth information corresponding to the model depth image, the model segmentation image, the model key point image, the model character image, the camera parameters and the camera position.

[0109] In some embodiments of the present application, a database can be constructed by obtaining a model rendering image, a model depth map, depth information corresponding to the model depth map, a model segmentation map, a model key point map, a model character map, camera parameters and a camera position.

[0110] In addition, in the field of images, obtaining a large number of annotated images as training data for the model is a crucial task. Since synthetic images have the advantages of covering more scenes and low cost, using synthetic images as training data can increase the diversity and richness of training data. However, the labels carried by such synthetic images depend on the synthesis effect. If the synthesis effect is not ideal, the labels of the synthetic images may be incomplete or inaccurate, affecting the training efficiency and accuracy of the model.

[0111] Therefore, in order to solve the technical problem that the synthetic image carries incomplete or inaccurate labels due to unsatisfactory synthesis effects, the embodiment of the present application will simultaneously record the corresponding labels in the process of obtaining the model rendering map, model depth map, model segmentation map, model key point map, and model character map. Any subsequent image obtained from the database carries the corresponding label, so that the second synthetic image obtained by fusing the model character map of the database with the virtual scene map has rich labels to meet the needs of training data.

[0112] In the above embodiment, the rendered model rendering is processed by the image generation model, so that the first character image subsequently obtained from the preset database has more detail features and edge features, thereby improving the fusion effect of the first character image and the virtual scene image. In addition, using the data in the preset database as training data can meet the richness and diversity of the training data, and at the same time, it can also ensure that the training data has accurate training labels.

[0113] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0114] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0115] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a ground area of ​​the virtual scene graph, wherein the ground area includes a target location; Acquire a first camera parameter corresponding to the target position under a first viewing angle; Determining a first character image from a preset database based on the first camera parameter, wherein the first character image is generated by processing a target rendering image based on a preset image generation model, wherein the target rendering image is a two-dimensional image obtained by converting a three-dimensional character model under a second viewing angle; Obtaining a second character image with a second pixel distribution according to a second camera parameter corresponding to the first character image, a first pixel distribution of the first character image, a third camera parameter corresponding to the virtual scene image at a third viewing angle, and a transformation matrix from the second viewing angle to the third viewing angle; Based on the depth information of the second person image, adding the second person image to the target position to obtain a first composite image; A second synthetic image is obtained based on the first synthetic image and the image generation model.

2. The image processing method according to claim 1, characterized in that: The step of obtaining a second synthetic image based on the first synthetic image and the image generation model comprises: Based on a first control condition, the synthetic rendering image is processed by the image generation model to obtain the second synthetic image, wherein the first control condition includes a synthetic depth map, a synthetic segmentation map, and a synthetic key point map corresponding to the first synthetic image.

3. The image processing method according to claim 1, characterized in that: The adding the second character image to the target position based on the depth information of the second character image to obtain a first composite image includes: Determine an occlusion area of ​​the second character image, where the occlusion area is an overlapping area between the second character image and a third character image in the virtual scene image; Based on the overlapping area, if the depth information of the second character image is greater than the depth information of the third character image, and the layer of the second character image is located above the layer of the third character image, move the layer of the second character image downward so that the layer of the second character image in the first composite image is located below the layer of the third character image; or, If the depth information of the second character image is less than the depth information of the third character image, and the layer of the second character image is below the layer of the third character image, move the layer of the second character image upward so that the layer of the second character image in the first composite image is above the layer of the third character image.

4. The image processing method according to claim 1, characterized in that: Determining the ground area of ​​the virtual scene graph includes: The ground area is determined according to scene information corresponding to the virtual scene graph.

5. The image processing method according to claim 4, characterized in that: The scene information includes ground parameters, and determining the ground area according to the scene information in the virtual scene graph includes: The ground area is determined based on a virtual scene graph carrying the ground parameters.

6. The image processing method according to claim 4, characterized in that: The scene information includes a depth map, and determining the ground area according to the scene information in the virtual scene map includes: Calculating point cloud data based on the depth map corresponding to the virtual scene map; The ground area is determined according to the point cloud data.

7. The image processing method according to claim 1, characterized in that: Determining the ground area of ​​the virtual scene graph includes: Calculating camera extrinsics according to the vanishing points in the virtual scene graph, wherein the camera extrinsics include a rotation matrix and a translation vector of a camera shooting the virtual scene graph relative to the ground area; Determining a ground equation according to the camera external parameters and preset ground features; Calculating ground parameters according to the ground equation; The ground area is determined according to the ground parameters.

8. The image processing method according to claim 1, characterized in that: The method further comprises constructing the database, comprising: Obtain model renderings, model depth maps, model segmentation maps, and model key point maps of a preset 3D model at multiple preset viewing angles, and record camera parameters corresponding to each preset viewing angle, as well as the camera position corresponding to each preset viewing angle; Based on a second control condition and a preset style prompt word, the model rendering image is processed by the image generation model to obtain model character images of multiple styles, wherein the second control condition includes the model depth image, the model segmentation image, and the model key point image; The database is constructed according to the model rendering image, the model depth image, the depth information corresponding to the model depth image, the model segmentation image, the model key point image, the model character image, the camera parameters and the camera position.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the image processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

Images