Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium. The method comprises the steps that a two-dimensional image and a three-dimensional conversion control are displayed, and the two-dimensional image comprises a first object wearing first clothes; in response to a trigger operation for the three-dimensional conversion control, displaying a three-dimensional image, the three-dimensional image including the first costume; and in response to a face image uploading operation, displaying the first costume fused with the uploaded face image. According to the invention, immersive try-on experience can be realized.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Virtual fitting refers to the use of virtual technology to display images of new clothing after trying it on. Virtual fitting technology is a technical application that allows users to put on new clothing and see the effect without taking off their clothes.

[0003] However, in the solutions provided by related technologies, the try-on effect is usually displayed in a two-dimensional image, that is, the try-on effect is displayed in a two-dimensional image space, resulting in the display of the try-on effect being not comprehensive and realistic enough, and also resulting in a poor try-on experience for users. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can realize an immersive fitting experience.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an image processing method, including:

[0007] Displaying a two-dimensional image and a three-dimensional conversion control, wherein the two-dimensional image includes a first object wearing a first garment;

[0008] In response to a trigger operation on the three-dimensional conversion control, displaying a three-dimensional image, wherein the three-dimensional image includes the first clothing;

[0009] In response to the facial image upload operation, the first clothing fused with the uploaded facial image is displayed.

[0010] The present application provides an image processing device, including:

[0011] A display module, used for displaying a two-dimensional image and a three-dimensional conversion control, wherein the two-dimensional image includes a first object wearing a first garment;

[0012] The display module is further configured to display a three-dimensional image in response to a trigger operation on the three-dimensional conversion control, wherein the three-dimensional image includes the first clothing;

[0013] The display module is further configured to display the first clothing integrated with the uploaded facial image in response to a facial image upload operation.

[0014] An embodiment of the present application provides an electronic device, including:

[0015] A memory for storing executable instructions;

[0016] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement the image processing method provided in the embodiment of the present application when executed by a processor.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] By converting a two-dimensional image into a three-dimensional image and fusing the facial image uploaded by the user with the first clothing included in the three-dimensional image to display the first clothing fused with the facial image uploaded by the user, a realistic three-dimensional fitting effect can be obtained, thereby enabling an immersive fitting experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of the architecture of an image processing system 100 provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0023] Figure 3 is a flowchart of an image processing method provided in an embodiment of the present application;

[0024] Figure 4A and Figure 4B is a flowchart of an image processing method provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application;

[0026] FIG. 6A to FIG. 6E is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application;

[0027] Figure 7 is a flowchart of an image processing method provided in an embodiment of the present application;

[0028] Figure 8 It is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0030] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] It is understandable that in the embodiments of the present application, when user information and other related data (such as the user's facial image) are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0032] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed may be executed.

[0036] 2) Two-dimensional (2D) image: refers to an image consisting of two dimensions, length and width. A two-dimensional image is also called a plane image, that is, a two-dimensional image only shows the plane that the user sees.

[0037] 3) Three-dimensional (3D) image: refers to an image composed of three dimensions: length, width, and height. A three-dimensional image is also called a stereoscopic image. A three-dimensional image displays a three-dimensional shape and looks three-dimensional. Users can view objects in the three-dimensional image from different perspectives by rotating it.

[0038] 4) Red, Green and Blue (RGB): It is a color standard that obtains various colors by changing the three color channels of red (R, Red), green (G, Green), and blue (B, Blue) and superimposing them on each other.

[0039] 5) Skinned Multi-Person Linear Model (SMPL): is a parametric human body model that can accurately represent different shapes and poses of the human body. The SMPL model is a learnable model that can better fit the shape of the human body and its deformation under different poses through training.

[0040] 6) Vertex: The parametric human body model can be considered as composed of multiple small triangles (or quadrilaterals), each of which can be used as a vertex. The more vertices there are, the more refined the parametric human body model will be.

[0041] 7) Semantic code: Also known as semantic code, it is a three-dimensional vector that includes the spatial coordinates and posture information of the object.

[0042] 8) Voxelization: It converts the geometric representation of an object into a voxel representation that is closest to the object, generating volume data, including the surface information and internal properties of the model. The voxels that represent a three-dimensional model are similar to the pixels that represent a two-dimensional image, except that they are expanded from two-dimensional points to three-dimensional cubic units. Voxelization can simplify the model and obtain a uniform and consistent grid, which is well applied in the process of slicing the model and physical simulation analysis.

[0043] 9) 3D human body model dataset: a 3D human body dataset with high resolution and including clothing surface features. Taking the Thuman dataset as an example, the Thuman dataset includes 7,000 human meshes in random poses wearing 230 types of clothing.

[0044] 10) Point cloud data: refers to a set of vectors in a three-dimensional coordinate system. In addition to geometric positions, point cloud data can also have color information. Color information is usually obtained by the camera to obtain color images, and then the color information of the pixels at the corresponding position is assigned to the corresponding points in the point cloud.

[0045] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can realize an immersive fitting experience. The electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as a terminal device, or can be implemented by a server and a terminal device in collaboration. The following is an example of the image processing method provided by the embodiments of the present application being implemented by a server and a terminal device in collaboration.

[0046] For example, see Figure 1 , Figure 1 is a schematic diagram of the architecture of the image processing system 100 provided in the embodiment of the present application, in order to realize the application of supporting the application of showing the fitting effect to the user more comprehensively and realistically, such as Figure 1 As shown, the image processing system 100 includes: a server 200, a network 300 and a terminal device 400, wherein the network 300 can be a local area network or a wide area network, or a combination of the two, and the terminal device 400 is a terminal device associated with a user, and a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as an e-commerce shopping client, a live broadcast client, a browser, etc.

[0047] In some embodiments, taking the client 410 as a browser as an example, the browser responds to the user's image search operation for the first object (e.g., a game character or an anime character), wherein the types of image search operations include text-to-image image search operations and image-to-image image search operations. For text-to-image image search operations, the user enters text (e.g., the name of the first object) in the search box; for image-to-image image search operations, the user enters an image (e.g., the image of the first object) in the search box. For example, taking the text-to-image image search operation as an example, the browser receives the name of the first object entered by the user in the search box, obtains a two-dimensional image associated with the first object from the server 200, such as the two-dimensional image may include the first object wearing the first clothing, and displays the two-dimensional image returned by the server 200 and a three-dimensional conversion control (e.g., a 3D mode button) in the search result display area. When a user clicks on the three-dimensional conversion control, the two-dimensional image displayed in the search result display area can be converted into a three-dimensional image, that is, the three-dimensional image of the first object is switched to be displayed in the search result display area, wherein the three-dimensional image includes the first clothing. Subsequently, the browser responds to the facial image upload operation triggered by the user and displays the first clothing that is integrated with the facial image uploaded by the user (including the facial image acquired in real time and the facial image taken in advance). In this way, the user can view the fitting effect from different perspectives, thereby improving the user's fitting experience.

[0048] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0049] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0050] For example, Figure 1 The server 200 in the embodiment may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a virtual reality (VR) device, an augmented reality (AR) device, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0051] In some embodiments, the terminal device can also implement the image processing method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native (Native) application (APP, APPlication), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or a browser; it can also be a small program, that is, a program that only needs to be downloaded to the browser environment to run. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0052] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a terminal device as an example, see Figure 2 , Figure 2is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0053] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0054] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0055] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0056] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0057] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0058] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0059] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0060] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0061] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0062] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The image processing device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a display module 5551, an acquisition module 5552, a determination module 5553, a control module 5554, a generation module 5555, a reconstruction module 5556, a refinement module 5557 and a synthesis module 5558. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the image processing device 555 excludes the implementation that only includes the display module 5551. The functions of each module will be explained below.

[0063] The image processing method provided in the embodiment of the present application will be specifically described below in conjunction with the exemplary application and implementation of the terminal device provided in the embodiment of the present application.

[0064] See also Figure 3 , Figure 3 is a flowchart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0065] It should be noted that Figure 3The method shown can be executed by various forms of computer programs running on the terminal device, and is not limited to the client. For example, it can also be the operating system, software module, script, and applet mentioned above. Therefore, the example of the client in the following should not be regarded as a limitation on the embodiments of the present application. In addition, for the convenience of expression, the terminal device and the client running on the terminal device are not specifically distinguished in the following.

[0066] In step 101, a two-dimensional image and a three-dimensional conversion control are displayed.

[0067] Here, the two-dimensional image includes a first object wearing a first garment, wherein the first garment may be a T-shirt, a windbreaker, a down jacket, a dress, etc., and the first object may be a game character in a game, a cartoon character, a virtual character in a film or TV series, or a real person in the real world, etc.

[0068] In some embodiments, the above-mentioned display of two-dimensional images and three-dimensional conversion controls can be implemented in the following manner: in response to an image search operation for a first object, at least one two-dimensional image and a three-dimensional conversion control are displayed in a search result display area, wherein at least one two-dimensional image includes a first object wearing a first clothing item.

[0069] For example, taking the client as a browser, a search box and a search result display area are displayed in the browser interface. When receiving a keyword (such as the name of the first object) entered by the user in the search box, and the search type selected by the user is a picture, the browser can send a search request to the server, wherein the search request carries the keyword entered by the user. After receiving the search request sent by the browser, the server returns a two-dimensional image related to the first object to the browser, for example, the two-dimensional image may include the first object wearing the first clothing, and then the browser can display the two-dimensional image returned by the server and a three-dimensional conversion control (such as a 3D mode button) in the search result display area.

[0070] It should be noted that the number of two-dimensional images returned by the server can be multiple, that is, the browser can display multiple two-dimensional images in the search result display area, and the clothing worn by the first object included in different two-dimensional images can be different. For example, suppose that the server returns three two-dimensional images to the browser, namely two-dimensional image 1 to two-dimensional image 3, where two-dimensional image 1 includes the first object wearing a T-shirt, two-dimensional image 2 includes the first object wearing a windbreaker, and two-dimensional image 3 includes the first object wearing a down jacket. The embodiment of the present application does not make any specific limitations on this.

[0071] In step 102, in response to a trigger operation on a three-dimensional conversion control, a three-dimensional image is displayed.

[0072] Here, the three-dimensional image may include the first apparel.

[0073] In some embodiments, the three-dimensional image may include a background area and a foreground area, wherein the background area includes the background of the stretched two-dimensional image, the foreground area includes a first object wearing a first garment, and the first object is three-dimensional.

[0074] For example, the background of a stretched two-dimensional image can be generated in the following way: identify the background from the two-dimensional image and construct a grid corresponding to the background; divide the background into multiple triangular faces and use the intersection points of the grid lines as the vertices of the triangular faces; paste the multiple triangular faces on the grid in the form of texture mapping, so that the three-dimensional image can be made more three-dimensional.

[0075] In some other embodiments, before displaying the three-dimensional image, the following processing may also be performed: obtaining multiple search result images for the first object, wherein the multiple search result images are obtained by photographing the first object from different perspectives; generating a three-dimensional face model of the first object based on the multiple search result images; obtaining a parameterized human body model; performing surface geometry reconstruction on the parameterized human body model to obtain a reconstructed parameterized human body model; performing fine processing on the reconstructed parameterized human body model to obtain a three-dimensional human body model of the first object; and synthesizing the three-dimensional human body model and the three-dimensional face model to obtain a three-dimensional image.

[0076] By way of example, the above-mentioned generation of a three-dimensional facial model of the first object based on multiple search result images can be achieved in the following manner: for each search result image in the multiple search result images, feature extraction is performed on the pixel points in the area where the face of the first object is located in the search result image to obtain multiple pixel point features (for example, including the coordinates of the pixel points, color information, etc.); the camera parameters (for example, camera intrinsic parameters) used when shooting the multiple search result images are obtained; the multiple pixel point features corresponding to the multiple search result images are converted into point cloud data, wherein the constraints of the conversion are the camera parameters; and based on the point cloud data, a three-dimensional facial model of the first object is generated.

[0077] It should be noted that, in order to generate a three-dimensional face model of the first object, after the client obtains multiple search result images for the first object from the server, it can filter out images of different perspectives of the first object as basic images for generating a three-dimensional face model, and the number is not less than 60. Then, for the filtered basic image, the features of each pixel in the image, such as the coordinates of the pixel, color information, etc., can be calculated. The principle of the image generation model is that three points form a surface. Through the pixel points of the image, it is ensured that each point on the model corresponds to three pixels to generate a surface, and then these pixel features are converted into point cloud data, and a three-dimensional face model is generated based on the point cloud data. In addition, since the pixel features extracted from the basic image correspond to the image coordinate system, and the point cloud data corresponds to the world coordinate system, when converting the pixel features into point cloud data, that is, when converting from the image coordinate system to the world coordinate system, the constraints of the transformation are the camera intrinsic parameters, such as the focal length of the camera, the pixel size, etc.

[0078] The following continues to describe the process of generating a three-dimensional face model of the first object based on the point cloud data.

[0079] In some embodiments, the above-mentioned generation of a three-dimensional face model of the first object based on point cloud data can be achieved in the following manner: first, the point cloud data is preprocessed, for example, including operations such as noise removal, filtering, alignment and fusion, and then the point clouds of different perspectives are registered to combine them into a unified coordinate system, for example, it can be achieved by an iterative closest point (ICP) algorithm or other global registration methods. Subsequently, surface information can be extracted from the point cloud, for example, a triangular mesh can be generated based on the point cloud data, or the point cloud data can be converted into a finer surface mesh, or a point-based method can be used to generate a surface. Finally, the generated three-dimensional face model can be optimized and simplified to reduce the complexity of the model and improve rendering performance, for example, including operations such as removing overlapping vertices, simplifying mesh topology, and remapping textures.

[0080] So far, the generation of the 3D face model of the first object based on the multiple search result images has been completed. The following will continue to describe the process of reconstructing the 3D human body model of the first object from a single image.

[0081] It should be noted that different techniques can be used to reconstruct a 3D human model from a single image. For example, an end-to-end convolutional neural network can be used to directly reconstruct the 3D geometry of the human body through volume regression. This method is a method that combines implicit functions with rich details and parametric representations to reconstruct a 3D human model that remains controllable and accurate even with clothes. Given a sparse 3D point cloud collected on the wearer's surface, an implicit network is used to jointly predict the semantic correspondence between the wearer's external 3D surface, the internal body surface, and the parametric body model. Subsequently, the body model is fitted to the inner surface using the correspondence, and then non-rigidly deformed (under a parametric body + displacement model) to the outer surface to capture clothing, facial and hair details. Of course; a method for learning time-consistent 3D reconstruction from monocular video can also be used. This method uses volumetric, implicit or parametric human models to reconstruct 3D humans from monocular videos. In addition, DeepHuman can also be used to reconstruct the 3D human body model. DeepHuman is a deep learning-based framework, which is an image-guided body-to-body conversion convolutional neural network (CNN) for 3D human body reconstruction from a single RGB image. The following takes DeepHuman as an example to specifically explain the process of reconstructing the 3D human body model of the first object from a single image.

[0082] In some embodiments, in order to reduce the ambiguity associated with the reconstruction of surface geometry, even for the reconstruction of invisible areas, the embodiments of the present application propose and utilize dense semantic representations (including semantic graphs and semantic bodies) generated from the SMPL model as additional input. In addition, a key feature of the network is the fusion of image features of different scales into three-dimensional space through volume feature transformation, which helps to restore accurate surface geometry. Visible surface details are further refined by a conventional fine network, which can be connected to the volume generation network using a volume normal projection layer. In addition, the embodiments of the present application also provide a three-dimensional real human model dataset (i.e., Thuman dataset) including 7,000 models, and the network is trained using training data generated by the dataset. Once the network is trained, it can predict an image of a person and the corresponding SMPL estimated occupancy and normal map of the visible surface.

[0083] For example, in order to provide a reasonable initialization and constrained degree of freedom output space for the network, the embodiment of the present application uses a parametric human body model (such as an SMPL model) to generate a 3D semantic body and a 2D semantic map to represent the input image of the body shape and posture parameter template of the estimated parameters. Among them, the input image, the 2D semantic map and the 3D semantic body will be input into an image-guided body-to-body conversion CNN for surface reconstruction. In order to restore surface geometry such as hairstyles or clothing contours as accurately as possible, the embodiment of the present application also proposes a multi-scale volume feature transformation to fuse these different scales of image guidance information into a three-dimensional volume. Finally, a volume normal projection layer is introduced according to the input image to further refine and enrich the visible surface details. This layer can be designed to connect the volume generation network and the normal refinement network and support end-to-end training. In summary, the embodiment of the present application can divide the reconstruction task of the three-dimensional human body model into three subtasks: a) parameterized body estimation from the input image; b) surface reconstruction from the image and the estimated body; c) refinement of visible surface details according to the image. That is, the entire modeling process is performed in a coarse-to-fine manner, starting from the overall estimation, then entering the surface reconstruction, and finally restoring the details of the surface frontal area. The above process is described in detail below.

[0084] In some embodiments, in order to estimate the external body (including shape parameters and posture parameters) of the first object in the two-dimensional image, the embodiments of the present application may use HMR and Simplify, wherein HMR is an end-to-end framework for reconstructing the entire human body three-dimensional mesh from a single RGB image, and is used for end-to-end recovery of human body shape and posture, and Simplify is a method for automatically estimating three-dimensional human body posture and shape from a single image, that is, both HMR and Simplify can infer the shape and posture parameters of the first object from a single image, and HMR and Simplify have complementary features, for example, HMR has strong predictive ability, but is not completely consistent with color images, while Simplify can align the SMPL model with the detected key points well, but relies on initialization to output reliable results. In order to obtain an accurate estimate of the SMPL image, the embodiments of the present application may first use HMR to obtain an initial estimate, and then use Simplify to improve the accuracy.

[0085] In some embodiments, in order to be able to input the SMPL estimation into the CNN, the following processing can also be performed before the surface geometry of the parameterized human body model is reconstructed: the spatial coordinates of the first object in a static posture are obtained from the two-dimensional image; a semantic code is predefined for each vertex on the parameterized human body model according to the spatial coordinates in the static posture; the parameterized human body model is voxelized into a voxel grid, and the semantic code is propagated to the occupied voxel grid to obtain a semantic body, and the semantic code is drawn onto the image plane to obtain a semantic map.

[0086] For example, in order to estimate the shape and posture parameters of the body of the first object, the embodiment of the present application introduces a dense semantic representation generated from the SMPL model. For example, a semantic code (i.e., a three-dimensional vector for characterizing the spatial coordinates and posture information of the first object) can be pre-defined for each vertex on the SMPL model according to the spatial coordinates of the first object in a static posture. Then, according to the SMPL estimation, the SMPL model is voxelized into a voxel grid, and then the semantic code is propagated to the occupied voxels to generate a semantic body (assuming it is denoted as V s ), and draw the semantic code onto the image plane to obtain a semantic map (assuming it is denoted as M s ). The dense semantic representation provided by the embodiment of the present application has the following three advantages: (1) it encodes the shape and posture information of the first object's body, thereby providing a reasonable initialization for the network and constraining the degrees of freedom of the output space; (2) it provides clues about the correspondence between three-dimensional voxels and two-dimensional image pixels; (3) it is easy to integrate into neural networks.

[0087] In some embodiments, after obtaining the semantic map and the semantic body, the above-mentioned surface geometry reconstruction of the parameterized human body model can be achieved in the following manner: initialize a three-dimensional occupied voxel grid, wherein the three-dimensional occupied voxel grid is used to represent the surface of the parameterized human body model, and the voxel values ​​within the surface of the three-dimensional occupied voxel grid are set to 1, and the other voxel values ​​are set to 0; encode the two-dimensional image and the semantic map at different scales to obtain a multi-scale two-dimensional feature map; and fuse the multi-scale two-dimensional feature map and the semantic body into the three-dimensional occupied voxel grid through multi-scale volume feature transformation.

[0088] For example, the embodiment of the present application may use an occupied volume to represent a surface. For example, a 3D occupied voxel grid (assuming that it is denoted as V 0), where the voxel value within the surface can be set to 1, and the other voxel values ​​can be set to 0. The fixed resolution of all occupied volumes can be 128×192×128, where the resolution of the y-axis can be set larger so that it can automatically adapt to the long axis of the observed human body. In order to reconstruct a 3D occupied voxel grid from a semantic volume with the help of an input image (i.e., a two-dimensional image, assumed to be denoted as I) and a semantic map, an embodiment of the present application proposes an image-guided volume-to-volume conversion grid, and uses a multi-scale volume feature transformation to fuse the two-dimensional image guidance information into the three-dimensional volume, thereby completing the surface geometry reconstruction. In other words, the network will utilize information from both the two-dimensional image and the three-dimensional volume.

[0089] In some embodiments, the above-mentioned refined processing of the reconstructed parameterized human body model can be achieved in the following manner: obtaining a three-dimensional human body model data set; training a grid based on the three-dimensional human body model data set, and extracting feature information of the first clothing from the two-dimensional image through the trained grid; projecting the three-dimensional occupied voxel grid into a two-dimensional normal map through a U-net network (U-Net); encoding the feature information of the first clothing on the two-dimensional normal map, and pasting the encoded two-dimensional normal map on the surface of the reconstructed parameterized human body model.

[0090] For example, due to resolution limitations, the voxel grid cannot capture subtle details, such as wrinkles in clothing. In order to further enrich and refine the geometric details of the visible parts of the surface, the embodiment of the present application also projects a 2D normal map directly from the 3D occupied voxel grid and refines it using U-Net. In other words, the embodiment of the present application uses a two-dimensional normal map to encode the geometric details of the visible surface, which can be calculated directly from the occupied volume using a differentiable volume to normal projection layer, which first projects the depth map directly from the occupied space, converts the depth map into a vertex map, and then calculates the normal map through a series of mathematical operations, thereby reducing memory requirements.

[0091] In addition, in order to train the network under supervision, the embodiment of the present application also provides a real-world 3D human model dataset Thuman, and a training corpus can be synthesized from the Thuman dataset. Once the network is trained, it can predict an image of a person and the corresponding SMPL estimated occupancy and visible surface discovery map. For example, a triangular mesh (or polygonal mesh) can be first extracted from the occupied body using the marching cube algorithm, and then refined according to the two-dimensional normal map to obtain the final reconstruction result.

[0092] It should be noted that the network provided in the embodiment of the present application is composed of three components, namely, an image encoder (assuming it is denoted as G), a volume-to-volume conversion network (assuming it is denoted as H) and a normal refinement network (assuming it is denoted as R), wherein the goal of the image encoder is to extract a multi-scale two-dimensional feature map (assuming it is denoted as Mf(k), k=1,...,K, K is a positive integer greater than 1), which is composed of an input image and a semantic map. The volume-to-volume conversion network is a volumetric U-type network (U-Net), wherein the semantic body and the multi-scale two-dimensional feature map serve as the input of the volume-to-volume conversion network, and the output is a 3D occupied voxel grid representing a surface. The volume-to-volume conversion network provided in the embodiment of the present application fuses multi-scale two-dimensional feature maps and allows them to enter the encoder through a multi-scale volume feature converter. Specifically, the multi-scale volume feature converter first maps the multi-scale two-dimensional feature map to the modular parameters through a convolution process and an activation process. Since the modular parameters are two-dimensional and the semantic body is three-dimensional, it is necessary to divide the semantic body into layers of slices with a height of 1 on the z-axis, and then perform affine transformation (i.e. linear transformation + translation) with the modular parameters to obtain the converted semantic map. After generating the 3D occupied voxel grid, the 3D occupied voxel network is projected into a two-dimensional normal map through the refinement network, for example, it can be calculated through a volume normal projection layer. All operations in the network are differentiable, so they can be trained or fine-tuned in an end-to-end manner.

[0093] So far, the construction from a single image to a 3D human body model has been completed. Since the modeling mentioned above is a process from coarse to fine, the modeling of a single image still has certain limitations for the construction of the face. Therefore, the 3D face model generated in the previous step can be used to replace and correct the face of the 3D human body model, thereby obtaining a 3D image of the first object.

[0094] In some embodiments, see Figure 4A , Figure 4A is a flow chart of the image processing method provided in the embodiment of the present application, such as Figure 4A As shown, after executing Figure 3 After step 102 shown, you can also perform Figure 4A Step 104 is shown, combining Figure 4A The steps shown are explained.

[0095] In step 104, in response to a collection operation on the first clothing item, the first clothing item is displayed in a collection interface of the virtual wardrobe prop.

[0096] In some embodiments, before responding to the collection operation on the first clothing, the following processing can also be performed: in response to the zoom-in operation on the three-dimensional image, the three-dimensional image is displayed based on the zoom-in mode, and prompt information is displayed, wherein the prompt information is used to prompt the collection of the first clothing; the trigger operation on the prompt information is identified as the collection operation on the first clothing.

[0097] For example, the above-mentioned zoom-in operation can be a click operation on a zoom-in key, or a multi-finger spread operation (for example, a four-finger spread operation or a three-finger spread operation). For example, when a user clicks a zoom-in key (which can be a virtual key or a physical key on the keyboard), the three-dimensional image can be displayed based on the zoom-in mode. For example, when the user clicks the zoom-in key for the first time, the three-dimensional image can be magnified 1.5 times. When the user clicks the zoom-in key again, the three-dimensional image can be magnified to 3 times. That is to say, each time the user clicks the zoom-in key, the three-dimensional image can be magnified again until it is magnified to the maximum multiple.

[0098] It should be noted that while the three-dimensional image is being magnified, a prompt message may also be displayed in the three-dimensional image, wherein the prompt message is used to prompt the user to collect the first clothing item worn by the first object. When a click operation of the user in response to the prompt message is received, the first clothing item may be added to the collection interface of the virtual wardrobe props, and a prompt message indicating successful collection is displayed. Of course, the click operation on the first clothing item may also be identified as a collection operation on the first clothing item. For example, when a click operation of the user on the first clothing item included in the three-dimensional image in the magnified mode is received, the first clothing item may be added to the collection interface of the virtual wardrobe props. This embodiment of the present application does not specifically limit this.

[0099] In other embodiments, see Figure 4B , Figure 4B is a flow chart of the image processing method provided in the embodiment of the present application, such as Figure 4B As shown, after executing Figure 3 After step 102 shown, you can also perform Figure 4B Step 105 shown combines Figure 4B The steps shown are explained.

[0100] In step 105 , in response to a sliding operation on the 3D image, the 3D image is controlled to rotate.

[0101] Here, the rotation angle can be a fixed value, for example, the three-dimensional image can be rotated 20 degrees each time the user slides; or the rotation angle can also be positively correlated with the sliding distance of the sliding operation, that is, the farther the user slides, the greater the angle of rotation of the three-dimensional image, for example, when the user slides a distance of 1 cm, the three-dimensional image can be rotated 10 degrees; when the user slides a distance of 2 cm, the three-dimensional image can be rotated 20 degrees. That is, after converting the two-dimensional image into a three-dimensional image, the user can rotate the first object in the three-dimensional image so that it can be observed from different perspectives.

[0102] In step 103, in response to the facial image upload operation, a first garment fused with the uploaded facial image is displayed.

[0103] In some embodiments, the facial image upload control can be displayed simultaneously with the three-dimensional image, that is, when a user clicks on the three-dimensional conversion control, in addition to switching to display the three-dimensional image of the first object, the facial image upload control can also be displayed at the same time. Step 103 can be implemented in the following way: in response to the trigger operation on the facial image upload control, a facial image (for example, a facial image acquired in real time or a facial image taken in advance) is uploaded, and the first clothing that is integrated with the uploaded facial image is displayed.

[0104] For example, the client is a browser, see Figure 5 , Figure 5 is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Figure 5As shown, the browser responds to the image search operation for the first object triggered by the user, for example, the browser receives the name of the first object entered by the user in the search box, sends a request to the server to obtain the two-dimensional image associated with the first object, and after receiving the two-dimensional image returned by the server, displays the two-dimensional image 502 and the three-dimensional conversion control 503 (for example, a 3D mode button) in the search result display area 501, wherein the two-dimensional image 502 includes the first object wearing the first clothing, for example, the virtual character A wearing a windbreaker. When the user clicks the three-dimensional conversion control 503, the three-dimensional image 504 is switched to be displayed in the search result display area 501, wherein the three-dimensional image 504 includes the first object wearing the first clothing, and the first object is three-dimensional. While displaying the three-dimensional image 504, a facial image upload control 505 can also be displayed. When a user clicks on the facial image upload control 505, a facial image 506 acquired in real time can be uploaded to the server, so that the server can merge the facial image 506 uploaded by the browser with the first clothing included in the three-dimensional image 504, and return the first clothing 507 merged with the uploaded facial image 506 to the browser for display by the browser.

[0105] It should be noted that the above-mentioned process of fusing the facial image 506 uploaded by the user with the first clothing included in the three-dimensional image 504 can also be implemented by the terminal device. For example, after the browser obtains the facial image uploaded by the user, it can call the local computing power of the terminal device to fuse the facial image uploaded by the user with the first clothing. The embodiment of the present application does not make specific limitations on this.

[0106] In other embodiments, the facial image upload control can also be displayed in the collection interface, that is, when displaying the three-dimensional image, the entrance to the virtual wardrobe props can also be displayed at the same time, and the above-mentioned step 103 can be implemented in the following way: in response to the trigger operation of the entrance to the virtual wardrobe props, the collection interface of the virtual wardrobe props is displayed, wherein the collection interface includes the first clothing (i.e., the physical part of the first object is hidden, such as the exposed hands, feet, neck, face, etc.) and the facial image upload control; in response to the trigger operation of the facial image upload control, the first clothing integrated with the uploaded facial image is displayed.

[0107] It should be noted that the first clothing in the two-dimensional image is two-dimensional, and the first clothing in the three-dimensional image is three-dimensional, that is, the first clothing in the three-dimensional image can be obtained by three-dimensional modeling of the first clothing in the two-dimensional image. Moreover, the first clothing displayed in the collection interface is also three-dimensional. In other words, the user can view the first clothing displayed in the collection interface from different perspectives by rotating. In addition, it should be noted that although the first clothing displayed in the collection interface hides the physical body of the first object, the parameters of the first object's body (such as limb parameters) can be attached to the voxel grid parameters of the first clothing for subsequent recovery.

[0108] For example, taking the client as a browser, when a user clicks on a three-dimensional conversion control, the three-dimensional image and the entrance to the virtual wardrobe props can be switched in the search result display area. When a user clicks on the entrance to the virtual wardrobe props, the collection interface of the virtual wardrobe props can be displayed, wherein the collection interface includes the first clothing previously collected by the user and the facial image upload control. When a user clicks on the facial image upload control, a gallery (such as a local album) can be displayed, wherein the gallery includes multiple candidate facial images (such as facial images pre-photographed by the user or pre-downloaded by the user from the network). The browser responds to the facial image selection operation triggered by the user. For example, assuming that a user clicks on the facial image 1 displayed in the gallery, the facial image 1 is sent to the server as an uploaded facial image, so that the server merges the facial image 1 with the first clothing included in the three-dimensional image, and returns the first clothing merged with the facial image 1 to the browser for display by the browser.

[0109] In other embodiments, the uploaded facial image may be acquired in real time or may be pre-photographed. In this case, before displaying the first garment incorporating the uploaded facial image, the following processing may be performed: displaying a facial image acquisition area (e.g., a circular or elliptical facial image acquisition area); using the facial image acquired in real time through the facial image acquisition area as the uploaded facial image; or, displaying a gallery (including a local gallery and a network gallery), wherein the gallery includes a plurality of candidate facial images; and in response to a facial image selection operation, using the facial image selected in the gallery as the uploaded facial image.

[0110] In some embodiments, before displaying the first clothing item fused with the uploaded facial image, the following processing may also be performed: performing three-dimensional modeling on the uploaded two-dimensional facial image to obtain a three-dimensional facial image; extracting posture parameters from a voxel grid included in a clothing model pre-built for the first clothing item; adjusting the three-dimensional facial image to an angle that matches the posture parameters; and fusing the adjusted three-dimensional facial image with the first clothing item to obtain the first clothing item fused with the uploaded facial image.

[0111] For example, after receiving the two-dimensional facial image uploaded by the user, the DeepHuman model can be used to perform three-dimensional modeling on the two-dimensional facial image uploaded by the user to obtain a three-dimensional facial image. In addition, when constructing a clothing model for the two-dimensional image, the voxel grid will mark the head posture parameters of the first object. Therefore, the three-dimensional facial image obtained by three-dimensional modeling can adjust the angle of the head with the help of the head posture parameters marked by the voxel grid, so that the three-dimensional facial image can be perfectly integrated with the first clothing.

[0112] It should be noted that the method of using the DeepHuman model to perform three-dimensional modeling on the uploaded two-dimensional facial image is similar to the process of constructing a three-dimensional human body model mentioned above, and can be implemented by referring to the process of reconstructing a three-dimensional human body model from a single image mentioned above, and the embodiments of the present application will not be repeated here.

[0113] In other embodiments, when the first garment is a full-body garment, the above-mentioned step 103 can also be implemented in the following manner: in response to the facial image upload operation, based on the limb parameters marked in the voxel grid included in the clothing model pre-built for the first garment, restore the limb image; and display the first garment that incorporates the uploaded facial image and limb image.

[0114] For example, when a user clicks on a facial image upload operation, marked limb parameters can be extracted from a voxel grid included in a pre-built clothing model for a first clothing item, and the extracted limb parameters can be rendered to restore the limb image. Subsequently, the facial image uploaded by the user, the restored limb image, and the first clothing item are fused to obtain the first clothing item that incorporates the uploaded facial image and the limb image. In this way, a complete human body synthesis can be displayed, thereby improving the user's fitting experience.

[0115] In some embodiments, when the collection interface includes multiple clothing items including a first clothing item, before responding to a trigger operation on a facial image upload control, the following processing may also be performed: in response to a clothing selection operation, displaying the selected first clothing item based on a focus state.

[0116] For example, assuming that the user has collected multiple garments including the first garment, that is, the collection interface displays multiple garments including the first garment, then before trying on, the user needs to select the garment that needs to be tried on from the multiple garments. For example, assuming that a click operation of the user on the first garment among the multiple garments displayed in the collection interface is received, the first garment can be displayed based on the focus state, for example, the first garment can be highlighted, or the first garment can be displayed based on the zoom mode to indicate that the first garment is currently in the selected state. In other words, the garment that the user wants to try on next is the first garment.

[0117] In other embodiments, when the selected first clothing is displayed based on the focus state, the following processing can also be performed: in response to a sliding operation on the first clothing, the first clothing is controlled to rotate in the collection interface, wherein the rotation angle is a fixed value, or the rotation angle is positively correlated with the sliding distance of the sliding operation.

[0118] For example, taking the first garment as garment A, when a sliding operation triggered by the user on garment A is received, garment A can be controlled to rotate in the collection interface to facilitate the user to observe garment A from different perspectives. In addition, the rotation angle of garment A can be a fixed value, for example, each time the user triggers a sliding operation on garment A, garment A rotates 30 degrees; of course, the rotation angle of garment A can also be positively correlated with the sliding distance of the sliding operation triggered by the user, that is, the longer the sliding distance of the sliding operation triggered by the user, the greater the rotation angle of garment A. For example, when the sliding distance of the sliding operation triggered by the user on garment A is 1 cm, garment A rotates 15 degrees in the collection interface accordingly; when the sliding distance of the sliding operation triggered by the user on garment A is 2 cm, garment A rotates 30 degrees in the collection interface accordingly.

[0119] In some embodiments, when the collection interface includes multiple items of clothing including a first item of clothing, the multiple items of clothing may be displayed in a specific sorting manner, wherein the sorting manner includes: the order of collection time of the multiple items of clothing from late to early; the order of colors of the multiple items of clothing from dark to light; the order of sizes of the multiple items of clothing from large to small.

[0120] For example, assuming that the user has collected 5 clothing items, namely clothing 1 to clothing 5, these 5 clothing items can be sorted in the collection interface in the order of collection time from late to early, that is, the clothing items with later collection time are sorted higher, so that it is convenient for users to give priority to the clothing items that have been collected recently; of course, these 5 clothing items can also be sorted in the order of color from dark to light, or from light to dark, so that it is convenient for users to give priority to the clothing items with the darkest or lightest color for fitting. In addition, these 5 clothing items can also be arranged in the order of size from large to small, or from small to large, so that it is convenient for users to give priority to the largest or smallest clothing items for fitting. In other words, different sorting dimensions (such as collection time dimension, color dimension, size dimension, etc.) can be displayed in the collection interface for users to choose.

[0121] In some embodiments, when the collection interface of the virtual wardrobe prop includes multiple clothing items including a first clothing item (i.e., the user has collected multiple clothing items), in response to a sliding operation on the first clothing item that is integrated with an uploaded facial image, the following processing can be performed: a second clothing item that is integrated with the uploaded facial image is displayed, wherein the second clothing item is the next clothing item in the collection interface that is located in a reference direction of the first clothing item, and the reference direction is opposite to the direction of the sliding operation.

[0122] For example, taking the first clothing item as clothing A, the collection interface may include other clothing items in addition to clothing A, such as clothing item B, clothing item C, and clothing item D, and the arrangement order of these four clothing items in the collection interface is: clothing item B, clothing item A, clothing item C, and clothing item D. After displaying clothing item A that is integrated with the uploaded facial image, the user may also switch to displaying the fitting effects of other clothing items by swiping left and right. For example, when a left swipe operation triggered by the user on clothing item A that is integrated with the uploaded facial image is received, the display may be switched to clothing item C that is integrated with the uploaded facial image; when a right swipe operation triggered by the user on clothing item A that is integrated with the uploaded facial image is received, the display may be switched to clothing item B that is integrated with the uploaded facial image.

[0123] In other embodiments, when the collection interface of the virtual wardrobe props only includes the first clothing (i.e., the user has only collected the first clothing), in response to a sliding operation on the first clothing that is integrated with the uploaded facial image, the following processing can be performed: obtaining a third clothing (i.e., obtaining a three-dimensional image of the third clothing), wherein the similarity between the third clothing and the first clothing is greater than a similarity threshold; and displaying the third clothing that is integrated with the uploaded facial image.

[0124] For example, taking the first item of clothing as clothing A, assuming that the user has only collected clothing A, when a sliding operation triggered by the user on clothing A that is integrated with the uploaded facial image is received, a third item of clothing (for example, assumed to be clothing E) whose similarity with clothing A is greater than the similarity threshold can be obtained from the network, and then the display of clothing E that is integrated with the uploaded facial image is switched, that is, clothing E similar to clothing A is recommended to the user, which improves the user's search efficiency and enhances the user's fitting experience.

[0125] It should be noted that the above-mentioned third clothing can be clothing worn by the first object, or it can also be clothing not worn by the first object. For example, it can be preferentially determined whether there is clothing worn by the first object whose similarity with the first clothing is greater than a similarity threshold. If so, the clothing is used as the third clothing; if not, clothing whose similarity with the first clothing is greater than the similarity threshold is obtained from clothing not worn by the first object. The embodiment of the present application does not make specific limitations on this.

[0126] The image processing method provided in the embodiment of the present application can obtain a realistic three-dimensional fitting effect by converting a two-dimensional image into a three-dimensional image and integrating the facial image uploaded by the user into the first clothing included in the three-dimensional image. That is, the user can comprehensively observe the fitting effect from multiple perspectives by rotating, thereby improving the user's fitting experience.

[0127] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0128] The embodiment of the present application provides an image processing method, which performs a three-dimensional transformation on the two-dimensional plane map obtained by the search based on the result map of the search for the first object (such as a game character in a game, a virtual character in a film or TV series, or a real person in the real world), creating a museum-style display method. Allow users to immerse themselves in the true appearance of the first object. To satisfy the user's pursuit of the first object and obtain satisfaction. In addition, the user can collect the clothes worn by the first object into a "virtual wardrobe". Then add your own facial image to fit into the clothes of the first object, and intuitively watch the fitting effect.

[0129] In some embodiments, when a user searches for an image of a first object in a browser, the two-dimensional plane image obtained by the search can be converted into a three-dimensional stereoscopic image. Three-dimensional images have high requirements for the quality and quantity of images. The feasibility of implementation in this scenario is relatively high, mainly because the number of images of the first object is large enough and the quality is high, satisfying the conditions for constructing a three-dimensional image of the first object. This helps users to have a more comprehensive understanding of the first object. In addition, for some fan users, the converted three-dimensional images can also be collected. This method refreshes the previous two-dimensional browsing method. Using the three-dimensional interactive method, users can interact with the three-dimensional images and collect the clothes worn by the first object.

[0130] In addition, the embodiment of the present application proposes to use coarse-to-fine three-dimensional modeling, first generating a high-precision basic face model of the first object based on multiple images, then generating corresponding three-dimensional models from each image for other parts of the first object, and finally using the high-precision basic face model to correct the face of a single image, thereby ensuring that the face model is of high-quality restoration.

[0131] The image processing method provided in the embodiment of the present application is described in detail below.

[0132] In some embodiments, see Fig. 6A , Fig. 6A is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Fig. 6A As shown, when the user searches for an image of the first object in the browser, the image of the first object obtained by the search is displayed in the form of a two-dimensional plane map in the search result display area 601, and a 3D mode button 602 is also displayed in the search result display area 601. When the user clicks the 3D mode button 602, the previous two-dimensional plane image is replaced by a three-dimensional stereoscopic image in the search result display area 601. For example, all three-dimensional stereoscopic images of the first object obtained by the search can be displayed in the form of a virtual square.

[0133] It should be noted that, while displaying the 3D stereoscopic image of the first object, a 2D mode button 603 may also be displayed, and the user may use the 2D mode button 603 to convert the image of the first object from the 3D stereoscopic image back into a 2D plane image.

[0134] In other embodiments, see Figure 6B , Figure 6B is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Figure 6BAs shown, in the interactive form of three-dimensional display, the background 604 of the three-dimensional image is taken from the background 605 of the two-dimensional plane image. For example, the background 605 of the two-dimensional plane image can be stretched to obtain the background 604 of the three-dimensional image, thereby making the display of the first object more three-dimensional.

[0135] In some embodiments, the user can also view the three-dimensional image by zooming in or out. For example, when the user zooms in on a three-dimensional image, the zoomed-in three-dimensional image can be rotated 360 degrees to observe it from different perspectives. At the same time, the user can also collect the clothing worn by the first object in the three-dimensional image.

[0136] For example, see Figure 6C , Figure 6C is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Figure 6C As shown, when a user's zoom operation on the three-dimensional stereoscopic image 606 is received, for example, a two-finger spread operation triggered by the user on the three-dimensional stereoscopic image 606 is received, the three-dimensional stereoscopic image 606 is zoomed in, and the zoomed three-dimensional stereoscopic image 606 is displayed in the search result display area. At the same time, a prompt message 607 can also be displayed near the zoomed three-dimensional stereoscopic image 606, wherein the prompt message 607 is used to prompt the user to collect the clothing worn by the first object. For example, when a user clicks on the prompt message 607, the clothing worn by the first object can be collected in the virtual wardrobe. In addition, when a sliding operation triggered by the user on the zoomed three-dimensional stereoscopic image 606 is received, the zoomed three-dimensional stereoscopic image 606 can be controlled to rotate, so as to facilitate the user to observe the first object from different perspectives.

[0137] In some embodiments, the clothes that the user has collected will be displayed on the inner page of the wardrobe (corresponding to the collection interface of the virtual wardrobe props mentioned above), and the user can click the entrance of the virtual wardrobe prop to enter and view all the clothes that he has collected. In addition, the user can also view the clothes from different perspectives by rotating left and right. In addition, after clicking the face image upload button, the user uploads an image or video, and realizes the real experience of the first object's clothing fitting effect by changing the face.

[0138] For example, see Fig.6D , Fig.6D is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Fig.6DAs shown, when a user clicks on the entrance 608 of the virtual wardrobe props, the page can be jumped from the search result display page to the wardrobe inner page 609, and all the clothes collected by the user are displayed in the wardrobe inner page 609. In addition, a face image upload button 610 is also displayed in the wardrobe inner page 609. When a user clicks on the face image upload button 610, a face image 611 acquired in real time can be uploaded, and a new three-dimensional stereoscopic image 612 obtained by fusing the face image 611 uploaded by the user with the clothes worn by the first object can be obtained, so that the user can truly experience the clothes of the first object, thereby improving the user's fitting experience.

[0139] In other embodiments, see Fig. 6E , Fig. 6E is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Fig. 6E As shown, when a sliding operation triggered by the user on the clothing 613 is received, the clothing 613 can be controlled to rotate, so that the user can view the clothing 613 from different perspectives, thereby improving the user experience.

[0140] The following will continue to explain in detail the image processing method provided in the embodiment of the present application from a technical perspective.

[0141] In some embodiments, when a user searches for an image of a first object, a client (e.g., a browser) pulls relevant image data from a server. The client identifies the person and background of the image data returned by the server, extracts multi-perspective images of the first object, and performs basic human body rendering. The single image is then subjected to branched three-dimensional modeling based on the hairstyle and clothing of the first object. The identified background is stretched and adapted using a fixed perspective. Finally, multiple models are synthesized, and the final synthesized three-dimensional face is proofread with the image, for example, proofreading of expressions, eyes, and other parts to ensure that the uniformly rendered face restores the true three-dimensional facial expression in different images.

[0142] For example, see Figure 7 , Figure 7 is a flowchart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 7 The steps shown are explained.

[0143] In step 201, in response to an image search operation for a first object triggered by a user, the client displays a two-dimensional plane image of the first object in a search result display area.

[0144] In step 202, in response to a user clicking operation on a 3D mode button, the client switches to display a three-dimensional stereoscopic image of the first object in a search result display area.

[0145] In some embodiments, before switching to display the three-dimensional stereoscopic image of the first object, the client can identify the face, background, and clothing of the image of the first object sent back by the server, stretch the identified background, perform three-dimensional synthesis on the identified face, and synthesize the stretched background with the three-dimensional synthesized face to obtain a three-dimensional image corresponding to each two-dimensional image.

[0146] In step 203, the client adds the user's favorite clothing to the inner page of the wardrobe in response to the clothing collection operation triggered by the user.

[0147] In step 204, in response to the user clicking the facial image upload button displayed on the inner page of the wardrobe, the client uploads the facial image acquired in real time, and displays a three-dimensional image obtained by fusing the facial image uploaded by the user with the clothing collected by the user.

[0148] In some embodiments, according to the previous process, the embodiments of the present application can synthesize a three-dimensional model for the same person in each image. From the algorithm model, the more image data there is, the higher the accuracy and restoration of the synthesized three-dimensional model. It happens that the image data of the first object is large in magnitude, and this advantage can be used to perform basic face modeling on images from different perspectives. With the basic three-dimensional face, the hairstyle, clothing, etc. on the single image are modeled. At the same time, the basic three-dimensional face needs to be used to proofread the expression of the single image. Finally, the human body, face, hairstyle and clothing are synthesized.

[0149] The following first describes the construction process of the basic three-dimensional face.

[0150] In some embodiments, after the client obtains the search result image of the first object from the server, it can filter images of different perspectives of the first object as basic images for building a three-dimensional face image, with the number being no less than 60. For these basic images, the features of each pixel in the image can be calculated. The principle of the image generation model is that three points form a surface. Here, it is necessary to ensure that each point on the model corresponds to three pixels to generate a surface. Based on the pixel features calculated by the model, the camera intrinsic parameters are used as the constraints of the conversion to convert the pixels of each image into point cloud data, and finally the model is synthesized based on the point cloud data to obtain a complete three-dimensional face information.

[0151] The following will continue to explain the three-dimensional synthesis process of other parts of a single image.

[0152] In some embodiments, the face of the first object can be synthesized in three dimensions with high precision, mainly because the amount of image data is large enough. However, the actions, expressions, hairstyles, and clothing of the first object in a single image are all different. Here it is necessary to achieve three-dimensional restoration in a single image. Human body reconstruction from a single RGB image is extremely challenging, not only because of the need to predict the shape of the invisible part, but also because of the need to restore the geometry of the visible surface. In order to realize the construction of a three-dimensional human body model, the embodiment of the present application refers to DeepHuman, a deep learning-based framework, to solve the above problems. Specifically, in order to provide reasonable initialization for the network and constrain the degrees of freedom of the output space, a parameterized human body model (such as an SMPL model) is used to characterize the shape and posture parameters of the body of the first object in the input image by generating a 3D semantic body and a corresponding 2D semantic map.

[0153] In order to restore surface geometries such as hairstyles or cloth contours with maximum accuracy, the embodiments of the present application refer to a multi-scale volume feature transformation to fuse these image-guided information of different scales into a three-dimensional volume. Finally, a volume normal projection layer is introduced to further refine and enrich visible surface details based on the input image. This layer is designed to connect the body generation network and the normal refinement network, and implement end-to-end training. In other words, the entire modeling process is performed in a coarse-to-fine manner, starting with overall estimation, then entering surface reconstruction, and finally restoring the details of the visible area of ​​the surface.

[0154] In some embodiments, to estimate the body of a first object in an image, embodiments of the present application use HMR and Simplify, both of which can infer shape and pose parameters from a single image. HMR and Simplify have complementary features: HMR has strong predictive power but is not completely consistent with color images, while Simplify aligns the SMPL model with the detected key points very well, but relies on initialization to output reliable results. In order to obtain an accurate estimate of the SMPL image, embodiments of the present application first use HMR to obtain an initial estimate, and then use Simplify to improve its accuracy.

[0155] In other embodiments, in order to estimate the body shape and posture, skin and other parameters of the first object, the embodiments of the present application introduce a dense semantic representation generated from the SMPL model. Specifically, according to the spatial coordinates in the static posture, a semantic code (i.e., a three-dimensional vector for representing the spatial coordinates) is predefined for each vertex on the SMPL model. Given the SMPL model corresponding to the first object, the semantic code is drawn onto the image plane to obtain a semantic map (assuming it is denoted as M s ), and generate a semantic volume (let’s call it V) by first encoding the SMPL model in dense semantics and then uploading the semantic code to the occupied voxel grid. s).

[0156] In some embodiments, for surface geometry reconstruction, the present application embodiment proposes a three-dimensional occupied voxel grid (assuming that it is denoted as V o ), where the voxel values ​​within the surface are set to 1 and the values ​​of other voxels are set to 0. All occupied volumes have a fixed resolution of 128×192×128, where the resolution of the y-axis is set. A 3D human body model usually has one main axis. In order to s With the help of semantic body V s Reconstruction V o , an embodiment of the present application proposes an image-guided volume-to-volume translation network (i.e., the above-mentioned volume-to-volume conversion network), in which multi-scale volume feature transformation is used to fuse two-dimensional image guidance information into a three-dimensional volume.

[0157] In other embodiments, for the identified clothing, the embodiments of the present application may refer to the open source Thuman dataset. Based on the Thuman dataset, dual fusion technology can be used for real-time human mesh reconstruction, and combined with a capture pipeline to quickly and effectively capture the external geometry of a human body wearing casual clothes with medium levels of surface details and textures. Posture parameters can also be added to the modeled clothing model to facilitate subsequent user collection and face replacement.

[0158] It should be noted that the construction from a single image to a three-dimensional model has been completed based on the above steps. The modeling mentioned above is a process from coarse to fine. Single-image modeling still has certain limitations for face construction. The first step has completed the three-dimensional modeling of the first object with high precision. Therefore, the three-dimensional model built based on the high-precision three-dimensional face and the single-image can be replaced and corrected. So far, the synthesis of the full three-dimensional model of a single image has been completed.

[0159] In some embodiments, Figure 6B As shown in the figure, in order to make the three-dimensional image more three-dimensional, the background of the two-dimensional plane image can be processed by curved stretching. Here, we first need to identify the background, and then build a grid for the identified background. Each intersection of the grid lines is the vertex of the triangle. The background image is pasted on the grid in the form of a texture map, and the image will be presented with a three-dimensional sense of curved surface.

[0160] The following is an explanation of the process of fusing the facial image uploaded by the user with the clothing.

[0161] In some embodiments, Figure 8As shown, the technical solution provided by the embodiment of the present application adds the function of collecting the clothing worn by the first object. In order to make the user feel that he is collecting the same clothing, the clothing collected here can be the body part of the first object that hides, such as exposed hands, feet, neck, face, etc. In addition, the parameters of the first object's body will be attached to the voxel grid parameters of the clothing to facilitate the user's subsequent face change. The user uploads his own single face image, multiple images or videos to build a three-dimensional face model. Here, the DeepHuman model can also be used to perform three-dimensional modeling on the face image uploaded by the user. In addition, when constructing a clothing model for a single image of the first object, the voxel grid will mark the head posture parameters. The uploaded face adjusts the angle of the head with the assistance of the voxel grid parameters of the clothing model so that the face obtained by the three-dimensional modeling can be completely integrated with the clothing. At the same time, the limb parameters of the previously collected clothing can also be restored to show a complete human body synthesis.

[0162] In summary, the embodiments of the present application propose a new search image browsing method. With the improvement of AR and VR technology, users can experience immersive consumption of content more personally. When a user browses a three-dimensional stereoscopic image of a first object, users who like the first object can browse related images more closely. Once a three-dimensional way of browsing images is established, users can interact with the first object more immersively. In addition, the technical solution provided by the embodiments of the present application also adds the ability to collect clothing worn by the first object. Users can use the collected clothing to replace faces, which can not only intuitively see the fitting effect of the upper body, but also get an equal satisfaction, thereby improving the user experience.

[0163] The following further describes an exemplary structure of the image processing device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules stored in the image processing device 555 of the memory 550 may include: a display module 5551 .

[0164] Display module 5551 is used to display a two-dimensional image and a three-dimensional conversion control, wherein the two-dimensional image includes a first object wearing a first clothing item; display module 5551 is also used to display a three-dimensional image in response to a trigger operation on the three-dimensional conversion control, wherein the three-dimensional image includes the first clothing item; display module 5551 is also used to display the first clothing item integrated with the uploaded facial image in response to a facial image upload operation.

[0165] In some embodiments, the display module 5551 is further configured to display the first clothing item in the collection interface of the virtual wardrobe props in response to a collection operation on the first clothing item.

[0166] In some embodiments, when the collection interface includes multiple items of clothing including a first item of clothing, the display module 5551 is also used to display a second item of clothing integrated with the uploaded facial image in response to a sliding operation on the first item of clothing integrated with the uploaded facial image, wherein the second item of clothing is the next item of clothing in the collection interface located in a reference direction of the first item of clothing, and the reference direction is opposite to the direction of the sliding operation.

[0167] In some embodiments, the image processing device 555 also includes an acquisition module 5552, which is used to acquire a third clothing item in response to a sliding operation on the first clothing item integrated with the uploaded facial image when the collection interface only includes the first clothing item, wherein the similarity between the third clothing item and the first clothing item is greater than a similarity threshold; the display module 5551 is also used to display the third clothing item integrated with the uploaded facial image.

[0168] In some embodiments, the display module 5551 is also used to respond to a zoom-in operation on the three-dimensional image, display the three-dimensional image based on the zoom-in mode, and display prompt information, wherein the prompt information is used to prompt the collection of the first clothing; the image processing device 555 also includes a determination module 5553, which is used to identify the trigger operation on the prompt information as a collection operation on the first clothing.

[0169] In some embodiments, the display module 5551 is also used to display the entrance of the virtual wardrobe props when displaying the three-dimensional image; the display module 5551 is also used to display the collection interface of the virtual wardrobe props in response to a trigger operation on the entrance of the virtual wardrobe props, wherein the collection interface includes the first clothing and a facial image upload control; and is used to display the first clothing integrated with the uploaded facial image in response to a trigger operation on the facial image upload control.

[0170] In some embodiments, the display module 5551 is further used to display a facial image acquisition area before displaying the first garment fused with the uploaded facial image; the determination module 5553 is further used to use the facial image acquired in real time by the facial image acquisition area as the uploaded facial image; or, the display module 5551 is further used to display a gallery, wherein the gallery includes multiple candidate facial images; the determination module 5553 is further used to respond to a facial image selection operation and use the facial image selected in the gallery as the uploaded facial image.

[0171] In some embodiments, when the collection interface includes multiple clothing items including a first clothing item, the display module 5551 is further used to display the selected first clothing item based on the focus state in response to a clothing item selection operation.

[0172] In some embodiments, the image processing device 555 also includes a control module 5554 for controlling the first clothing to rotate in the collection interface in response to a sliding operation on the first clothing, wherein the rotation angle is a fixed value, or the rotation angle is positively correlated with the sliding distance of the sliding operation.

[0173] In some embodiments, when the collection interface includes multiple items of clothing including a first item of clothing, the multiple items of clothing are displayed in a specific sorting manner, wherein the sorting manner includes: the order of collection time of the multiple items of clothing from late to early; the order of colors of the multiple items of clothing from dark to light; the order of sizes of the multiple items of clothing from large to small.

[0174] In some embodiments, the control module 5554 is further used to control the three-dimensional image to rotate in response to a sliding operation on the three-dimensional image, wherein the rotation angle is a fixed value, or the rotation angle is positively correlated with the sliding distance of the sliding operation.

[0175] In some embodiments, the display module 5551 is further used to display at least one two-dimensional image and a three-dimensional conversion control in the search result display area in response to an image search operation for the first object, wherein the at least one two-dimensional image includes the first object wearing the first clothing.

[0176] In some embodiments, the acquisition module 5552 is further used to acquire multiple search result images for the first object, wherein the multiple search result images are obtained by photographing the first object from different perspectives; the image processing device 555 also includes a generation module 5555, a reconstruction module 5556, a refinement module 5557 and a synthesis module 5558, wherein the generation module 5555 is used to generate a three-dimensional face model of the first object based on the multiple search result images; the acquisition module 5552 is further used to acquire a parameterized human body model; the reconstruction module 5556 is further used to perform surface geometry reconstruction on the parameterized human body model to obtain a reconstructed parameterized human body model; the refinement module 5557 is used to perform refinement processing on the reconstructed parameterized human body model to obtain a three-dimensional human body model of the first object; the synthesis module 5558 is used to synthesize the three-dimensional human body model with the three-dimensional face model to obtain a three-dimensional image.

[0177] In some embodiments, the generation module 5555 is also used to extract features of the pixel points in the face area of ​​the first object in each of the multiple search result images to obtain multiple pixel point features; obtain the camera parameters used when shooting the multiple search result images; convert the multiple pixel point features corresponding to the multiple search result images into point cloud data, wherein the constraints of the conversion are the camera parameters; and generate a three-dimensional face model of the first object based on the point cloud data.

[0178] In some embodiments, the reconstruction module 5556 is further used to initialize a three-dimensional occupied voxel grid, wherein the three-dimensional occupied voxel grid is used to represent the surface of the parameterized human body model, and the voxel values ​​within the surface of the three-dimensional occupied voxel grid are set to 1, and the other voxel values ​​are set to 0; the two-dimensional image and the semantic map are encoded at different scales to obtain a multi-scale two-dimensional feature map; and the multi-scale two-dimensional feature map and the semantic body are fused into the three-dimensional occupied voxel grid through multi-scale volume feature transformation.

[0179] In some embodiments, the determination module 5553 is also used to obtain the spatial coordinates of the first object in a static posture from the two-dimensional image; predefine a semantic code for each vertex on the parameterized human body model according to the spatial coordinates in the static posture; voxelize the parameterized human body model into a voxel grid, and propagate the semantic code to the occupied voxel grid to obtain a semantic body, and draw the semantic code onto the image plane to obtain a semantic map.

[0180] In some embodiments, the refinement module 5557 is also used to obtain a three-dimensional human body model data set; train a network based on the three-dimensional human body model data set, and extract feature information of the first clothing from the two-dimensional image through the trained network; project the three-dimensional occupied voxel grid into a two-dimensional normal map through a U-shaped grid; encode the feature information of the first clothing on the two-dimensional normal map, and paste the encoded two-dimensional normal map on the surface of the reconstructed parameterized human body model.

[0181] In some embodiments, the three-dimensional image includes a background area and a foreground area, wherein the background area includes the background of the stretched two-dimensional image, the foreground area includes a first object wearing a first garment, and the first object is three-dimensional.

[0182] In some embodiments, the generation module 5555 is also used to generate the background of the stretched two-dimensional image in the following manner: identifying the background from the two-dimensional image and constructing a grid corresponding to the background; dividing the background into multiple triangular faces and using the intersection points of the grid lines as the vertices of the triangular faces; and pasting the multiple triangular faces on the grid in the form of a texture map.

[0183] In some embodiments, the generation module 5555 is also used to perform three-dimensional modeling on the uploaded two-dimensional facial image to obtain a three-dimensional facial image; extract posture parameters from the voxel grid included in the clothing model pre-built for the first clothing; adjust the three-dimensional facial image to an angle matching the posture parameters; and fuse the adjusted three-dimensional facial image with the first clothing to obtain the first clothing fused with the uploaded facial image.

[0184] In some embodiments, the display module 5551 is also used to respond to a facial image upload operation, restore a limb image based on limb parameters marked in a voxel grid included in a pre-built clothing model for the first clothing, and display the first clothing that is a fusion of the uploaded facial image and the limb image.

[0185] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 , Figure 4A ,or Figure 4B The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0186] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the computer device executes the image processing method described in the embodiment of the present application.

[0187] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the image processing method provided by the present application embodiment, for example, Figure 3 , Figure 4A ,or Figure 4B The image processing method is shown.

[0188] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0189] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0190] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0191] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Displaying a two-dimensional image and a three-dimensional conversion control, wherein the two-dimensional image includes a first object wearing a first garment; In response to a trigger operation on the three-dimensional conversion control, displaying a three-dimensional image, wherein the three-dimensional image includes the first clothing; In response to the facial image upload operation, the first clothing fused with the uploaded facial image is displayed.

2. The method according to claim 1, characterized in that The method further comprises: In response to a collection operation on the first clothing item, the first clothing item is displayed in a collection interface of a virtual wardrobe prop.

3. The method according to claim 2, characterized in that When the collection interface includes a plurality of garments including the first garment, the method further includes: In response to a sliding operation on the first clothing item integrated with the uploaded facial image, a second clothing item integrated with the uploaded facial image is displayed, wherein the second clothing item is the next clothing item in the collection interface located in a reference direction of the first clothing item, and the reference direction is opposite to the direction of the sliding operation.

4. The method according to claim 2, characterized in that: When the collection interface includes only the first clothing, the method further includes: In response to a sliding operation on the first clothing fused with the uploaded facial image, a third clothing is acquired, wherein the similarity between the third clothing and the first clothing is greater than a similarity threshold; The third clothing fused with the uploaded facial image is displayed.

5. The method according to claim 2, characterized in that: Before responding to the collection operation on the first clothing, the method further includes: In response to a zoom-in operation on the three-dimensional image, displaying the three-dimensional image based on a zoom-in mode and displaying prompt information, wherein the prompt information is used to prompt the user to collect the first clothing; The triggering operation for the prompt information is identified as a collection operation for the first clothing.

6. The method according to any one of claims 1 to 5, characterized in that: When displaying the three-dimensional image, the method further includes: Display the entrance of virtual wardrobe props; The step of displaying the first clothing integrated with the uploaded facial image in response to the facial image uploading operation comprises: In response to a trigger operation on the entrance of the virtual wardrobe prop, displaying a collection interface of the virtual wardrobe prop, wherein the collection interface includes the first clothing and a facial image upload control; In response to a trigger operation on the facial image upload control, the first clothing fused with the uploaded facial image is displayed.

7. The method according to claim 6, characterized in that Before displaying the first garment integrated with the uploaded facial image, the method further includes: Display the facial image acquisition area; Using the facial image acquired in real time through the facial image acquisition area as the uploaded facial image; or, displaying a gallery, wherein the gallery includes a plurality of candidate facial images; In response to the facial image selection operation, the facial image selected in the gallery is used as the facial image to be uploaded.

8. The method according to claim 6 or 7, characterized in that: When the collection interface includes a plurality of garments including the first garment, before responding to a trigger operation on the facial image upload control, the method further includes: In response to the clothing selection operation, the first clothing selected is displayed based on the focus state.

9. The method according to claim 8, characterized in that When displaying the selected first clothing based on the focus state, the method further includes: In response to a sliding operation on the first clothing item, the first clothing item is controlled to rotate in the collection interface, wherein the rotation angle is a fixed value, or the rotation angle is positively correlated with the sliding distance of the sliding operation.

10. The method according to any one of claims 6 to 9, characterized in that: When the collection interface includes multiple garments including the first garment, the multiple garments are displayed in a specific sorting manner, wherein the sorting manner includes: The order of the collection time of the plurality of garments from latest to earliest; The order of the colors of the plurality of clothing items from dark to light; The sizes of the plurality of garments are arranged in descending order.

11. The method according to any one of claims 1 to 10, characterized in that: When displaying the three-dimensional image, the method further includes: In response to a sliding operation on the three-dimensional image, the three-dimensional image is controlled to rotate, wherein the rotation angle is a fixed value, or the rotation angle is positively correlated with the sliding distance of the sliding operation.

12. The method according to any one of claims 1 to 10, characterized in that: The two-dimensional image display and three-dimensional conversion control comprises: In response to an image search operation for the first object, at least one two-dimensional image and a three-dimensional conversion control are displayed in a search result display area, wherein the at least one two-dimensional image includes the first object wearing a first garment.

13. The method according to any one of claims 1 to 10, characterized in that: Before displaying the three-dimensional image, the method further includes: Acquire a plurality of search result images for the first object, wherein the plurality of search result images are obtained by photographing the first object from different perspectives; generating a three-dimensional face model of the first object based on the plurality of search result graphs; Obtaining a parametric human body model; Performing surface geometry reconstruction on the parameterized human body model to obtain the reconstructed parameterized human body model; Performing refinement processing on the reconstructed parameterized human body model to obtain a three-dimensional human body model of the first object; The three-dimensional human body model and the three-dimensional face model are synthesized to obtain the three-dimensional image.

14. The method according to claim 13, characterized in that The step of generating a three-dimensional face model of the first object based on the plurality of search result graphs comprises: For each of the multiple search result images, extracting features of pixel points in the area where the face of the first object is located in the search result image to obtain multiple pixel point features; Obtaining camera parameters used when photographing the plurality of search result images; Converting the plurality of pixel features respectively corresponding to the plurality of search result images into point cloud data, wherein the constraint condition of the conversion is the camera parameter; Based on the point cloud data, a three-dimensional face model of the first object is generated.

15. The method according to claim 13 or 14, characterized in that The surface geometry reconstruction of the parameterized human body model comprises: Initializing a three-dimensional occupied voxel grid, wherein the three-dimensional occupied voxel grid is used to represent the surface of the parameterized human body model, and the voxel values ​​within the surface of the three-dimensional occupied voxel grid are set to 1, and the other voxel values ​​are set to 0; Performing encoding processing at different scales on the two-dimensional image and the semantic map to obtain a multi-scale two-dimensional feature map; The multi-scale two-dimensional feature map and semantic volume are fused into the three-dimensional occupied voxel grid through multi-scale volume feature transformation.

16. The method according to claim 15, characterized in that Before performing surface geometry reconstruction on the parameterized human body model, the method further comprises: Acquire the spatial coordinates of the first object in a static posture from the two-dimensional image; predefine a semantic code for each vertex on the parameterized human body model according to the spatial coordinates in the static posture; voxelizing the parameterized human body model into a voxel grid and propagating the semantic code into the occupied voxel grid to obtain the semantic volume, and The semantic code is drawn onto an image plane to obtain the semantic map.

17. The method according to claim 15 or 16, characterized in that The reconstructed parameterized human body model is refined, including: Acquire a three-dimensional human body model data set; Training a network based on the three-dimensional human body model data set, and extracting feature information of the first clothing from the two-dimensional image through the trained network; Projecting the three-dimensional occupied voxel grid into a two-dimensional normal map via a U-shaped grid; The characteristic information of the first clothing is encoded on the two-dimensional normal map, and the encoded two-dimensional normal map is pasted on the surface of the reconstructed parameterized human body model.

18. The method according to any one of claims 1 to 17, characterized in that The three-dimensional image includes a background area and a foreground area, wherein the background area includes a stretched background of the two-dimensional image, and the foreground area includes the first object wearing the first clothing, and the first object is three-dimensional.

19. The method according to claim 18, characterized in that The method further comprises: The background of the stretched 2D image is generated by: identifying a background from the two-dimensional image and constructing a grid corresponding to the background; Divide the background into a plurality of triangular facets, and use the intersection points of the grid lines as the vertices of the triangular facets; The plurality of triangular facets are pasted onto the mesh in a texture map manner.

20. The method according to any one of claims 1 to 19, characterized in that Before displaying the first garment integrated with the uploaded facial image, the method further includes: Perform three-dimensional modeling on the uploaded two-dimensional facial image to obtain a three-dimensional facial image; extracting pose parameters from a voxel grid included in a pre-built clothing model for the first clothing; Adjusting the three-dimensional facial image to an angle matching the posture parameter; The adjusted three-dimensional facial image is fused with the first clothing to obtain the first clothing fused with the uploaded facial image.

21. The method according to any one of claims 1 to 19, characterized in that The step of displaying the first clothing integrated with the uploaded facial image in response to the facial image uploading operation comprises: In response to the facial image uploading operation, restoring the limbs image based on the limbs parameters marked in the voxel grid included in the pre-built clothing model for the first clothing; The first clothing is displayed with the uploaded facial image and the limbs image integrated therewith.

22. An image processing device, characterized in that: The device comprises: A display module, used for displaying a two-dimensional image and a three-dimensional conversion control, wherein the two-dimensional image includes a first object wearing a first garment; The display module is further configured to display a three-dimensional image in response to a trigger operation on the three-dimensional conversion control, wherein the three-dimensional image includes the first clothing; The display module is further configured to display the first clothing integrated with the uploaded facial image in response to a facial image upload operation.

23. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 21 when executing the executable instructions stored in the memory.

24. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the image processing method described in any one of claims 1 to 21 is implemented.

25. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 21 is implemented.

Images