Image processing method, device, computer equipment and storage medium

Abstract

The present application provides an image processing method, device, computer equipment and storage medium, which relate to the field of image processing. The image processing method includes: obtaining a user image, extracting user key point features from the user image; mapping the user key point features to a preset initial virtual image, generating a first three-dimensional virtual image corresponding to the user; performing multi-view image acquisition on the first three-dimensional virtual image through a virtual image acquisition device matrix, and acquiring a multi-view image for the first three-dimensional virtual image; acquiring a three-dimensional virtual image corresponding to the user based on the multi-view image, and sending the three-dimensional virtual image to a display terminal, so that the display terminal displays the three-dimensional virtual image. Users do not need to wear AR devices to interact with three-dimensional virtual images, which can effectively improve the convenience of interaction and reduce costs.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and the present application relates to an image processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Virtual images are a technology that has been developed on the Internet in recent years. They can reflect the individuality and creativity of users and are deeply loved by netizens. Virtual images can include two-dimensional (2D) virtual images or three-dimensional (3D) virtual images, which can be applied to various scenarios, such as instant messaging, shooting videos, interacting with virtual images, etc.

[0003] Taking the scenario of user interaction with a virtual image as an example, currently, when a user interacts with a three-dimensional virtual image, they need to wear an AR (Augmented Reality) device, resulting in low interaction convenience. Summary of the invention

[0004] In view of the shortcomings of the existing methods, the present application proposes an image processing method, device, electronic device and storage medium to solve the technical problem of low convenience in interacting with a three-dimensional virtual image in the existing technology.

[0005] On the one hand, an embodiment of the present application provides an image processing method, including:

[0006] Acquire a user image, and extract key point features of the user from the user image;

[0007] Mapping the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user;

[0008] Performing multi-perspective image acquisition on the first three-dimensional virtual image through a virtual image acquisition device matrix to obtain multi-perspective images of the first three-dimensional virtual image; wherein the virtual image acquisition device matrix includes a plurality of virtual image acquisition devices with different perspectives;

[0009] A three-dimensional virtual image corresponding to the user is acquired based on the multi-view images, and the three-dimensional virtual image is sent to a display terminal so that the display terminal displays the three-dimensional virtual image.

[0010] In some possible implementations, before extracting the user key point features from the user image, the method further includes:

[0011] Obtaining the type of virtual image to be synthesized selected by the user; wherein the virtual image type includes a half-body image and a full-body image;

[0012] The extracting user key point features from the user image includes:

[0013] User key point features corresponding to the avatar type are extracted from the user image.

[0014] In some possible implementations, before mapping the key point features of the user to a preset initial virtual image, the method further includes:

[0015] extracting user attribute information corresponding to the type of the avatar from the user image;

[0016] Based on the user attribute information, the initial virtual image matching the user is determined from the candidate initial images.

[0017] In some possible implementations, mapping the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user includes:

[0018] Mapping the key point features of the user to the initial virtual image to generate a two-dimensional virtual image corresponding to the user;

[0019] The two-dimensional virtual image is converted into the first three-dimensional virtual image.

[0020] In some possible implementations, acquiring a three-dimensional virtual image corresponding to the user based on the multi-view images includes:

[0021] Identifying a second three-dimensional virtual image of the user from the multi-view image, and segmenting the second three-dimensional virtual image from the multi-view image;

[0022] The three-dimensional virtual image is generated based on the second three-dimensional virtual image and a preset virtual background image.

[0023] In some possible implementations, generating the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image includes:

[0024] calculating the depth of field of the second three-dimensional virtual image to obtain a third three-dimensional virtual image;

[0025] The third three-dimensional virtual image and the virtual background image are synthesized to obtain the three-dimensional virtual image.

[0026] In some possible implementations, before generating the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image, the method further includes:

[0027] receiving the virtual background image selected by a user;

[0028] or,

[0029] The virtual background image corresponding to the initial virtual image is determined.

[0030] On the other hand, an embodiment of the present application provides an image processing device, including:

[0031] An extraction module, used to obtain a user image and extract user key point features from the user image;

[0032] A generating module, used for mapping the key point features of the user to a preset initial virtual image, and generating a first three-dimensional virtual image corresponding to the user;

[0033] A capture module, configured to perform multi-view image capture on the first three-dimensional virtual image through a virtual image capture device matrix to obtain multi-view images of the first three-dimensional virtual image; wherein the virtual image capture device matrix includes a plurality of virtual image capture devices with different viewpoints;

[0034] The display module is used to obtain a three-dimensional virtual image corresponding to the user based on the multi-view images, and send the three-dimensional virtual image to a display terminal so that the display terminal displays the three-dimensional virtual image.

[0035] In some possible implementations, an acquisition module is further included, for acquiring the type of virtual image to be synthesized selected by the user; wherein the virtual image type includes a half-body image and a full-body image;

[0036] When the extraction module extracts the user key point features from the user image, it is specifically used to:

[0037] User key point features corresponding to the avatar type are extracted from the user image.

[0038] In some possible implementations, a determination module is further included, which is used to:

[0039] extracting user attribute information corresponding to the type of the avatar from the user image;

[0040] Based on the user attribute information, the initial virtual image matching the user is determined from the candidate initial images.

[0041] In some possible implementations, when mapping the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, the generation module is specifically configured to:

[0042] Mapping the key point features of the user to the initial virtual image to generate a two-dimensional virtual image corresponding to the user;

[0043] The two-dimensional virtual image is converted into the first three-dimensional virtual image.

[0044] In some possible implementations, when the display module acquires the three-dimensional virtual image corresponding to the user based on the multi-view image, it is specifically used to:

[0045] Identifying a second three-dimensional virtual image of the user from the multi-view image, and segmenting the second three-dimensional virtual image from the multi-view image;

[0046] The three-dimensional virtual image is generated based on the second three-dimensional virtual image and a preset virtual background image.

[0047] In some possible implementations, when the display module generates the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image, it is specifically used to:

[0048] calculating the depth of field of the second three-dimensional virtual image to obtain a third three-dimensional virtual image;

[0049] The third three-dimensional virtual image and the virtual background image are synthesized to obtain the three-dimensional virtual image.

[0050] In some possible implementations, a receiving module is further included, which is used to:

[0051] receiving the virtual background image selected by a user;

[0052] or,

[0053] The virtual background image corresponding to the initial virtual image is determined.

[0054] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-mentioned image processing method.

[0055] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0056] The beneficial effects of the technical solution provided by this application are:

[0057] Multi-perspective image capture is performed on the user's first three-dimensional virtual image through a virtual image capture device matrix to obtain multi-perspective images of the first three-dimensional virtual image, and then a three-dimensional virtual image corresponding to the user is obtained based on the multi-perspective image, so that the display terminal can display the three-dimensional virtual image. The user does not need to wear an AR device to interact with the three-dimensional virtual image, which can effectively improve the convenience of interaction and reduce costs.

[0058] In addition, the user key point features extracted from the user image are mapped to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, and then a virtual three-dimensional image is obtained based on the first three-dimensional virtual image, so that the virtual three-dimensional image that interacts with the user can be customized according to the user's personality, thereby improving the user experience.

[0059] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0061] Figure 1 A schematic diagram of an implementation environment of an image processing method provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a virtual image acquisition device matrix provided in an example of the present application;

[0064] Figure 4 A schematic diagram of a solution for extracting key point features of a user provided as an example of this application;

[0065] Figure 5 A schematic diagram of a solution for extracting key point features of a user provided as an example of this application;

[0066] Figure 6 A schematic diagram of a solution for obtaining a three-dimensional virtual image provided as an example of the present application;

[0067] Figure 7 A schematic diagram of an image processing system provided for an example of the present application;

[0068] Figure 8 A schematic diagram of an image processing scheme provided for an example of the present application;

[0069] Fig. 9A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0072] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" indicates that it is implemented as "A", or is implemented as "A", or is implemented as "A and B".

[0073] The following is a detailed description of the technical solution of the present application and how the technical solution of the present application solves the above technical problems with specific embodiments. It should be noted that the following implementations can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different implementations will not be described repeatedly.

[0074] Figure 1 is a schematic diagram of an implementation environment of an image processing method provided by an embodiment of the present invention, see Figure 1The image acquisition terminal 101 acquires a user image and sends the user image to the server 102. The server 102 extracts the user key point features from the user image. The server 102 maps the user key point features to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user. The server 102 performs multi-view image acquisition on the first three-dimensional virtual image through a virtual image acquisition device matrix to obtain a multi-view image for the first three-dimensional virtual image. The server 102 sends the three-dimensional virtual image to the display terminal 103, and the display terminal 103 displays the three-dimensional virtual image.

[0075] Understandably, Figure 1 This represents an application scenario in an example, and does not limit the application scenario of the image processing method of the present application. In other application scenarios, the display terminal and the image acquisition terminal may be the same terminal.

[0076] It can be understood by technicians in this technical field that the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone (such as Android phones, iOS phones, etc.), a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Devices), a PDA (personal digital assistant), a desktop computer, a smart home appliance, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal, a vehicle-mounted computer, etc.), a smart speaker, a smart watch, etc. The terminal and the server can be directly or indirectly connected by wired or wireless communication, but are not limited to this. The specific can also be determined based on the actual application scenario requirements, and is not limited here.

[0077] Figure 2 The flowchart of an image processing method provided in the embodiment of the present application is shown in FIG. The execution subject of the method may be a server. Figure 2 As shown, the method may include the following steps:

[0078] Step S201: acquiring a user image, and extracting user key point features from the user image.

[0079] The user image may include a user face image, a user half-body image, or a user full-body image.

[0080] Specifically, the image acquisition terminal may acquire the user image and then send the user image to the server.

[0081] During the specific implementation process, the image acquisition terminal can acquire video in the visible area, and if a user's face or body is detected, the user's image is acquired.

[0082] Among them, the user key point features may include the user's facial key points, limb key points and rotation data, and the rotation data may include movement information of the user's body key points during movement.

[0083] Step S202: Mapping the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user.

[0084] The initial virtual image may be a default fixed setting, or may be a virtual image matched by the server that best matches the user image, or may be selected by the user from candidate virtual images.

[0085] Among them, the initial virtual image can be a two-dimensional image, and the first three-dimensional virtual image can be a three-dimensional image. By mapping the user's key point features to the preset initial virtual image, a two-dimensional virtual image corresponding to the user can be obtained, and then the two-dimensional virtual image is converted into a three-dimensional image, that is, the first three-dimensional virtual image is obtained.

[0086] During the specific implementation process, the server can map the key point features of different areas of the user to the initial virtual image one by one. For example, the user's key points in the face area can be mapped to the face area of ​​the initial virtual image, and the user's key points in the upper limb area can be mapped to the upper limb area of ​​the initial virtual image, thereby generating a two-dimensional virtual image corresponding to the user.

[0087] Step S203: performing multi-view image acquisition on the first three-dimensional virtual image through a virtual image acquisition device matrix to obtain multi-view images of the first three-dimensional virtual image.

[0088] Wherein, the virtual image acquisition device matrix includes multiple virtual image acquisition devices with different viewing angles.

[0089] Specifically, there is no limit to the number of virtual image acquisition devices, and there is no limit to the location of the settings, but the virtual image acquisition device matrix needs to be able to capture images of the first three-dimensional virtual image from all angles, that is, it can capture images of the first three-dimensional virtual image at 360 degrees without blind spots.

[0090] like Figure 3As shown, in an example, 9 virtual image acquisition devices with different perspectives can be evenly distributed to form a virtual image acquisition device matrix, i.e., the 9-view camera matrix shown in the figure, to acquire multi-perspective images of the first three-dimensional virtual image, i.e., the multi-perspective images of the 3D image after mapping expressions / actions as shown in the figure.

[0091] Specifically, a virtual image acquisition device matrix may be used to perform multi-view image acquisition on the first three-dimensional virtual image to obtain a video stream for the first three-dimensional virtual image. The multi-view images may be video frame images in the acquired video stream.

[0092] Step S204: acquiring a three-dimensional virtual image corresponding to the user based on the multi-view images, and sending the three-dimensional virtual image to a display terminal so that the display terminal displays the three-dimensional virtual image.

[0093] The three-dimensional virtual image may include a second three-dimensional virtual image corresponding to the user, and may also be provided with a corresponding virtual background image.

[0094] Specifically, the user's second three-dimensional virtual image can be obtained from the multi-view image, and then a three-dimensional virtual image can be synthesized based on the user's second three-dimensional virtual image and the virtual background image. The specific process of obtaining the three-dimensional virtual image will be further elaborated below.

[0095] Each video frame image in the collected video stream is processed to obtain a three-dimensional virtual image, which is equivalent to processing the video stream to obtain a video stream containing the three-dimensional virtual image, so that the user's expression and body movements can be synchronized.

[0096] In the above embodiment, multi-perspective image capture is performed on the user's first three-dimensional virtual image through a virtual image capture device matrix to obtain a multi-perspective image of the first three-dimensional virtual image, and then a three-dimensional virtual image corresponding to the user is obtained based on the multi-perspective image, so that the display terminal can display the three-dimensional virtual image, and the user does not need to wear an AR device to interact with the three-dimensional virtual image, which can effectively improve the convenience of interaction and reduce costs.

[0097] In addition, the user key point features extracted from the user image are mapped to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, and then a virtual three-dimensional image is obtained based on the first three-dimensional virtual image, so that the virtual three-dimensional image that interacts with the user can be customized according to the user's personality, thereby improving the user experience.

[0098] In some possible implementations, before step S201 extracts the user key point features from the user image, the following may also be included:

[0099] Get the type of virtual image to be synthesized selected by the user.

[0100] The virtual image types include half-body images and full-body images.

[0101] During the specific implementation process, the user can select the type of virtual image to be synthesized through touch, click, keyboard input, gesture recognition, etc.

[0102] In a specific implementation process, the key point features of the user may correspond to the type of virtual image selected by the user.

[0103] Step S201 extracts user key point features from the user image, which may include:

[0104] User key point features corresponding to the avatar type are extracted from the user image.

[0105] Specifically, a user image corresponding to the virtual image type selected by the user may be obtained, and then user key point features corresponding to the virtual image type may be extracted from the user image.

[0106] In some embodiments, if the user chooses to synthesize a half-body image, the user's half-body image can be obtained, and then the user's key point features corresponding to the half-body image can be extracted from the half-body image, which may include the user's facial key point features, facial rotation data, etc.

[0107] like Figure 4 As shown, in one example, extracting user key point features may include the following steps:

[0108] The user selects the type of avatar, for example, the user can control the options on the screen of the image acquisition terminal through gesture recognition, confirm the option through gestures such as making a fist, and select a half-body image or a full-body image, that is, a face image or a full-body 3D image;

[0109] The image acquisition terminal acquires the video stream of the set area in real time. If the user chooses to acquire a half-body image, i.e., a 3D image of a face, head detection is performed. If a head is detected in the video stream of the set area, a temporary head identification, i.e., a head ID (Identity document) is generated, and the head ID is tracked to see if it appears in the picture within N seconds.

[0110] If the head ID appears in the picture within N seconds, face detection is performed;

[0111] If no face is detected, the head ID is abandoned and the step of tracking the head ID is returned; if a face is detected, the largest face in the picture is selected to generate a face ID, that is, the face ID shown in the figure;

[0112] If the face ID and the tracked head ID are the same person, the user key point features are extracted from the face, that is, the extracted expression shown in the figure, including the face key points and rotation data;

[0113] If the face ID and the tracked head ID are not the same person, then based on the face ID and the tracked head ID, it is determined whether a new face ID has entered the screen and a tracking ID is determined;

[0114] Determine whether the face corresponding to the tracking ID continues to appear in the picture. If so, enter the step of extracting the user's key point features from the face; if not, return to the step of tracking the head ID.

[0115] In some embodiments, if the user chooses to synthesize a full-body image, the user's full-body image is obtained, and then the user's key point features corresponding to the full-body image are extracted from the full-body image, which may include the user's facial key point features, body key point features, facial rotation data, and body rotation data.

[0116] Among them, the key point features of the human body may include the user's limb features and body features, such as fatness, body shape, height, etc.

[0117] like Figure 5 As shown, in one example, extracting user key point features may include the following steps:

[0118] The user selects the avatar type;

[0119] The image acquisition terminal collects the video stream of the set area in real time. If the user chooses to collect the full body image, human body detection is performed;

[0120] If a human body is detected in the video stream of the set area, the user with the largest human body recognition frame in the picture is selected to generate a human body ID;

[0121] Determine whether the user corresponding to the human ID is out of the screen;

[0122] If yes, go back and re-test the human body;

[0123] If the user remains in the picture within N seconds, the key point features of the user are extracted, that is, the key points and rotation data of the human body shown in the figure;

[0124] The user's key point features are input into the business system, so as to map the user's key point features to the preset initial virtual image, and then perform the step of generating the final three-dimensional virtual image.

[0125] The process of determining the initial virtual image will be described below in conjunction with embodiments.

[0126] In some possible implementations, the initial virtual image may be a default one, may be selected by the user, or may be determined based on the user attribute information of the user.

[0127] For example, users can switch 3D images through specific gestures, such as the OK gesture. After the system recognizes the gesture, a list of preset 3D images and a button to generate an image will pop up. If the user selects a half-body image, a list of face 3D images will pop up, and a button to generate a face 3D image will pop up; if the user selects a full-body image, a list of full-body 3D images will pop up, and a button to generate a full-body 3D image will pop up. In this state, the user can slide the palm left and right to select an initial virtual image from the preset images, or provide an initial virtual image option; if the user selects an initial virtual image from the candidate virtual images, the initial virtual image and its supporting background will be enlarged and displayed in full screen.

[0128] In some possible implementations, before step S202 maps the user key point features to a preset initial virtual image, the following may also be included:

[0129] (1) extracting user attribute information corresponding to the type of the virtual image from the user image;

[0130] (2) Determining the initial virtual image that matches the user from the candidate initial images based on the user attribute information.

[0131] The user attribute information may include half-body attribute information and full-body attribute information of the user.

[0132] If the virtual image type selected by the user is a half-body image, the corresponding user attribute information may include attribute information such as gender, age group, skin color, hairstyle, etc.; if the virtual image type selected by the user is a full-body image, the corresponding user attribute information may include attribute information such as gender, skin color, hairstyle, height, fatness, top type, top color, bottom type, bottom color, etc.

[0133] Specifically, the server selects an initial virtual image that matches the user from candidate initial images according to the user attribute information.

[0134] If the virtual image type selected by the user is a half-body image, the candidate initial image that is closest to the user in terms of gender, age, skin color, hairstyle, etc. can be selected as the initial virtual image; if the virtual image type selected by the user is a full-body image, the candidate initial image that is closest to the user in terms of gender, age, skin color, hairstyle, height, body shape, etc. can be selected as the initial virtual image.

[0135] The above embodiment describes the specific process of determining the initial virtual image. The following will further describe the process of generating the first three-dimensional virtual image in conjunction with the embodiment.

[0136] In some possible implementations, step S202 maps the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, which may include:

[0137] (1) Mapping the key point features of the user to the initial virtual image to generate a two-dimensional virtual image corresponding to the user;

[0138] (2) Converting the two-dimensional virtual image into the first three-dimensional virtual image.

[0139] Specifically, the initial virtual image may be a two-dimensional image, and key point features of different areas of the user may be mapped one-to-one to the initial virtual image, that is, the key point features cover the initial features of the corresponding areas in the initial virtual image to obtain a two-dimensional virtual image.

[0140] In a specific implementation process, left-eye image data and right-eye image data may be collected separately for the two-dimensional virtual image, and the first three-dimensional virtual image may be synthesized according to the left-eye image data and the right-eye image data.

[0141] The above embodiment describes the specific process of determining the first three-dimensional virtual image. The following will further describe the process of generating a three-dimensional virtual image in conjunction with the embodiment.

[0142] In some possible implementations, step S204 of acquiring a three-dimensional virtual image corresponding to the user based on the multi-view images may include:

[0143] (1) identifying a second three-dimensional virtual image of the user from the multi-view image, and segmenting the second three-dimensional virtual image from the multi-view image;

[0144] (2) Generating the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image.

[0145] Specifically, a recognition algorithm may be used to recognize the second three-dimensional virtual image in the multi-view image, and then segment the second three-dimensional virtual image, that is, perform a cutout process.

[0146] Specifically, generating the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image may include:

[0147] a. calculating the depth of field (DOF) of the second three-dimensional virtual image to obtain a third three-dimensional virtual image;

[0148] b. Synthesize the third three-dimensional virtual image and the virtual background image to obtain the three-dimensional virtual image.

[0149] like Figure 6 As shown, for each video frame image in the multi-view video stream captured by the virtual image acquisition device matrix, that is, the multi-view image, an algorithm is used to identify the target, that is, the second three-dimensional virtual image, and the second three-dimensional virtual image is cut out, that is, the second three-dimensional virtual image is segmented from the multi-view image, and the depth of field is recalculated to obtain a third three-dimensional virtual image, that is, new data, and the third three-dimensional virtual image and the virtual background image are synthesized, that is, the new data is synthesized with the original data to generate a three-dimensional virtual image.

[0150] In some possible implementations, before generating the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image, the method may further include:

[0151] receiving the virtual background image selected by a user;

[0152] or,

[0153] The virtual background image corresponding to the initial virtual image is determined.

[0154] Specifically, the virtual background image can be selected by the user, or it can be determined according to the initial virtual image when determining the initial virtual image that matches the user. Different virtual background images can be set according to different initial virtual images, or the virtual background image can be determined according to the user's choice, which can enrich the diversity of the finally generated three-dimensional virtual image.

[0155] In order to more clearly explain the image processing method of the present application, the image processing method of the present application will be further explained in detail below in conjunction with an image processing system.

[0156] like Figure 7 As shown, in one example, the image processing system of the present application may include:

[0157] Hardware acquisition terminal: used to acquire user images, that is, to acquire the video stream of the visible area video to obtain the user image;

[0158] The AI ​​(Artificial Intelligence) computing server is used to extract frames from the video stream collected by the hardware acquisition terminal, perform face detection or body detection, and generate a tracking ID; and extract the user's key point features in the user image, that is, the expression recognition shown in the figure (extracting dense key points of the face) and extracting face rotation data; it is also used to recognize the user's gestures to determine the type of virtual image selected by the user, that is, the simulation category;

[0159] Business server: used to map the key point features of the user extracted by the AI ​​computing server to the initial virtual image, that is, to the target model, to generate a two-dimensional virtual image corresponding to the user; and to convert the two-dimensional virtual image into the first three-dimensional virtual image, that is, the two-dimensional to three-dimensional conversion shown in the figure, to generate the first three-dimensional virtual image; and can also interact with the user, receive the virtual image type selected by the user, that is, the simulation category, and send the virtual image type selected by the user to the hardware acquisition terminal, so that the hardware acquisition terminal acquires the corresponding video stream data;

[0160] The 3D display terminal is used to receive the first three-dimensional virtual image generated by the business service terminal, perform multi-view image acquisition on the first three-dimensional virtual image through a virtual image acquisition device matrix, obtain multi-view images for the first three-dimensional virtual image, and display them, i.e., the 9-view display imaging shown in the figure.

[0161] like Figure 8 As shown, in the above example, the image processing method may include the following steps:

[0162] The user selects the type of avatar, i.e., half-body / full-body 3D avatar as shown in the figure;

[0163] If the user selects a half-body image, then key features of the user corresponding to the image are collected, that is, facial expressions and facial features are collected;

[0164] Extracting expression information and facial features, mapping key point features of the user to the initial virtual image, and generating a first three-dimensional virtual image corresponding to the user; that is, generating a half-body 3D image as shown in the figure;

[0165] If the user selects a full-body image, the key features of the user corresponding to the full-body image are collected, that is, facial expressions, facial features, body movements, and body features;

[0166] Extracting human motion information, facial features, and human attribute features, mapping the key point features of the user to the initial virtual image, and generating a first three-dimensional virtual image corresponding to the user; that is, generating a full-body 3D image as shown in the figure;

[0167] Using a virtual image acquisition device matrix to perform multi-view image acquisition on the first three-dimensional virtual image, obtaining multi-view images of the first three-dimensional virtual image, that is, automatically loading the model into a preset three-dimensional software camera matrix as shown in the figure;

[0168] Acquire a three-dimensional virtual image corresponding to the user based on the multi-view images, that is, perform automated three-dimensional processing through an algorithm as shown in the figure;

[0169] The three-dimensional virtual image is sent to a display terminal so that the display terminal displays the three-dimensional virtual image; that is, the synchronous expressions and actions of the 3D image generated by the display as shown in the figure.

[0170] The above-mentioned image processing method performs multi-perspective image acquisition on the user's first three-dimensional virtual image through a virtual image acquisition device matrix to obtain multi-perspective images for the first three-dimensional virtual image, and then obtains a three-dimensional virtual image corresponding to the user based on the multi-perspective image, so that the display terminal can display the three-dimensional virtual image, and the user does not need to wear an AR device to interact with the three-dimensional virtual image, which can effectively improve the convenience of interaction and reduce costs.

[0171] Furthermore, the user key point features extracted from the user image are mapped to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, and then a virtual three-dimensional image is obtained based on the first three-dimensional virtual image, so that the virtual three-dimensional image that interacts with the user can be customized according to the user's personality, thereby improving the user experience.

[0172] Fig. 9 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0173] An extraction module 901 is used to obtain a user image and extract key point features of the user from the user image;

[0174] A generating module 902, configured to map the key point features of the user to a preset initial virtual image, and generate a first three-dimensional virtual image corresponding to the user;

[0175] The acquisition module 903 is used to acquire multi-view images of the first three-dimensional virtual image through a virtual image acquisition device matrix to obtain multi-view images of the first three-dimensional virtual image; wherein the virtual image acquisition device matrix includes a plurality of virtual image acquisition devices with different viewpoints;

[0176] The display module 904 is configured to acquire a three-dimensional virtual image corresponding to the user based on the multi-view images, and send the three-dimensional virtual image to a display terminal so that the display terminal displays the three-dimensional virtual image.

[0177] In some possible implementations, an acquisition module is further included, for acquiring the type of virtual image to be synthesized selected by the user; wherein the virtual image type includes a half-body image and a full-body image;

[0178] When extracting the user key point features from the user image, the extraction module 901 is specifically used to:

[0179] User key point features corresponding to the avatar type are extracted from the user image.

[0180] In some possible implementations, a determination module is further included, which is used to:

[0181] extracting user attribute information corresponding to the type of the avatar from the user image;

[0182] The initial virtual image matching the user is determined from the candidate initial images based on the user attribute information.

[0183] In some possible implementations, when mapping the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, the generating module 902 is specifically configured to:

[0184] Mapping the key point features of the user to the initial virtual image to generate a two-dimensional virtual image corresponding to the user;

[0185] The two-dimensional virtual image is converted into the first three-dimensional virtual image.

[0186] In some possible implementations, when the display module 904 acquires the three-dimensional virtual image corresponding to the user based on the multi-view image, it is specifically configured to:

[0187] Identifying a second three-dimensional virtual image of the user from the multi-view image, and segmenting the second three-dimensional virtual image from the multi-view image;

[0188] The three-dimensional virtual image is generated based on the second three-dimensional virtual image and a preset virtual background image.

[0189] In some possible implementations, when the display module 904 generates the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image, it is specifically used to:

[0190] calculating the depth of field of the second three-dimensional virtual image to obtain a third three-dimensional virtual image;

[0191] The third three-dimensional virtual image and the virtual background image are synthesized to obtain the three-dimensional virtual image.

[0192] In some possible implementations, a receiving module is further included, which is used to:

[0193] receiving the virtual background image selected by a user;

[0194] or,

[0195] The virtual background image corresponding to the initial virtual image is determined.

[0196] The above-mentioned image processing device performs multi-perspective image acquisition on the user's first three-dimensional virtual image through a virtual image acquisition device matrix, obtains multi-perspective images for the first three-dimensional virtual image, and then obtains a three-dimensional virtual image corresponding to the user based on the multi-perspective image, so that the display terminal can display the three-dimensional virtual image. The user does not need to wear an AR device to interact with the three-dimensional virtual image, which can effectively improve the convenience of interaction and reduce costs.

[0197] Furthermore, the user key point features extracted from the user image are mapped to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, and then a virtual three-dimensional image is obtained based on the first three-dimensional virtual image, so that the virtual three-dimensional image that interacts with the user can be customized according to the user's personality, thereby improving the user experience.

[0198] Fig.10 1 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10 As shown, the computer device includes: a memory and a processor; at least one program stored in the memory, which, when executed by the processor, can achieve the following compared with the prior art:

[0199] Through the virtual image acquisition device matrix, multi-view image acquisition is performed on the user's first three-dimensional virtual image, and a multi-view image for the first three-dimensional virtual image is obtained. Then, based on the multi-view image, a three-dimensional virtual image corresponding to the user is obtained, so that the display terminal can display the three-dimensional virtual image, and the user does not need to wear an AR device to interact with the three-dimensional virtual image, which can effectively improve the convenience of interaction and reduce costs. The user's key point features extracted from the user image are mapped to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, and then a virtual three-dimensional image is obtained based on the first three-dimensional virtual image, so that the virtual three-dimensional image that interacts with the user can be customized according to the user's personalization, thereby improving the user experience.

[0200] In an alternative embodiment, a computer device is provided, such as Fig.10 As shown, Fig.10 The computer device 1000 shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the computer device 1000 may also include a transceiver 1004, which may be used for data interaction between the computer device and other computer devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 1004 is not limited to one, and the structure of the computer device 1000 does not constitute a limitation on the embodiments of the present application.

[0201] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0202] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0203] The memory 1003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0204] The memory 1003 is used to store the application code (computer program) for executing the solution of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement the contents shown in the above method embodiment.

[0205] The computer equipment includes but is not limited to virtualized computer equipment, virtual machines, servers, service clusters, user terminals, etc.

[0206] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the image processing method in the aforementioned method embodiment.

[0207] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned image processing method.

[0208] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0209] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0210] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0211] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0212] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0213] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0214] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, an extraction module may also be described as a "module for extracting key point features of a user".

[0215] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. An image processing method, characterized in that: include: Acquire a user image, and extract user key point features from the user image; wherein the user key point features include at least one of the following: face key point features, body key point features, face rotation data, and body rotation data; the body key point features include at least one of the following: limb features and body features of the user; Mapping the key point features of the user to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user; Performing multi-perspective image acquisition on the first three-dimensional virtual image through a virtual image acquisition device matrix to obtain multi-perspective images of the first three-dimensional virtual image; wherein the virtual image acquisition device matrix includes a plurality of virtual image acquisition devices with different perspectives; Acquire a three-dimensional virtual image corresponding to the user based on the multi-view images, and send the three-dimensional virtual image to a display terminal so that the display terminal displays the three-dimensional virtual image; The mapping of the key point features of the user into a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user includes: mapping the key point features of the user into the initial virtual image to generate a two-dimensional virtual image corresponding to the user; and converting the two-dimensional virtual image into the first three-dimensional virtual image.

2. The image processing method according to claim 1, characterized in that: Before extracting the user key point features from the user image, the method further includes: Obtaining the type of virtual image to be synthesized selected by the user; wherein the virtual image type includes a half-body image and a full-body image; The extracting user key point features from the user image includes: User key point features corresponding to the avatar type are extracted from the user image.

3. The image processing method according to claim 2, characterized in that: Before mapping the key point features of the user to the preset initial virtual image, the method further includes: extracting user attribute information corresponding to the type of the avatar from the user image; Based on the user attribute information, the initial virtual image matching the user is determined from the candidate initial images.

4. The image processing method according to claim 1, characterized in that: The acquiring of a three-dimensional virtual image corresponding to the user based on the multi-view images comprises: Identifying a second three-dimensional virtual image of the user from the multi-view image, and segmenting the second three-dimensional virtual image from the multi-view image; The three-dimensional virtual image is generated based on the second three-dimensional virtual image and a preset virtual background image.

5. The image processing method according to claim 4, characterized in that: The generating the three-dimensional virtual image based on the second three-dimensional virtual image and a preset virtual background image includes: calculating the depth of field of the second three-dimensional virtual image to obtain a third three-dimensional virtual image; The third three-dimensional virtual image and the virtual background image are synthesized to obtain the three-dimensional virtual image.

6. The image processing method according to claim 4 or 5, characterized in that: Before generating the three-dimensional virtual image based on the second three-dimensional virtual image and the preset virtual background image, the method further includes: receiving the virtual background image selected by a user; or, The virtual background image corresponding to the initial virtual image is determined.

7. An image processing device, characterized in that: The device comprises: An extraction module is used to obtain a user image and extract user key point features from the user image; wherein the user key point features include at least one of the following: face key point features, body key point features, face rotation data and body rotation data; the body key point features include at least one of the following: limb features and body features of the user; A generating module, used for mapping the key point features of the user to a preset initial virtual image, and generating a first three-dimensional virtual image corresponding to the user; A capture module, configured to perform multi-view image capture on the first three-dimensional virtual image through a virtual image capture device matrix to obtain multi-view images of the first three-dimensional virtual image; wherein the virtual image capture device matrix includes a plurality of virtual image capture devices with different viewpoints; A display module, configured to acquire a three-dimensional virtual image corresponding to the user based on the multi-view images, and send the three-dimensional virtual image to a display terminal so that the display terminal displays the three-dimensional virtual image; Among them, when the generation module maps the user's key point features to a preset initial virtual image to generate a first three-dimensional virtual image corresponding to the user, it is specifically used to: map the user's key point features to the initial virtual image to generate a two-dimensional virtual image corresponding to the user; and convert the two-dimensional virtual image into the first three-dimensional virtual image.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images