Eyeball data processing method, device, computer equipment and storage medium

Abstract

The embodiments of the present application disclose an eyeball data processing method, device, computer equipment and storage medium, which can be applied to cloud technology, computer vision, smart transportation, assisted driving and other scenarios. The method includes: obtaining a first key point set corresponding to the eye socket of the first virtual face from the point cloud of the first virtual face; obtaining a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball; based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, determine a first adjustment parameter, the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face. Adaptive adjustment of the point cloud of the virtual eyeball using the embodiments of the present application can make the virtual eyeball more matched with the eye socket of the virtual face, and improve the matching efficiency of the eyeball and the eye socket.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an eyeball data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the continuous development of network technology, virtual human technology has been widely used in the reconstruction of human faces in 3D games, 3D film and television works, short videos and other fields. However, in virtual human technology, the reconstructed face does not include eyeballs, so the authenticity of the virtual human needs to be improved. In order to better improve the authenticity of the virtual human and generate more beautiful results, it is necessary to hang the virtual eyeball on the reconstructed 3D face. That is, for a 3D model of a virtual face and a 3D model of a virtual eyeball, it is necessary to achieve the matching of the virtual eyeball and the eye socket of the virtual face.

[0003] At present, the method to achieve the matching of virtual eyeballs and eye sockets of virtual faces is mainly to use the 3D coordinates of the corners of the eyes and the upper and lower eye sockets of the virtual face to approximately estimate the position of the virtual eyeball on the x and y coordinate axes, and then continuously adjust the value in the z direction until the phenomenon of penetration occurs. However, this method is inefficient by constantly trying to adjust the value in the z direction, and the fit between the virtual eyeball and the eye socket of the virtual face is poor, which reduces the authenticity of the virtual person. Summary of the invention

[0004] The embodiments of the present application provide an eyeball data processing method, apparatus, computer equipment and storage medium, which can quickly and accurately match virtual eyeballs with eye sockets of virtual faces, thereby improving the authenticity of virtual humans.

[0005] In a first aspect, an embodiment of the present application provides an eyeball data processing method, the method comprising:

[0006] Acquire a first set of key points corresponding to eye sockets of the first virtual face from the point cloud of the first virtual face;

[0007] Acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball;

[0008] Based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, a first adjustment parameter is determined, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face.

[0009] In a second aspect, an embodiment of the present application provides an eyeball data processing device, the device comprising:

[0010] An acquisition module, configured to acquire a first set of key points corresponding to eye sockets of the first virtual face from a point cloud of the first virtual face;

[0011] The acquisition module is further used to acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball;

[0012] A processing module, used to determine a first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, wherein the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face.

[0013] In a third aspect, an embodiment of the present application provides a computer device, which includes a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores a computer program, and the processor is used to call the computer program to execute the eye data processing method provided in the embodiment of the present application.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the eye data processing method provided in the embodiment of the present application is implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the eyeball data processing method provided in an embodiment of the present application.

[0016] In the embodiment of the present application, the computer device obtains a first key point set corresponding to the eye socket of the first virtual face from the point cloud of the first virtual face; obtains a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball; and determines a first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face. By using the embodiment of the present application, the point cloud of the virtual eyeball is adaptively adjusted according to the adjustment parameter, which not only improves the matching efficiency of the eyeball and the eye socket, but also makes the virtual eyeball more matched with the eye socket of the virtual face, thereby improving the authenticity of the virtual person. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic diagram of an eyeball data processing solution provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an eyeball data processing method provided in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a first virtual human face provided in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a virtual eyeball provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the matching effect of the eyeball and the eye socket provided in an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an eyeball data processing device provided in an embodiment of the present application;

[0024] Figure 7 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In order to facilitate understanding of the embodiments of the present application, the eye data processing method of the present application is described below.

[0027] In order to improve the authenticity of virtual people, this application embodiment provides an eyeball data processing solution. Figure 1 , Figure 1 is a schematic diagram of an eyeball data processing solution provided in an embodiment of the present application. Figure 1The general implementation process of the eyeball data processing scheme proposed in the embodiment of the present application is described as follows: first, the computer device 101 obtains the first virtual face 102 and the virtual eyeball 103, and in response to the user's operation instructions for the virtual eyeball 103 and the first virtual face 102, the virtual eyeball 103 is mounted on the eye socket of the first virtual face 102; secondly, the computer device performs point sampling around the eye socket of the first virtual face 102, obtains a first key point set corresponding to the eye socket of the first virtual face 102 from the point cloud of the first virtual face 102 (i.e., the key point set on the eye socket, or referred to as target), and obtains a first key point set corresponding to the eye socket of the first virtual face 102 from the point cloud of the first virtual face 102 Then, the computer device 101 determines the adjustment parameters such as the rotation amount R, the translation amount t and the scaling amount s of the point cloud of the virtual eyeball 103 transformed to the point cloud of the eye socket of the first virtual face based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face 102 and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball 103; finally, the point cloud points included in the point cloud of the virtual eyeball 103 are adjusted according to R, t and s to align the point cloud of the virtual eyeball 103 with the point cloud of the eye socket of the first virtual face 102.

[0028] Practice has shown that the eyeball data processing scheme proposed in the embodiment of the present application can have the following beneficial effects: ① By obtaining the first key point set corresponding to the eye socket of the first virtual face and the second key point set corresponding to the first key point set in the virtual eyeball, and based on the coordinates of each key point in the first key point set and the coordinates of each key point in the second key point set, the first adjustment parameter is determined, so that the virtual eyeball at any position can be perfectly combined with the eye socket of the virtual face at any position according to the first adjustment parameter, and there will be no problem of model penetration, thereby improving the authenticity of the virtual person and providing a better user experience. ② Since the first adjustment parameter includes one or more of the rotation amount, translation amount and scaling amount, the scheme can cope with the situation where the eye sockets of different virtual people are of different sizes, and the virtual eyeball can be perfectly fitted with the eye socket of the virtual person based on the scaling amount. ③ The matching of the virtual eyeball and the eye socket of the virtual face can be completed more efficiently and quickly, generating a more beautiful result, making it easier to use in the corresponding products and improving the user experience of the products.

[0029] It should be noted that: in the specific implementation, the above scheme can be executed by a computer device, which can be a terminal or a server; wherein the terminal mentioned here can include but is not limited to: smart phones, tablet computers, laptops, desktop computers, smart watches, smart TVs, smart car terminals, etc.; various clients (Application, APP) can be run in the terminal, such as video playback clients, social clients, browser clients, information flow clients, education clients, etc. The server mentioned here can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms, etc. Furthermore, the computer device mentioned in the embodiments of the present application may be located outside or inside the blockchain network, without limitation. The so-called blockchain network is a network composed of a peer-to-peer network (P2P network) and a blockchain, and the blockchain refers to a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. It is essentially a decentralized database, a string of data blocks (or blocks) generated by cryptographic methods.

[0030] The eyeball data processing method provided in the embodiment of the present application can be implemented based on artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology and application system of simulating, extending and expanding human intelligence, perceiving the environment, acquiring knowledge and using knowledge to obtain the best results using digital computers or machines controlled by digital computers. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, which involves a wide range of fields, including both hardware-level technology and software-level technology. AI basic technology generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other major directions.

[0031] The eyeball data processing method provided in the embodiment of the present application mainly relates to computer vision technology (Computer Vision, CV) in AI technology. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, follow and measure targets, and further perform graphic processing to make computer processing more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and map construction and other technologies.

[0032] It should be noted that the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0033] Based on the above solution, this application embodiment proposes an eyeball data processing method, see Figure 2 , Figure 2 is a flow chart of an eyeball data processing method provided in an embodiment of the present application. The method is executed by a computer device, such as Figure 2 As shown, the eyeball data processing method may include the following steps S201-S203:

[0034] S201. Acquire a first set of key points corresponding to eye sockets of the first virtual face from the point cloud of the first virtual face.

[0035] Point cloud refers to a massive set of points of target surface characteristics, and the point cloud of the first virtual face is a set of all point cloud points on the surface of the first virtual face.

[0036] The first key point set is a set of key points corresponding to the eye sockets of the first virtual face. Among them, the key points are combined with the local feature descriptors to form a key point descriptor, which is often used to represent the original data and is representative and descriptive. Using key points can speed up the subsequent data processing.

[0037] In an optional embodiment, the computer device obtains a first set of key points corresponding to the eye sockets of the first virtual face from the point cloud of the first virtual face, including: obtaining identification information of the key points corresponding to the eye sockets of the first virtual face from a target topological structure corresponding to the first virtual face, the target topological structure including identification information of the key points corresponding to each part of the first virtual face; based on the identification information of the key points corresponding to the eye sockets, obtaining a first set of key points corresponding to the eye sockets from the point cloud of the first virtual face, the first set of key points including the identification information of the key points corresponding to the eye sockets.

[0038] It is understandable that in 3D modeling, topology refers to the point-line-surface layout, structure, and connection of the polygonal mesh model. The above-mentioned target topology structure refers to the point-line-surface layout, structure, and connection of the first virtual face model, which indicates the identification information of the key points corresponding to each part of the virtual face (such as the first virtual face, the second virtual face, etc.), such as the identification information of the eye socket key points, the identification information of the nose key points, the identification information of the mouth key points, etc.

[0039] Taking the first virtual face as an example, assume that the target topological structure corresponding to the first virtual face includes 100 key points, denoted as a1, a2, a3, ..., a 100 , where 1, 2, ..., 100 are identification information of each key point (or called the key point subscript). The computer device can obtain the subscripts (e.g., 51, 52, ..., 70) of N (e.g., 20) key points corresponding to the eye sockets of the first virtual face from the 100 key points; based on the subscripts of the 20 key points corresponding to the eye sockets, the set of the 20 key points corresponding to the eye sockets can be obtained from the point cloud of the first virtual face, and the set includes the subscripts of the 20 key points corresponding to the eye sockets, which are recorded as index-T. See Figure 3 , Figure 3 A schematic diagram of a first virtual face provided in an embodiment of the present application. Figure 3 In the figure, the black dots are the key points of the eye sockets of the first virtual face.

[0040] In an optional embodiment, the computer device may obtain a first set of key points corresponding to the eye sockets of the first virtual face from the point cloud of the first virtual face through a point cloud key point extraction algorithm. The point cloud key point extraction algorithm includes but is not limited to a scale-invariant feature transform (SIFT) algorithm, a Harris key point extraction algorithm, an internal shape descriptor (ISS) algorithm, etc.

[0041] S202: Acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball.

[0042] The second key point set is a key point set corresponding to the virtual eyeball, wherein the key points corresponding to the virtual eyeball correspond to the key points corresponding to the eye sockets of the first virtual face.

[0043] Optionally, before obtaining the second key point set corresponding to the first key point set from the point cloud of the virtual eyeball, the user may manually mount the virtual eyeball on the eye socket of the first virtual face. Optionally, the computer device may obtain the initial position of the virtual eyeball in the eye socket of the first virtual face and store the initial position.

[0044] In an optional embodiment, the computer device obtains the second key point set corresponding to the first key point set from the point cloud of the virtual eyeball, including: obtaining at least one vertex included in the point cloud of the virtual eyeball; and determining the second key point set corresponding to the first key point set from at least one vertex based on the distance between the at least one vertex and the key points included in the first key point set.

[0045] In an optional embodiment, the computer device determines the second key point set corresponding to the first key point set from at least one vertex according to the distance between at least one vertex and the key points included in the first key point set, including: for any key point in the first key point set, based on the coordinates of each vertex in at least one vertex and the coordinates of any key point, determining the distance between each vertex and any key point; obtaining the target vertex with the smallest distance to any key point; using the target vertex as the key point corresponding to any key point in the second point cloud, and adding the target vertex to the second key point set corresponding to the first key point set.

[0046] Optionally, the computer device may use a K-Nearest Neighbor (KNN) algorithm to find a second key point set corresponding to the N orbital key points in the point cloud of the virtual eyeball, which is recorded as a set T.

[0047] Among them, the process of using the KNN algorithm to find the key points of the virtual eyeball corresponding to the key points of the eye socket is as follows:

[0048]

[0049]

[0050] The computer device can obtain the index set of key points in the point cloud of the virtual eyeball corresponding to the eye socket key points through the above method, which is recorded as index-S. Figure 4 , Figure 4 A schematic diagram of a virtual eye key point provided in an embodiment of the present application. Figure 4 In the figure, the black dots are the key points of the virtual eyeball.

[0051] S203. Determine a first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, wherein the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face.

[0052] In an optional implementation, the computer device may obtain the coordinates of the key points corresponding to the virtual eyeball from the point cloud of the virtual eyeball according to index-S, and use them to represent the 3D coordinates of the initial eyeball (a j , b j 、c j ), denoted as Source; according to index-S, the coordinates of the key points corresponding to the eye sockets are obtained from the point cloud of the first virtual face, which are used to represent the 3D coordinates of the eye sockets of the initial virtual face (x i ,y i 、z i ), recorded as Target.

[0053] In an optional embodiment, the computer device determines the first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, including: calling the iterative nearest point method to align the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, to obtain the first adjustment parameter, the first adjustment parameter including one or more of the rotation amount R, the translation amount t and the scaling amount s.

[0054] Among them, the Iterative Closest Points (ICP) algorithm is the most classic point cloud registration algorithm. It includes two parts: corresponding point search and pose solution. The algorithm can be used to find the matching relationship between point sets, and the result of the solution is the translation and rotation between the two point sets. Optionally, the computer device can use the ICP algorithm to obtain k corresponding relationship point pairs between the virtual eyeball point cloud and the eye socket point cloud of the virtual face, and calculate the rigid body transformation Ti so that the sum of the distances of the k pairs of matching points is minimized. Using the required matrix, the virtual eyeball point cloud is transformed to the coordinate system of the eye socket point cloud of the virtual face, and the error function between the transformed virtual eyeball point cloud and the eye socket point cloud of the virtual face is estimated. If the error is greater than the threshold, then continue to iterate until the given error requirement is met. That is to say, each iteration, the entire model will be closer, and each time the closest point is found again, and then the rotation and translation matrix is ​​calculated, and the variance error is compared. If it is not satisfied, continue to iterate. Finally, the adjustment parameters of the virtual eyeball point cloud transformed to the eye socket point cloud of the virtual face are obtained.

[0055] In an optional embodiment, after obtaining the first adjustment parameter, the computer device may adjust each point cloud point included in the point cloud of the virtual eyeball according to the first adjustment parameter so that the point cloud of the virtual eyeball is aligned with the point cloud of the eye socket of the first virtual face.

[0056] In an optional implementation, a virtual eyeball corresponds to a set of first adjustment parameters (i.e., R, t, s). Optionally, the computer device adjusts each point cloud point included in the point cloud of the virtual eyeball according to the first adjustment parameters by performing a multiplication operation on the first adjustment parameters and each point cloud point included in the point cloud of the virtual eyeball.

[0057] In the embodiment of the present application, since the adjustment parameters include the scaling amount, when the eye sockets of different virtual faces are of different sizes, the point cloud of the virtual eyeball can be adjusted based on the scaling amount to better align the point cloud of the virtual eyeball with the point cloud of the eye socket of the virtual face. And by aligning the point cloud of the virtual eyeball with the point cloud of the eye socket of the virtual face using the adjustment parameters, the problem of penetration between the virtual eyeball and the eye socket of the virtual face can be effectively avoided.

[0058] In an optional implementation, the computer device may also obtain a second virtual face; if the second virtual face corresponds to the same topological structure as the first virtual face, a second adjustment parameter is determined based on the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball. The second adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the second virtual face.

[0059] Since the identification information of 3D points in the same topological structure is the same, the semantic information they represent is also the same. Therefore, when the second virtual face corresponds to the same topological structure as the first virtual face, the first key point set corresponding to the eye socket of the first virtual face obtained from the point cloud of the first virtual face can be directly used as the key point set corresponding to the eye socket of the second virtual face. 30 For example, assuming that the first virtual face corresponds to point A in the topological structure 30 represents the left eye corner of the first virtual face. Then, when the second virtual face corresponds to the same topological structure as the first virtual face, point A 30 It also represents the left eye corner of the second virtual face. That is to say, for any virtual face having the same topological structure as the first virtual face, the identification information of the key points corresponding to its eye sockets remains unchanged.

[0060] In this embodiment, the computer device determines the second adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, including: calling the iterative nearest point method to align the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball to obtain the second adjustment parameter.

[0061] In this implementation, the computer device may also adjust each point cloud point included in the point cloud of the virtual eyeball according to the second adjustment parameter so that the point cloud of the virtual eyeball is aligned with the point cloud of the eye socket of the second virtual face.

[0062] In an optional embodiment, the computer device may also obtain the initial position of the virtual eyeball in the eye socket of the first virtual face before adjusting the point cloud of the virtual eyeball based on the first adjustment parameter; and add the virtual eyeball to the eye socket of the second virtual face based on the initial position.

[0063] As can be seen from the foregoing, the computer device has stored the initial position of the virtual eyeball in the eye socket of the first virtual face. Then, when the second virtual face corresponds to the same topological structure as the first virtual face, the computer device can call the stored initial position; based on the initial position, the virtual eyeball is directly added to the eye socket of the second virtual face. In this way, the user does not need to manually mount the virtual eyeball in the second virtual face again, which can improve the efficiency of mounting the virtual eyeball.

[0064] It is understandable that when multiple virtual faces correspond to the same topological structure, the computer device only needs to execute step S501 and step S502 once. That is, for any other virtual face with the same topology as the first virtual face and the virtual eyeball with the same topological structure as the virtual eyeball, the computer device can determine different adjustment parameters based on the coordinates of each key point in the first key point set in the point cloud of the other virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball. Thus, according to the adjustment parameters, each point cloud point included in the point cloud of the virtual eyeball is adaptively adjusted so that the point cloud of the virtual eyeball is aligned with the point cloud of the eye socket of the virtual face.

[0065] In order to verify the accuracy of the eyeball data processing method provided in the embodiment of the present application, the eyeball data processing method was used to test virtual faces of different face shapes. Figure 5 , Figure 5 It is a schematic diagram of the matching effect of the eyeball and the eye socket provided in the embodiment of the present application. Figure 5In the figure, pictures 501, 504 and 507 refer to the schematic diagrams of virtual face 1, virtual face 2 and virtual face 3 without virtual eyeballs; pictures 502, 505 and 508 refer to the schematic diagrams of the front view of virtual face 1, virtual face 2 and virtual face 3 with virtual eyeballs; pictures 503, 506 and 509 refer to the schematic diagrams of the side view of virtual face 1, virtual face 2 and virtual face 3 with virtual eyeballs. Figure 5 As shown, the eyeball data processing method provided by the present application can perfectly combine eyes at any position with virtual people at any position, without any penetration phenomenon, and no problems can be seen from the front and side angles with the naked eye. This verifies that the eyeball data processing method provided by the present application can improve the authenticity of virtual people and provide a better user experience.

[0066] In the embodiment of the present application, the computer device obtains a first key point set corresponding to the eye socket of the first virtual face from the point cloud of the first virtual face; obtains a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball; and determines a first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face. By using the embodiment of the present application, the point cloud of the virtual eyeball is adaptively adjusted according to the adjustment parameter, which not only improves the matching efficiency of the eyeball and the eye socket, but also makes the virtual eyeball more matched with the eye socket of the virtual face, thereby improving the authenticity of the virtual person.

[0067] In addition, in the embodiment of the present application, since the computer device determines the adjustment parameters based on the coordinates of the key points corresponding to the virtual eyeball and the coordinates of the key points corresponding to the eye sockets of the virtual face, even if different virtual faces are in different coordinate systems, the computer can determine the adjustment parameters by obtaining the coordinates of the key points corresponding to the eye sockets of the virtual face and the coordinates corresponding to the key points of the virtual eyeball. Therefore, the method provided in the embodiment of the present application can be applied to the reconstruction of the faces of characters in multiple scenes such as 3D games / 3D film and television works / short videos. Compared with the 3D deformable face statistical model (3DMorphable Face Model, 3DMM), the eyeball data processing method provided in the present application can be more efficient and quickly mount the virtual eyeball in the eye socket of the virtual face, generating a more beautiful result, making it easier to use in corresponding products and improving the user experience of the product.

[0068] It should be noted that when the embodiments of the present application are applied to specific products or technologies, the first virtual face, virtual eyeballs, second virtual face, etc. involved in the embodiments of the present application are obtained after obtaining the user's permission or consent; and the collection, use and processing of the first virtual face, virtual eyeballs, second virtual face, etc. need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0069] Based on the description of the above-mentioned embodiments of the eyeball data processing method, the present application also proposes an eyeball data processing device, which can be a computer program (including program code) running in a computer device. The eyeball data processing device can execute Figure 2 The eye data processing method shown in Figure 6 , Figure 6 is a schematic diagram of an eyeball data processing device provided in an embodiment of the present application, and the eyeball data processing device may include the following modules:

[0070] An acquisition module 601 is used to acquire a first key point set corresponding to the eye sockets of the first virtual face from the point cloud of the first virtual face;

[0071] The acquisition module 601 is further used to acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball;

[0072] Processing module 602 is used to determine a first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face.

[0073] In an optional implementation, when the acquisition module 601 is used to acquire the first key point set corresponding to the eye socket of the first virtual face from the point cloud of the first virtual face, it is specifically used to:

[0074] Acquire identification information of key points corresponding to eye sockets of the first virtual face from a target topological structure corresponding to the first virtual face, where the target topological structure includes identification information of key points corresponding to various parts of the first virtual face;

[0075] Based on the identification information of the key points corresponding to the eye sockets, a first key point set corresponding to the eye sockets is obtained from the point cloud of the first virtual face, where the first key point set includes the identification information of the key points corresponding to the eye sockets.

[0076] In an optional implementation, when the acquisition module 601 is used to acquire the second key point set corresponding to the first key point set from the point cloud of the virtual eyeball, it is specifically used to:

[0077] Acquire at least one vertex included in the point cloud of the virtual eyeball;

[0078] According to the distance between at least one vertex and the key points included in the first key point set, a second key point set corresponding to the first key point set is determined from the at least one vertex.

[0079] In an optional implementation, when the acquisition module 601 is used to determine the second key point set corresponding to the first key point set from at least one vertex according to the distance between at least one vertex and the key points included in the first key point set, it is used to:

[0080] For any key point in the first key point set, based on the coordinates of each vertex in the at least one vertex and the coordinates of any key point, determine the distance between each vertex and any key point;

[0081] Get the target vertex with the smallest distance to any key point;

[0082] The target vertex is used as a key point corresponding to any key point in the point cloud of the virtual eyeball, and the target vertex is added to a second key point set corresponding to the first key point set.

[0083] In an optional implementation, the processing module 602 is used to determine the first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, specifically for:

[0084] The iterative closest point method is called to align the coordinates of each key point in the first key point set in the point cloud of the first virtual face with the coordinates of each key point in the second key point set in the virtual eye point cloud to obtain a first adjustment parameter.

[0085] In an optional embodiment, the processing module 602 is also used to adjust each point cloud point included in the point cloud of the virtual eyeball according to a first adjustment parameter so that the point cloud of the virtual eyeball is aligned with the point cloud of the eye socket of the first virtual face, and the first adjustment parameter includes one or more of rotation amount, translation amount and scaling amount.

[0086] In an optional implementation, the acquisition module 601 is further configured to acquire a second virtual face.

[0087] In an optional embodiment, if the second virtual face corresponds to the same topological structure as the first virtual face, the processing module 602 is also used to determine a second adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, and the second adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the second virtual face.

[0088] In an optional implementation, the acquisition module 601 is further used to acquire the initial position of the virtual eyeball in the eye socket of the first virtual face before the point cloud of the virtual eyeball is adjusted based on the first adjustment parameter.

[0089] In an optional implementation, the processing module 602 is further configured to add a virtual eyeball to the eye socket of the second virtual face based on the initial position.

[0090] According to one embodiment of the present application, Figure 2 Each step involved in the method shown can be Figure 6 The eyeball data processing device shown in the figure is executed by various modules. For example, Figure 2 The steps S201 and S202 shown in FIG. Figure 6 The acquisition module 601 shown in FIG. 1 is executed, and step S203 can be performed by Figure 6 The processing module 602 shown in FIG.

[0091] According to another embodiment of the present application, the program can be executed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Figure 6 The eyeball data processing device shown in the embodiment of the present application is used to implement the eyeball data processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the above-mentioned computer device through the computer-readable storage medium and run therein.

[0092] It can be understood that the specific implementation of each module in the eye data processing device provided in the embodiment of the present application and the beneficial effects that can be achieved can be referred to the description of the aforementioned eye data processing method embodiment, and will not be repeated here.

[0093] Based on the description of the above method embodiment and device embodiment, the present application embodiment also provides a computer device. Figure 7The computer device at least includes a processor 701, a memory 702, and a communication interface 703. The processor 701, the memory 702, and the communication interface 703 may be connected via a bus 704 or other methods. The embodiment of the present application takes the connection via the bus 704 as an example.

[0094] Among them, the processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off instructions sent by the user to the computer device, and control the computer device to perform power on and off operations; for another example, the CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 703 can optionally include a standard wired interface, a wireless interface (such as Wi-Fi, a mobile communication interface, etc.), which is controlled by the processor 701 to send and receive data. The memory 702 (Memory) is a memory device in the computer device for storing computer programs and data. It can be understood that the memory 702 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 702 provides a storage space, which stores the operating system of the computer device, which may include but is not limited to: Windows system, Linux system, Android system, iOS system, etc., and this application does not limit this. In an optional implementation, the processor 701 of the embodiment of the present application may perform the following operations by running the computer program stored in the memory 702:

[0095] Acquire a first key point set corresponding to the eye sockets of the first virtual face from the point cloud of the first virtual face;

[0096] Acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball;

[0097] Based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, a first adjustment parameter is determined, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual face.

[0098] In an optional implementation, when the processor 701 is used to obtain the first key point set corresponding to the eye socket of the first virtual face from the point cloud of the first virtual face, it is specifically used to:

[0099] Acquire identification information of key points corresponding to eye sockets of the first virtual face from a target topological structure corresponding to the first virtual face, where the target topological structure includes identification information of key points corresponding to various parts of the first virtual face;

[0100] Based on the identification information of the key points corresponding to the eye sockets, a first key point set corresponding to the eye sockets is obtained from the point cloud of the first virtual face, where the first key point set includes the identification information of the key points corresponding to the eye sockets.

[0101] In an optional implementation, when the processor 701 is used to obtain the second key point set corresponding to the first key point set from the point cloud of the virtual eyeball, it is specifically used to:

[0102] Acquire at least one vertex included in the point cloud of the virtual eyeball;

[0103] According to the distance between at least one vertex and the key points included in the first key point set, a second key point set corresponding to the first key point set is determined from the at least one vertex.

[0104] In an optional implementation, the processor 701, when used to determine, from at least one vertex according to the distance between the at least one vertex and the key points included in the first key point set, a second key point set corresponding to the first key point set, is configured to:

[0105] For any key point in the first key point set, based on the coordinates of each vertex in the at least one vertex and the coordinates of any key point, determine the distance between each vertex and any key point;

[0106] Get the target vertex with the smallest distance to any key point;

[0107] The target vertex is used as a key point corresponding to any key point in the point cloud of the virtual eyeball, and the target vertex is added to a second key point set corresponding to the first key point set.

[0108] In an optional implementation, the processor 701, when used to determine the first adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the first virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, is specifically used to:

[0109] The iterative closest point method is called to align the coordinates of each key point in the first key point set in the point cloud of the first virtual face with the coordinates of each key point in the second key point set in the virtual eye point cloud to obtain a first adjustment parameter.

[0110] In an optional embodiment, the processor 701 is also used to adjust each point cloud point included in the point cloud of the virtual eyeball according to a first adjustment parameter so that the point cloud of the virtual eyeball is aligned with the point cloud of the eye socket of the first virtual face, and the first adjustment parameter includes one or more of rotation amount, translation amount and scaling amount.

[0111] In an optional implementation, the processor 701 is further configured to:

[0112] Obtaining a second virtual face;

[0113] If the second virtual face corresponds to the same topological structure as the first virtual face, a second adjustment parameter is determined based on the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball. The second adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the second virtual face.

[0114] In an optional implementation, the processor 701 is further configured to:

[0115] obtaining an initial position of the virtual eyeball in the eye socket of the first virtual human face before adjusting the point cloud of the virtual eyeball based on the first adjustment parameter;

[0116] The virtual eyeballs are added to the eye sockets of the second virtual face based on the initial positions.

[0117] In a specific implementation, the processor 701, memory 702, and communication interface 703 described in the embodiments of the present application can execute the implementation method of the computer device described in the eye data processing method provided in the embodiments of the present application, and can also execute the implementation method described in the eye data processing device provided in the embodiments of the present application, which will not be repeated here.

[0118] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer executes the eyeball data processing method of any possible implementation method described above. The specific implementation method can be referred to the above description, which will not be repeated here.

[0119] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the eye data processing method of any possible implementation method described above. The specific implementation method can be referred to the above description, which will not be repeated here.

[0120] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0121] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0122] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for processing eyeball data, characterized in that: The method comprises: Obtaining a first virtual face and virtual eyeballs; In response to a user's mounting operation on the virtual eyeball on the first virtual face, mounting the virtual eyeball on the eye socket of the first virtual face; Acquire the initial position of the virtual eyeball in the eye socket of the first virtual face, and store the initial position; Acquire a first set of key points corresponding to eye sockets of the first virtual face from the point cloud of the first virtual face; Acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball; Calling an iterative closest point method to align the coordinates of each key point in the first key point set in the point cloud of the first virtual human face with the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball to obtain a first adjustment parameter; the first adjustment parameter includes one or more of a rotation amount, a translation amount, and a scaling amount, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual human face; Obtaining a second virtual face; If the second virtual face and the first virtual face correspond to the same topological structure, adding the virtual eyeball to the eye socket of the second virtual face based on the stored initial position; wherein the topological structure is used to indicate identification information of key points corresponding to various parts of the virtual face; The first key point set is used as the key point set corresponding to the eye sockets of the second virtual face, and based on the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, a second adjustment parameter is determined, and the second adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye sockets of the second virtual face.

2. The method according to claim 1, characterized in that: The acquiring, from the point cloud of the first virtual face, a first set of key points corresponding to the eye sockets of the first virtual face comprises: Acquire identification information of key points corresponding to eye sockets of the first virtual face from a target topological structure corresponding to the first virtual face, wherein the target topological structure includes identification information of key points corresponding to various parts of the first virtual face; Based on the identification information of the key points corresponding to the eye sockets, a first key point set corresponding to the eye sockets is obtained from the point cloud of the first virtual face, where the first key point set includes the identification information of the key points corresponding to the eye sockets.

3. The method according to claim 1, characterized in that The acquiring, from the point cloud of the virtual eyeball, a second key point set corresponding to the first key point set comprises: Acquire at least one vertex included in the point cloud of the virtual eyeball; According to the distance between the at least one vertex and the key points included in the first key point set, a second key point set corresponding to the first key point set is determined from the at least one vertex.

4. The method according to claim 3, characterized in that The determining, from the at least one vertex according to the distance between the at least one vertex and the key points included in the first key point set, a second key point set corresponding to the first key point set comprises: For any key point in the first key point set, based on the coordinates of each vertex in the at least one vertex and the coordinates of the any key point, determine the distance between each vertex and the any key point; Obtain the target vertex with the shortest distance from any key point; The target vertex is used as a key point corresponding to any key point in the point cloud of the virtual eyeball, and the target vertex is added to a second key point set corresponding to the first key point set.

5. The method according to claim 1, characterized in that The method further comprises: Each point cloud point included in the point cloud of the virtual eyeball is adjusted according to the first adjustment parameter so that the point cloud of the virtual eyeball is aligned with the point cloud of the eye socket of the first virtual face.

6. An eyeball data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a first virtual face and a virtual eyeball; a processing module, configured to mount the virtual eyeball on the eye socket of the first virtual face in response to a user's mounting operation on the virtual eyeball on the first virtual face; The acquisition module is further used to acquire the initial position of the virtual eyeball in the eye socket of the first virtual face and store the initial position; The acquisition module is further used to acquire a first key point set corresponding to the eye sockets of the first virtual face from the point cloud of the first virtual face; The acquisition module is further used to acquire a second key point set corresponding to the first key point set from the point cloud of the virtual eyeball; The processing module is further used to call the iterative closest point method to align the coordinates of each key point in the first key point set in the point cloud of the first virtual human face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball to obtain a first adjustment parameter; the first adjustment parameter includes one or more of a rotation amount, a translation amount, and a scaling amount, and the first adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye socket of the first virtual human face; The acquisition module is further used to acquire a second virtual face; The processing module is further configured to add the virtual eyeball to the eye socket of the second virtual face based on the stored initial position if the second virtual face corresponds to the same topological structure as the first virtual face; wherein the topological structure is used to indicate identification information of key points corresponding to various parts of the virtual face; The processing module is also used to use the first key point set as the key point set corresponding to the eye sockets of the second virtual face, and determine a second adjustment parameter based on the coordinates of each key point in the first key point set in the point cloud of the second virtual face and the coordinates of each key point in the second key point set in the point cloud of the virtual eyeball, wherein the second adjustment parameter is used to adjust the point cloud of the virtual eyeball so that the virtual eyeball matches the eye sockets of the second virtual face.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the eye data processing method according to any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The computer device includes a memory, a communication interface and a processor, wherein the memory, the communication interface and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the eye data processing method described in any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product comprises a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the eye data processing method according to any one of claims 1 to 5 is implemented.

Images