Object processing method, related apparatus and storage medium

Abstract

Embodiments of the present application disclose an object processing method, related device and storage medium, and relate to the field of computer vision and machine learning. The method comprises: generating an initial three-dimensional model according to a target video comprising N objects to be modeled with the same object type, N>1; determining M object images corresponding to M objects to be modeled in the target video frame on a video frame imaging plane, and mapping key points in the initial three-dimensional model to the M object images to determine the visibility probability of the two-dimensional plane points of the key points in each object image. Since the visibility probability can identify the probability of each object to be modeled not being occluded in the target video frame, the key points of the initial three-dimensional model can be corrected according to the visibility probability to obtain a more accurate three-dimensional model. Thus, the adverse effects of occlusion between multiple objects to be modeled on three-dimensional modeling of multiple targets can be reduced, and the generation quality of the three-dimensional model is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to an object processing method, related apparatus, and storage medium. Background Technology

[0002] 3D reconstruction refers to establishing mathematical models of 3D objects suitable for computer representation and processing. Reconstructing 3D objects in complex scenes based on monocular red-green-blue (RBG) video is a long-standing and challenging task in computer vision and graphics. It can be applied to various fields such as 3D face reconstruction, metaverse, and robotics, and can be used to reconstruct 3D objects of various types, such as faces and human bodies.

[0003] In related technologies, 3D object reconstruction can be achieved by establishing a mapping relationship between 3D objects and 3D models. When performing multi-object 3D reconstruction, a shared 3D model needs to be created for objects of the same type. Compared to single-object 3D modeling based on video containing a single object, multi-object 3D modeling based on video containing multiple objects of the same type improves the efficiency of 3D modeling. This is because video containing multiple objects of the same type can simultaneously provide relevant data for multiple objects of the same type, and different objects of the same type can all be used to create a shared 3D model. In contrast, video containing a single object only provides relevant data for that single object. Therefore, reading a single video frame containing multiple objects of the same type is equivalent to reading multiple video frames containing a single object. In other words, single-object 3D modeling requires more video data compared to multi-object 3D modeling.

[0004] However, in videos used for multi-object 3D modeling, multiple objects belonging to the same category may occlude each other relative to the acquisition device. For example, in a video containing multiple faces, face A may occlude face B in some video frames relative to the acquisition device. This type of occlusion will directly affect the quality of the subsequent 3D model generation. Summary of the Invention

[0005] This application provides an object processing method, related apparatus, and storage medium that can reduce the adverse effects of occlusion between multiple objects on the 3D modeling of multiple targets and improve the quality of the generated 3D model.

[0006] In a first aspect, embodiments of this application provide an object processing method, including:

[0007] An initial 3D model is generated based on the target video, wherein the target video includes N objects of the same type to be modeled, and the N objects to be modeled share the initial 3D model, where N>1;

[0008] For a target video frame, determine M object images of M objects to be modeled in the target video frame on the video frame imaging plane. The M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N.

[0009] Keypoints in the initial 3D model are mapped to M object images, and the visibility probability of each keypoint in a 2D plane point within each object image is determined. The target images in the M object images include the target object. The target 2D plane points in the target images correspond to the keypoints, and the visibility probability of the target 2D plane points is used to indicate the probability that the target object's 2D plane points are not occluded in the target video frame.

[0010] The target 3D model is obtained by correcting the corresponding key points in the initial 3D model based on the visibility probability.

[0011] Secondly, embodiments of this application also provide an object processing apparatus, including a transceiver module and a processing module:

[0012] The transceiver module is used for:

[0013] Acquire the target video;

[0014] Processing module, used for:

[0015] An initial 3D model is generated based on the target video, wherein the target video includes N objects of the same type to be modeled, and the N objects to be modeled share the initial 3D model, where N>1;

[0016] For a target video frame, determine M object images of M objects to be modeled in the target video frame on the video frame imaging plane. The M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N.

[0017] Keypoints in the initial 3D model are mapped to M object images, and the visibility probabilities of the keypoints as 2D planar points in each object image are determined. The object images in the M object images include the target object, and the target 2D planar points in the target images correspond to the keypoints. The visibility probabilities of the target 2D planar points are used to indicate the probability that the target object's 2D planar points are not occluded in the target video frame.

[0018] The target 3D model is obtained by correcting the corresponding key points in the initial 3D model based on the visibility probability.

[0019] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0021] In summary, this application provides an object processing method, related apparatus, and storage medium. The method includes: for a target video containing N objects of the same object type (N>1), in order to obtain a shared 3D model for the N objects, firstly, an initial 3D model shared by the N objects is generated based on the target video; then, for a target video frame in the target video, on the imaging plane of the video frame, M object images corresponding to the M objects to be modeled in the target video frame are determined, where 2≤M≤N, and key points in the initial 3D model are mapped to the M object images, and the visibility probabilities of the key points in the two-dimensional plane points of the M object images are determined. Taking the target object among the M objects to be modeled as an example, the target object corresponds to the target image in the M object images, and the two-dimensional plane points mapped by the key points in the target image are the target two-dimensional plane points. Since the visibility probability of the target two-dimensional plane points can indicate the probability that the target two-dimensional plane points are not occluded in the target video frame, the visibility probability of the target two-dimensional plane points can indicate the probability that the target object is not occluded in the target video frame. In other words, the visibility probabilities corresponding to the M object images can each identify the probability that the object to be modeled for each object image is not occluded in the target video frame. Since occluded objects will affect the quality of the initial 3D model and the accuracy of its key points, the key points of the initial 3D model can be corrected based on the visibility probabilities of the M object images corresponding to the target video frame, resulting in a more accurate target 3D model. Therefore, the adverse effects of occlusion between multiple objects on multi-object 3D modeling can be reduced, improving the quality of the generated 3D model. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of the object processing method provided in the embodiments of this application;

[0024] Figure 2 This is a flowchart illustrating the object processing method provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an object processing embodiment provided in this application;

[0026] Figure 4 A schematic diagram illustrating the determination of the density and color of sampling points, provided for an embodiment of this application;

[0027] Figure 5 A schematic diagram illustrating the determination of a rotation matrix and scaling factor, provided for an embodiment of this application;

[0028] Figure 6 This is a schematic block diagram of an object processing device provided in an embodiment of this application;

[0029] Figure 7 A schematic block diagram of a physical device for an object processing apparatus provided in an embodiment of this application.

[0030] Figure 8 This is a schematic diagram of the structure of a terminal in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of a server structure in one embodiment of this application. Detailed Implementation

[0032] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0033] This application provides an object processing method, related apparatus, and storage medium. The object processing method can be executed by the object processing apparatus provided in this application, or by a computer device that integrates the object processing apparatus. The object processing apparatus can be implemented in hardware or software, and the computer device can be a terminal or a server.

[0034] When the computer device is a server, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0035] When the computer device is a terminal, the terminal may include, but is not limited to, smart terminals with multimedia data processing functions (e.g., video data playback function, music data playback function), such as smartphones, tablets, laptops, desktop computers, smart TVs, smart speakers, personal digital assistants (PDAs), desktop computers, and smartwatches.

[0036] The solutions in this application can be implemented based on artificial intelligence technology, specifically involving computer vision technology in artificial intelligence technology and cloud computing, cloud storage and database in cloud technology, which will be described separately below.

[0037] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0038] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0039] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, model robustness testing, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as fingerprint recognition.

[0040] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0041] The solutions in this application can be implemented based on cloud technology, specifically involving cloud computing, cloud storage, and database technologies, which will be described below.

[0042] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied to cloud computing business models. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. The embodiments of this application can use cloud technology to save identification results.

[0043] Cloud storage is a new concept that extends and develops from cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of storage devices (also called storage nodes) of various types in a network through application software or application interfaces to work together and provide data storage and business access functions. In the embodiments of this application, network configuration and other information can be stored in this storage system for easy retrieval by the server.

[0044] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0045] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0046] A database, simply put, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, with minimal redundancy, and independent of application programs.

[0047] A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as SQL (Structured Query Language) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages. In this embodiment, the identification results can be stored in the DBMS for easy retrieval by the server.

[0048] It should be noted that the service terminal involved in the embodiments of this application can be a device that provides voice and / or data connectivity to the service terminal, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. Examples include mobile phones (or "cellular" phones) and computers with mobile terminals, such as portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and / or data with a wireless access network. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, and Personal Digital Assistants (PDAs).

[0049] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the object processing method provided in an embodiment of this application. The object processing method is applied to... Figure 1 In the object processing system, the object processing system includes a terminal 11 and a server 12, which communicate with each other via a network.

[0050] In practical applications, users can capture video through terminal 11, and terminal 11 sends the captured video to server 12 via the network, so that server 12 can execute the object processing method provided in this application embodiment. For example, based on a target video including N objects to be modeled, an initial three-dimensional model shared by the N objects to be modeled is generated. The N objects to be modeled have the same object type, and N>1. For the target video frame of the target video, M object images of the M objects to be modeled in the target video frame are determined on the imaging plane of the video frame. The M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N. The key points in the initial 3D model are mapped to M object images. The visibility probabilities of the key points in the 2D plane points of the M object images are determined. For the target object among the M objects to be modeled, the target object corresponds to the target image in the M object images. The key points in the target image are mapped to the target 2D plane points. The visibility probabilities of the target 2D plane points are used to identify the probability that the target 2D plane points of the target object are not occluded in the target video frame. The key points in the initial 3D model are corrected according to the visibility probabilities to obtain a target 3D model shared by N objects to be modeled.

[0051] To facilitate understanding, the following will be combined with... Figure 2 Please provide a detailed explanation.

[0052] Figure 2 This is a flowchart illustrating the object processing method provided in an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps S210-S240.

[0053] S210. Generate an initial 3D model based on the target video. The target video includes N objects of the same type to be modeled. The N objects to be modeled share the initial 3D model, where N>1.

[0054] Video refers to various technologies that capture, record, process, store, transmit, and reproduce a series of still images as electrical signals. When continuous image changes exceed 24 frames per second, according to the principle of visual persistence, the human eye cannot distinguish a single still image; what appears to be a smooth and continuous visual effect is called video.

[0055] The object to be modeled is the object in the video that awaits 3D modeling. It can be a human body, a part of a human body, or various other objects. The object to be modeled can be a rigid or non-rigid body. The object type is used to identify the type of the object to be modeled, such as identifying the object as a human body or a certain type of object. There is no specific limitation on the object type of the object to be modeled here.

[0056] In this embodiment, the target video refers to a video containing N objects to be modeled, where N>1, meaning the target video will include multiple objects to be modeled. It should be noted that the N objects to be modeled need to have the same object type; that is, the N objects to be modeled must be of the same object type to ensure that multi-object 3D modeling can be performed based on the N objects. For example, the N objects to be modeled can all be of the object type "face".

[0057] A 3D model is a shared model obtained by modeling N objects in 3D. It can be located in a 3D canonical space and is a shared model in the canonical space corresponding to the N objects. For example, when the N objects are all human faces, the corresponding 3D model can be a 3D human face model. For example, a 3D model can be a MESH model.

[0058] The initial 3D model refers to the uncorrected 3D model corresponding to N objects to be modeled. The server can generate an initial 3D model shared by N objects to be modeled based on the target video containing the N objects. It should be noted that although the target video is dynamic, the initial 3D model generated from the target video is a static model.

[0059] In the process of multi-target 3D modeling of N objects to be modeled, occlusion may occur between the N objects relative to the acquisition device. Since the server does not consider occlusion, it will treat the occluded object as an unoccluded object for 3D modeling, resulting in a poor quality initial 3D model. For example, when the N objects are N faces, face A may occlude half of face B. The server will treat the occluded half of face B as the entire face for 3D modeling, resulting in a poor-quality initial 3D face model. In other words, because occlusion is not considered, the generated initial 3D model in S210 has poor quality.

[0060] S220. For the target video frame of the target video, determine the M object images of the M objects to be modeled in the target video frame on the video frame imaging plane, where 2≤M≤N.

[0061] Since occlusion between N objects to be modeled in the target video will result in poor quality of the initial 3D model generated in S210, the server will correct the adverse effects of occlusion in subsequent steps to ensure the effect of multi-object 3D modeling.

[0062] In response, the server can determine whether there is occlusion between multiple objects to be modeled in multiple video frames of the target video. A video frame refers to a frame image in the target video that includes multiple objects to be modeled.

[0063] See Figure 3This figure is a schematic diagram of object processing provided in an embodiment of this application. Figure 3 In this process, four video frames containing the face of the object to be modeled are selected from each video frame according to the time progression, and 3D modeling is performed based on the target object in the four video frames.

[0064] For ease of explanation, the following explanation will take one video frame from the multiple video frames included in the target video, namely the target video frame, as an example. The target video frame can be the video frame corresponding to the target video at time t.

[0065] It should be noted that since the N objects to be modeled in the target video may not appear in every video frame included in the target video, the target video frame includes M objects to be modeled, 2≤M≤N. That is, the target video frame may include N objects to be modeled, or it may only include some of the N objects to be modeled.

[0066] Furthermore, for videos used for 3D modeling of a single target, and for video frames in the target video that only include a single object to be modeled, since the single object to be modeled may also be occluded due to the acquisition angle of the acquisition device, for example, when the object to be modeled is face A, the acquisition angle of the acquisition device may only be able to capture the left half of face A, that is, face A is occluded. Therefore, the object processing method provided in this embodiment can also be used to reduce the adverse effects of occlusion on the 3D modeling of a single target.

[0067] For the target video frame, since there is no correlation between whether each of the M objects to be modeled is occluded, that is, when object A of the M objects to be modeled is occluded, object B of the M objects to be modeled may or may not be occluded, the server needs to determine whether each of the M objects to be modeled is occluded.

[0068] To address this, the server can first determine the M object images of the M objects to be modeled in the target video frame on the target video frame's imaging plane. Here, the target video frame's imaging plane refers to the two-dimensional imaging plane corresponding to the target video frame, which is the same as the two-dimensional imaging plane corresponding to the target video. The object image refers to the corresponding two-dimensional image of the object to be modeled on the target video frame's imaging plane. The M object images correspond one-to-one with the M objects to be modeled, meaning that each of the M objects to be modeled has a corresponding object image. This lays the foundation for determining whether each of the M objects to be modeled is occluded in subsequent steps.

[0069] S230. Map the key points in the initial 3D model to M object images, and determine the visibility probability of the key points as 2D plane points in each object image.

[0070] A key point refers to a point in the initial 3D model, which can be any point on the surface of the initial 3D model. Since the initial 3D model is shared by N objects to be modeled, that is, the initial 3D model is shared by M objects to be modeled, the server can map the key points to the M object images corresponding to the M objects to be modeled, thereby determining the visibility probability of the key points as two-dimensional plane points in the M object images.

[0071] The following explanation uses one of the M objects to be modeled, i.e., the target object, as an example. The target object can refer to any one of the M objects to be modeled.

[0072] The target image in the M object images includes the target object; that is, the two-dimensional image of the target object corresponding to the imaging plane of the video frame is the target image. The target two-dimensional plane points in the target image correspond to the keypoints; that is, the two-dimensional plane points obtained by mapping the keypoints to the target image are the target two-dimensional plane points. For example, keypoints can be mapped to the target image using the following formula:

[0073] X t1 =G t -1 M *→t (X *1 )

[0074] Among them, X t1 G represents the coordinates of a point on the target two-dimensional plane. t -1 G represents the principal transformation matrix. t The inverse matrix of the principal transformation matrix is ​​used to identify the transformations required for the model to be reconstructed during the 3D reconstruction of the object to be reconstructed, while the inverse matrix of the principal transformation matrix is ​​used to identify the transformations required for the initial 3D model when 3D modeling the object to be modeled; M *→t This represents the mapping from normal space to camera space. Normal space is the space where the initial 3D model resides, while camera space refers to the 3D space corresponding to the target video. Camera space and normal space are not the same; X *1 Represents the coordinates of key points.

[0075] The visibility probability of a target 2D plane point is used to identify the probability that the target 2D plane point of the target object is not occluded in the target video frame. For example, when the visibility probability of the target 2D plane point is high, it means that the target 2D plane point is more likely to be occluded. Or, when the visibility probability of the target 2D plane point is low, it means that the target 2D plane point is less likely to be occluded, that is, it is more likely to be occluded.

[0076] Since the visibility probability of a target 2D plane point can indicate the probability that the target 2D plane point is not occluded in the target video frame, and the target 2D plane point is a 2D plane point obtained by mapping key points to the target image corresponding to the target object, the visibility probability of the target 2D plane point can indicate the probability that the target object is not occluded in the target video frame. For example, when the visibility probability of the target 2D plane point is large, it means that the target object is more likely to be unoccluded, or when the visibility probability of the target 2D plane point is small, it means that the target object is less likely to be occluded.

[0077] In other words, the visibility probabilities of key points in the two-dimensional plane points of the M object images can represent the probability that the object to be modeled in each of the M object images is not occluded in the target video frame. That is, the visibility probabilities of the two-dimensional plane points in the M object images can be used to accurately determine whether the M objects to be modeled are occluded in the target video frame, laying the foundation for subsequent correction of the initial 3D model.

[0078] In one possible implementation, step S230 maps key points in the initial 3D model to M object images, and determines the visibility probability of each key point as a 2D planar point in one of the M object images, including:

[0079] The visual prediction model maps key points in the initial 3D model to M object images, and determines the visibility probability of each key point as a 2D plane point in the M object images. The visual prediction model is a network model used to predict the visibility probability of key points as 2D plane points in object images.

[0080] Since the accuracy of the visibility probability is closely related to whether the corresponding object to be modeled is occluded in the target video frame, when determining the visibility probability through the visibility prediction model, it is necessary to obtain a visibility prediction model that can accurately predict the visibility probability. That is, it is necessary to train the initial visibility prediction model, which refers to the visibility prediction model that has not been trained.

[0081] The visual prediction model is trained in the following way:

[0082] S11. Obtain sample videos for training, and an initial sample 3D model generated based on Q sample objects included in the sample videos, where Q>1.

[0083] A sample object refers to an object in a video used as a training sample. A sample video refers to a video containing Q sample objects that can be used to train a visual prediction model. A sample video consists of multiple sample video frames. It should be noted that, in order to ensure that the trained visual prediction model can be used for the target video, the object types of the Q sample objects must be the same as the object types of the N objects to be modeled.

[0084] A sample 3D model refers to a shared model obtained by 3D modeling Q sample objects. An initial sample 3D model refers to an uncorrected sample 3D model corresponding to Q sample objects. The server can generate an initial sample 3D model based on the Q sample objects included in the sample video.

[0085] To train a visual prediction model, the server can acquire sample videos and corresponding initial sample 3D models.

[0086] S12. For the target sample video frame of the sample video, determine the P sample object images of the P sample objects in the target sample video frame on the video frame imaging plane, where 2≤P≤Q.

[0087] A target sample video frame refers to one frame among multiple sample video frames included in a sample video. For ease of explanation, the following explanation uses a target sample video frame as an example.

[0088] It should be noted that since the Q sample objects included in the sample video may not appear in every sample video frame included in the sample video, the target sample video frame includes P sample objects, 2≤P≤Q. That is, the target sample video frame may include Q sample objects, or it may only include some of the Q sample objects.

[0089] The server can determine the images of P sample objects in the target sample video frame on the video frame imaging plane. The video frame imaging plane is the two-dimensional imaging plane corresponding to the target sample video frame, which is the same as the two-dimensional imaging plane corresponding to the sample video. The sample object image is the corresponding two-dimensional image of the sample object on the video frame imaging plane. The P sample object images correspond one-to-one with the P sample objects, that is, each of the P sample objects has a corresponding sample object image, which lays the foundation for predicting whether each of the P sample objects is occluded in subsequent steps.

[0090] S13. Using the initial visual prediction model, map the key points of the sample in the initial three-dimensional model to P sample object images, and determine the visual prediction probability of the key points of the sample in the two-dimensional plane points of the P sample object images.

[0091] A sample key point refers to a point in the initial sample 3D model. It can be any point on the surface of the initial sample 3D model. Since the initial sample 3D model is shared by Q sample objects, that is, the initial sample 3D model is shared by P sample objects, the server can map the key points to the P sample object images corresponding to the P sample objects respectively, thereby determining the visual prediction probability of the sample key points as two-dimensional plane points in the P sample object images respectively.

[0092] The following explanation uses one of the P sample objects, namely the target sample object, as an example. The target sample object can refer to any one of the P sample objects.

[0093] The target sample image in a set of P sample object images includes the target sample object, and the target sample plane points in the target sample image correspond to the sample key points. For example, the target sample plane points can be determined using the following formula:

[0094] p t0＝ ∏ t0 G t -1 M *→t (X *0 )

[0095] Where, p t0 Represents the coordinates of a point on the target sample plane; ∏ t0 G represents the projection matrix of the camera corresponding to the sample video; t -1 G represents the principal transformation matrix. t The inverse matrix of the principal transformation matrix is ​​used to identify the transformations that need to be performed on the initial 3D model when modeling the object to be modeled in 3D. In other words, it can also be used to identify the transformations that need to be performed on the initial 3D model when modeling the sample object in 3D. This represents the mapping relationship from the normal space to the camera space. The normal space is not only the 3D space where the initial 3D model is located, but also the 3D space where the initial sample 3D model is located. The camera space is not only the camera space where the target video is located, but also the camera space where the sample video is located. *0 Represents the coordinates of key points in the sample.

[0096] The visual prediction probability of a target sample plane point is used to identify the probability that a target sample plane point of a target sample object is not occluded in the target sample video frame.

[0097] Since the visual prediction probability of the target sample plane point can identify the probability that the target sample plane point is not occluded in the target sample video frame, and the target sample plane point is a two-dimensional plane point obtained by mapping the sample key point to the target sample image corresponding to the target sample object, the visual prediction probability of the target sample plane point can identify the probability that the target sample object is not occluded in the target sample video frame, and is a prediction value obtained through the initial visual prediction model.

[0098] S14. Based on the difference between the visual prediction probability and the visual label of the target sample plane point, construct a loss function, and train the initial visual prediction model based on the loss function to obtain the visual prediction model.

[0099] Visual labels are used to identify whether a target sample plane point is occluded; they are the actual values ​​indicating whether a target sample plane point is occluded. This index function represents the visible labels of the target sample plane points. A value of 0 indicates that the target sample plane point is occluded, and the visible label... A value of 1 indicates that the target sample plane point is not occluded.

[0100] Based on the difference between the visual prediction probability (predicted value) and the visual label (true value), the server can construct a loss function and train the initial visual prediction model based on the loss function. This allows the initial visual prediction model to learn more about how to accurately predict whether the target sample plane point is occluded, resulting in a visual prediction model that can accurately predict the visual probability.

[0101] By training the initial visual prediction model, a corresponding visual prediction model is obtained, and the visual probability is accurately determined based on the visual prediction model, so that the initial 3D model can be reliably corrected in subsequent steps based on the accurate visual probability.

[0102] In one possible implementation, S14 constructs a loss function based on the difference between the visual prediction probability and the visual label of the target sample plane point, including:

[0103] Based on the occlusion status indicated by the visual tag, the difference weights of the corresponding target sample plane points are determined. Specifically, when the target sample plane point indicated by the visual tag is not occluded, the corresponding first difference weight is a first value; when the target sample plane point indicated by the visual tag is occluded, the corresponding second difference weight is a second value, and the first value is greater than the second value.

[0104] A loss function is constructed based on the difference between the visual prediction probability and the visual label of the target sample plane point, as well as the corresponding difference weight.

[0105] In this embodiment, after determining the visibility probability, the server can correct the initial 3D model in subsequent steps based on the visibility probability, thereby reducing the adverse effects of occluded objects on the initial 3D model. Therefore, to ensure the effectiveness of the correction and avoid misidentifying unoccluded objects as occluded objects, the server needs to train the initial visibility prediction model to learn more about occlusion during training. This ensures that the visibility probability determined by the obtained visibility prediction model will not incorrectly identify unoccluded objects as occluded objects, thus preventing erroneous modifications to the initial 3D model.

[0106] To address this, during the construction of the loss function, the server can determine the difference weights of the target sample plane points based on the occlusion status indicated by the visual labels. Specifically, when the visual label indicates that the target sample plane point is not occluded, the corresponding first difference weight has a first value; when the visual label indicates that the target sample plane point is occluded, the corresponding second difference weight has a second value, with the first value being greater than the second value. By ensuring that the first value of the difference weight is greater than the second value, the initial visual prediction model can learn more about occlusion, minimizing the possibility of the visual prediction model mispredicting an occluded model as an occluded model. This reduces the likelihood that the occluded model identified by the trained visual prediction model is actually an occluded model, ensuring that the identified occluded models are all truly occluded models, thus guaranteeing correct correction of the initial 3D model and improving the model's robustness.

[0107] By increasing the difference weights in the case of no occlusion, we can minimize the serious error of the visual prediction model predicting the unoccluded model to be reconstructed as the occluded model to be reconstructed, thereby minimizing the need for incorrect corrections to the initial 3D model in subsequent steps.

[0108] In one possible implementation, the first value is a hyperparameter, and the second value is used for adjustment based on the loss function during model training, for example, by... This index function represents the difference weight, and the target sample plane points. This indicates that the target sample plane point is not actually occluded, and the corresponding first difference weight can be 1. This indicates that the target sample plane point is actually occluded, and the corresponding second difference weight can be α, where α is the annealing parameter, which can be initialized to 1 and can decay during training.

[0109] By dynamically adjusting the difference weights under occlusion conditions during training, the ability of the initial visual prediction model to learn occlusion-related knowledge can be improved, and the possibility of overfitting during model training can be reduced.

[0110] In one possible implementation, the visual tags for the target sample video frame are determined as follows:

[0111] Determine the semantic mask for P sample objects in the target sample video frame. The semantic mask is used to identify the pixel range of the P sample objects; and

[0112] Based on the semantic masks of P sample objects, visual labels are determined for pixels in the corresponding P sample object images. In the target sample image, for pixels within the semantic mask of the target sample object, the corresponding visual label is "not occluded," and for pixels not within the semantic mask of the target sample object, the corresponding visual label is "occluded."

[0113] For target sample video frames, the visual labels of pixels can be quickly obtained through semantic masks.

[0114] Specifically, we can first determine the semantic mask for P sample objects in the target sample video frame. The semantic mask is used to identify the pixel range of the P sample objects. It should be noted that there is a one-to-one corresponding semantic mask for each sample object in the P sample objects.

[0115] Then, based on the semantic masks of P sample objects, the visual labels of pixels are determined in the corresponding P sample object images. Taking the target sample object among the P sample objects as an example, for the target sample object, there exists a corresponding semantic mask. In the target sample image, for pixels that are within the semantic mask of the target sample object, the semantic meaning of the pixel is the target sample object, so the corresponding visual label is "not occluded". For pixels that are not within the semantic mask of the target sample object, the semantic meaning of the pixel is not the target sample object. It may be the occluded part of the target sample object, or it may be other sample objects. Regardless of what the pixel is actually, since the semantic meaning of the pixel is not the target sample object, the corresponding visual label of the pixel is "occluded".

[0116] By using semantic masks for object recognition, the visual labels of pixels in each sample object image can be quickly determined, thereby enabling the determination of the visual labels of the target sample plane points.

[0117] In one possible implementation, for a target two-dimensional planar point, step S230 maps key points in the initial three-dimensional model to M object images, and determines the visibility probability of each key point as a two-dimensional planar point in one of the M object images, including:

[0118] Based on the ray from the camera's acquisition point to the target two-dimensional plane point in the acquired video frame, multiple sampling points for the target two-dimensional plane point are determined; and

[0119] The visibility probability of the target two-dimensional plane point is determined based on the sub-visibility probabilities corresponding to multiple sampling points.

[0120] In camera space, extending the line connecting the camera capture point of the target video frame to each pixel on the camera plane can form multiple rays. Taking the ray from the camera capture point of the target video frame to a point on the target two-dimensional plane as an example, the server samples this ray according to time (or depth), which can obtain multiple sampling points for the target two-dimensional plane point.

[0121] The server can determine the visibility probability of a target 2D plane point based on the sub-visibility probabilities corresponding to multiple sampling points. The sub-visibility probability of each sampling point identifies the probability that the sampling point is recorded by the camera in the video frame's imaging plane, or in other words, the probability that the sampling point is captured by the camera. This probability can be determined using the following formula:

[0122]

[0123] Where, τ n σ represents the sub-visibility probability of the nth sampling point; n The density of the nth sample can be obtained using the NeRF framework, such as... Figure 4 As shown, δ can be obtained simultaneously when calculating color. n This represents the interval between the nth sampling point and the (n+1)th sampling point.

[0124] See Figure 4 This figure is a schematic diagram illustrating how to determine the density and color of sampling points according to an embodiment of this application. Figure 4 In this study, a two-layer multilayer perceptron (MLP) was constructed. The first MLP was used to encode the sampling points corresponding to the target 2D plane points. The second MLP combined the encoded sampling points with the view direction parameters of the target 2D plane points to obtain the color C of the sampling points. t And density σ.

[0125] Using the above formula, the visibility probability of a target two-dimensional plane point can be determined based on the sub-visibility probabilities corresponding to multiple sampling points.

[0126] Accordingly, in one possible implementation, during the training of the initial prediction model, the loss function used for training can be determined using the following formula:

[0127]

[0128] Among them, L sil Represents the loss function; This indicator function represents the difference weight; The visible prediction probability of a target sample plane point is obtained by summing the sub-visual prediction probabilities of multiple sampling points corresponding to the target sample plane point. This index function represents the visual label of the target sample plane point.

[0129] S240. Correct the corresponding key points in the initial 3D model according to the visibility probability to obtain the target 3D model.

[0130] After determining the visibility probabilities of key points in the two-dimensional plane points of the M object images in S230, i.e., determining the probability that each of the M objects to be modeled is not occluded in the target video frame, since the server in S210 did not consider occlusion when generating the initial 3D model, that is, during the process of generating the initial 3D model, the server will treat the occluded objects to be modeled as unoccluded objects to be modeled for 3D modeling, resulting in poor quality of the generated initial 3D model. Therefore, in order to reduce the adverse effects of occlusion among the M objects to be modeled on the initial 3D model, the server can correct the corresponding key points in the initial 3D model according to the visibility probabilities, i.e., reduce the adverse effects of occluded objects to be modeled on the initial 3D model, and obtain the corrected target 3D model. The target 3D model refers to the corrected shared model corresponding to the N objects to be modeled, which is consistent with the initial 3D model. The target 3D model is also a static model.

[0131] It should be noted that, based on the explanation using the target video frame as an example, for multiple video frames included in the target video, the server can determine the visibility probability of each of the multiple objects to be modeled included in each of the multiple video frames through the relevant steps provided in this embodiment. This allows the server to correct the initial 3D model based on the visibility probability of each of the multiple objects to be modeled included in each video frame, thereby enabling the correction of the initial 3D model by the multiple video frames included in the target video.

[0132] In one possible implementation, the object handling methods also include:

[0133] The object to be reconstructed is determined from the M objects to be modeled in the target video frame;

[0134] Determine the 3D key points of the object to be reconstructed in the camera space of the target video, and the distance between them and the 3D mapping points in the camera space;

[0135] The rotation matrix and scaling factor used to adapt to differences in object size are determined based on the distance;

[0136] By using a rotation matrix and a scaling factor, the target 3D model is transformed into a 3D modeling model of the object to be reconstructed in the corresponding target video frame.

[0137] After obtaining the shared target 3D model for N objects to be modeled, the server can perform 3D reconstruction of each of the N objects based on the shared target 3D model. The following explanation uses a target video frame as an example.

[0138] The server can determine the objects to be reconstructed from M objects to be modeled in the target video frame. The objects to be reconstructed refer to the objects in the target video that are waiting to be reconstructed in three dimensions.

[0139] In complex real-world scenarios, among M objects of the same type to be modeled, different objects may differ in local height, volume, or proportion. In other words, different objects may exhibit individual size variations. For instance, when the M objects are M faces, the noses, eyes, and other features of different faces may show significant size differences. Therefore, different objects to be reconstructed from the M objects may also exhibit individual size variations. The server needs to be mindful of these size differences during the 3D reconstruction process.

[0140] Specifically, the server can first determine the distance between 3D keypoints and 3D mapping points in camera space. Camera space refers to the 3D space corresponding to the target video, which is different from the canonical space where the target 3D model resides. 3D keypoints can be any point of the object to be reconstructed within camera space, and can be defined by x... t This indicates that a key point can be any point specified in the target 3D model, and can be specified using o. * Indicates; a 3D mapping point is the mapped position of a specified keypoint in camera space, which can be expressed as o t express.

[0141] In one possible implementation, the keypoint can be designated as the center point of the target 3D model. The center point of the target 3D model refers to the object center of the target 3D model, which can be approximately obtained by averaging the surface sampling points of the target 3D model using a surface function.

[0142] As mentioned earlier, since 3D keypoints and 3D mapping points are mappings of the object to be reconstructed and the target 3D model in the same 3D space, the distance between 3D keypoints and 3D mapping points can reflect the shape difference between the object to be reconstructed and the target 3D model in the same 3D space. Therefore, the server can determine the rotation matrix and scaling factor based on the shape difference reflected by the distance, which is used to identify how the target 3D model needs to be selected and scaled to deform into a shape that conforms to the object to be reconstructed, that is, to transform the target 3D model into a 3D modeling model of the object to be reconstructed in the corresponding target video frame.

[0143] To further improve the accuracy of determining the rotation matrix and scaling factor, when determining the rotation matrix and scaling factor based on distance, the viewpoint direction parameters and 3D deformation encoding of the camera acquisition points of the target video can also be referenced.

[0144] Since the viewpoint direction parameter encodes the viewpoint direction of 3D keypoints, and the viewpoint direction of 3D keypoints is the direction formed between the 3D keypoint and the center of the object to be reconstructed, the viewpoint direction is used to determine whether the target feature point is visible. The 3D deformation encoding encodes the deformation that occurs when the 3D keypoint transforms from camera space to the normal space where the target 3D model is located. Therefore, the server can determine the rotation matrix and scaling factor based on the distance between the 3D keypoint and the 3D mapping point, the viewpoint direction parameter, and the 3D deformation encoding. The rotation matrix is ​​used to identify the rotation that the 3D keypoint needs to undergo during the 3D reconstruction of the object to be reconstructed, and the scaling factor is used to identify the scaling that the 3D keypoint needs to undergo during the 3D reconstruction of the object to be reconstructed. The scaling factor can be used to scale the 3D keypoints to allow the target 3D model to adapt to the size differences between different objects to be reconstructed.

[0145] As one possible implementation, the rotation matrix and scaling factor can be determined using a neural network approach. For example, a fully connected architecture based on a Neural Radiance Field (NeRF) can be used to establish a dense SE(3) field to compute the 3D keypoint x. t Around the three-dimensional mapping point o t The rotation matrix and scaling factor are shown in the following formulas:

[0146]

[0147] in, S represents the rotation matrix; t Indicates the scaling factor; x t -o t d represents the distance between the 3D keypoint and the 3D mapping point; t Indicates the viewpoint direction parameter; φ t It represents 3D deformation encoding; MLP represents a fully connected network layer.

[0148] See Figure 5 This figure is a schematic diagram illustrating the determination of the rotation matrix and scaling factor according to an embodiment of this application. Figure 5 In this study, a two-layer MLP was constructed. The first layer of MLP was used to encode the 3D key points, and the second layer of MLP combined the encoded 3D key points with the viewpoint direction parameters and 3D deformation encoding to obtain the rotation matrix and scaling factor.

[0149] It should be noted that, in actual 3D reconstruction, 3D keypoints may require translation in addition to rotation. Therefore, in one possible implementation, the server can also determine a translation matrix. This translation matrix identifies the translations required for the 3D keypoints during the 3D reconstruction process. The server can determine the local transformation matrix of the 3D keypoints based on the rotation matrix, translation matrix, and scaling factor. The local transformation matrix can be expressed by the following formula:

[0150]

[0151] in, Represents the local transformation matrix; Represents the rotation matrix; S represents the translation matrix; t This represents the scaling factor.

[0152] In one possible implementation, regarding the aforementioned "transforming the target 3D model into a 3D modeling model of the object to be reconstructed in the target video frame using the rotation matrix and the scaling factor," a master transformation matrix can be further introduced as an intermediate parameter:

[0153] Since a 3D keypoint is any point in the camera space of the object to be reconstructed in the target video frame, the server can determine the principal transformation matrix of the object to be reconstructed in the target video frame using the rotation matrix and scaling factor corresponding to the 3D keypoint. The principal transformation matrix can be used to identify the transformations that the object to be reconstructed needs to undergo during the 3D reconstruction process. The principal transformation matrix can be expressed by the following formula:

[0154]

[0155] Among them, G t Represents the principal transformation matrix; g t represents the hidden pose encoding, which is a learned hidden encoding vector without actual physical definition, obtained by the convolutional network acting on the target video frame; MLP represents a fully connected network layer, whose input g is a network layer. t ;R t Let R represent the principal rotation matrix. t ∈SO(3), the main rotation matrix can be obtained from the rotation matrix. To achieve the matrix transformation between the local rotation matrix and the main rotation matrix, the main rotation matrix can be approximated by considering the local rotation matrices of all 3D keypoints, and the Frobenius criterion can be used to penalize the deviation of the optimal rotation matrix. That is, the main rotation matrix is ​​determined by seeking a local rotation matrix that satisfies the preset conditions; T t T represents the principal translation matrix. t ∈R3 You can directly use the translation matrix as the main translation matrix.

[0156] It should be noted that in the process of 3D reconstruction of different objects to be reconstructed, the scaling factor of each 3D key point is actually learned. Since the scaling factor is established on the 3D field, it is continuous with respect to adjacent coordinate points. Therefore, the target 3D model used for 3D reconstruction will not fail due to the large individual differences between different objects to be reconstructed.

[0157] The server can transform the target 3D model into a 3D modeling model of the object to be reconstructed in the corresponding target video frame based on the 3D key points and the principal transformation matrix of the object. The 3D modeling model is the model obtained after 3D reconstruction of the object in the target video frame. For example, see... Figure 3 The following formula can be used to reconstruct 3D keypoints based on the principal transformation matrix:

[0158] x * =M t→* (G t x t )

[0159] Where, x * This represents the coordinates of key points in 3D reconstruction. Key points in 3D reconstruction are points on the 3D model that correspond to key points in the 3D model; M t→* This represents the mapping from camera space to canonical space; G t Denotes the principal transformation matrix; x t Represents the coordinates of 3D key points.

[0160] Using the above formula, 3D key points can be reconstructed based on the master transformation matrix. The 3D key points are points of the object to be reconstructed in the camera space, thereby transforming the target 3D model into the 3D modeling model of the object to be reconstructed in the corresponding target video frame.

[0161] It should be noted that, for multiple video frames included in the target video, the method provided in this embodiment can be used to determine the 3D modeling model corresponding to the object to be reconstructed in each video frame, thereby obtaining a 3D modeling model that dynamically changes with the target video.

[0162] In the process of 3D reconstruction of different objects to be reconstructed in the target video, a scaling factor can be used to process the shared target 3D model so that the 3D model corresponding to different objects to be reconstructed can reflect the individual size differences.

[0163] The three-dimensional reconstruction method provided in this application embodiment can quickly and effectively convert the target three-dimensional model into a three-dimensional modeling model of various shapes. When the object to be reconstructed is a face, the face and the target three-dimensional model can be mapped into the same three-dimensional space. Based on the distance between the three-dimensional key points of the face and the three-dimensional mapping points, the rotation matrix and scaling factor corresponding to the face are determined. The target three-dimensional model is then transformed into a three-dimensional modeling model of the face according to the rotation matrix and scaling factor, and the three-dimensional modeling model is used as the three-dimensional face model of the face.

[0164] This 3D face model can be used for 3D face attacks. For example, it can be used to generate input for various face recognition models under test, and then the recognition performance of different face recognition models can be tested based on the recognition results, such as whether it can correctly identify fake face attacks. Alternatively, the 3D face model can be mixed with other real faces and used for face recognition to determine the accuracy and correctness of the face recognition model.

[0165] Furthermore, by pre-creating a large number of templates for objects to be reconstructed, mass reconstruction can be achieved based on the target 3D model. For example, mass reconstruction of 3D face models for different faces can improve the effectiveness of face attack detection in a short time and test the stability of various face recognition models.

[0166] As can be seen from the above technical solution, for a target video containing N objects of the same type to be modeled (N>1), in order to obtain a target 3D model that can be shared by the N objects to be modeled, an initial 3D model shared by the N objects to be modeled is first generated based on the target video. Then, for the target video frame in the target video, on the imaging plane of the video frame, M object images corresponding to the M objects to be modeled in the target video frame are determined, where 2≤M≤N. The key points in the initial 3D model are mapped to the M object images, and the visibility probability of the key points in the two-dimensional plane points of the M object images is determined. Taking the target object among the M objects to be modeled as an example, the target object corresponds to the target image in the M object images, and the two-dimensional plane points mapped by the key points in the target image are the target two-dimensional plane points. Since the visibility probability of the target two-dimensional plane points can indicate the probability that the target two-dimensional plane points are not occluded in the target video frame, the visibility probability of the target two-dimensional plane points can indicate the probability that the target object is not occluded in the target video frame. In other words, the visibility probabilities corresponding to the M object images can each identify the probability that the object to be modeled for each object image is not occluded in the target video frame. Since occluded objects will affect the quality of the initial 3D model and the accuracy of its key points, the key points of the initial 3D model can be corrected based on the visibility probabilities of the M object images corresponding to the target video frame, resulting in a more accurate target 3D model. Therefore, the adverse effects of occlusion between multiple objects on multi-object 3D modeling can be reduced, improving the quality of the generated target 3D model.

[0167] Figure 6 This is a schematic block diagram of an object processing apparatus provided in an embodiment of this application. Figure 6 As shown, corresponding to the above object processing method, this application also provides an object processing apparatus 600. The object processing apparatus 600 is used to manage microservice links, the microservice links including multiple microservices, and each microservice including at least two versions. The object processing apparatus 600 includes a transceiver module 601 and a processing module 602, wherein:

[0168] Transceiver module 601 is used for:

[0169] Obtain the target video containing N objects to be modeled;

[0170] Processing module 602 is used for:

[0171] An initial 3D model is generated based on the target video. The target video includes N objects of the same type to be modeled. The N objects to be modeled share the initial 3D model, where N>1.

[0172] For a target video frame, determine M object images of M objects to be modeled in the target video frame on the video frame imaging plane. The M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N.

[0173] Keypoints in the initial target 3D model are mapped to M object images, and the visibility probability of each keypoint in a 2D plane point within each object image is determined. The target images in the M object images include the target object, and the target 2D plane points in the target images correspond to the keypoints. The visibility probability of the target 2D plane points is used to indicate the probability that the target object's 2D plane points are not occluded in the target video frame.

[0174] The target 3D model is obtained by correcting the corresponding key points in the initial target 3D model based on the visibility probability.

[0175] As one possible implementation, the object reconstruction apparatus further includes a storage module for:

[0176] The storage module is used to store the data required to perform the 3D reconstruction method, such as target videos and object images.

[0177] The transceiver module 601 is also used for:

[0178] Output the target 3D model.

[0179] As one possible implementation, processing module 602 is specifically used for:

[0180] By using a visual prediction model, key points in the initial 3D model are mapped to M object images, and the visibility probability of each key point in a 2D plane point in each object image is determined.

[0181] The visual prediction model is trained in the following way:

[0182] Obtain sample videos for training, and an initial sample 3D model generated based on Q sample objects included in the sample videos. The sample videos include multiple sample video frames, and the Q sample objects are of the same type as the N objects to be modeled, where Q>1.

[0183] For a target sample video frame of a sample video, determine the P sample object images of P sample objects in the target sample video frame on the video frame imaging plane. The P sample object images correspond one-to-one with the P sample objects, and 2≤P≤Q.

[0184] Using an initial visual prediction model, keypoints in the initial 3D model are mapped to P sample object images. The visual prediction probabilities of the keypoints in the P sample object images are determined. The target sample images in the P sample object images include target sample objects. Target sample plane points in the target sample images correspond to keypoints. The visual prediction probability of the target sample plane points is used to indicate the probability that the target sample plane points of the target sample object are not occluded in the target sample video frame.

[0185] Based on the difference between the visual prediction probability and the visual label of the target sample plane point, a loss function is constructed, and the initial visual prediction model is trained based on the loss function to obtain the visual prediction model. The visual label is used to identify whether the target sample plane point is occluded.

[0186] As one possible implementation, processing module 602 is specifically used for:

[0187] Based on the occlusion status indicated by the visual tags, the difference weights of the corresponding target sample plane points are determined. Specifically, when the target sample plane point indicated by the visual tag is not occluded, the corresponding first difference weight is a first value; when the target sample plane point indicated by the visual tag is occluded, the corresponding second difference weight is a second value, and the first value is greater than the second value.

[0188] A loss function is constructed based on the difference between the visual prediction probability and the visual label of the target sample plane point, as well as the corresponding difference weight.

[0189] In one possible implementation, the first value is a hyperparameter, and the second value is used to adjust the model based on the loss function during training.

[0190] In one possible implementation, processing module 602 is also used for:

[0191] Determine the semantic mask for P sample objects in the target sample video frame. The semantic mask is used to identify the pixel range of the P sample objects; and

[0192] Based on the semantic masks of P sample objects, visual labels are determined for pixels in the corresponding P sample object images. In the target sample image, for pixels within the semantic mask of the target sample object, the corresponding visual label is "not occluded," and for pixels not within the semantic mask of the target sample object, the corresponding visual label is "occluded."

[0193] In one possible implementation, processing module 602 is specifically used for:

[0194] For a target two-dimensional plane point, multiple sampling points are determined based on the ray from the camera's acquisition point in the target video frame to the target two-dimensional plane point; and

[0195] The visibility probability of the target two-dimensional plane point is determined based on the sub-visibility probabilities corresponding to multiple sampling points.

[0196] In one possible implementation, processing module 602 is also used for:

[0197] The object to be reconstructed is determined from the M objects to be modeled in the target video frame;

[0198] Determine the distance between the 3D key points of the object to be reconstructed in the camera space of the target video and the 3D mapping points in the camera space. The 3D mapping points are the mapping positions of the specified key points in the target 3D model in the camera space.

[0199] The rotation matrix and scaling factor used to adapt to differences in object size are determined based on the distance;

[0200] The target 3D model is transformed into a 3D modeling model of the object to be reconstructed in the corresponding target video frame by using the rotation matrix and the scaling factor.

[0201] One possible implementation is to designate the key point as the center point of the target 3D model.

[0202] The object processing apparatus in the embodiments of this application has been described above from the perspective of modular functional entities. The object processing apparatus in the embodiments of this application will be described below from the perspective of hardware processing.

[0203] It should be noted that in the embodiments of this application (including...) Figure 6 In the various embodiments shown, the physical devices corresponding to all transceiver modules can be Figure 7 The transceiver shown can have physical devices corresponding to all processing modules. Figure 7 The processor shown.

[0204] When it has such Figure 7 In the structure shown, the processor, transceiver, and memory implement the same or similar functions as the transceiver module and the processing module provided in the aforementioned device embodiments corresponding to this device. Figure 7 The computer program that needs to be called when the memory executes the above object processing methods.

[0205] Figure 6 The system shown can have, for example Figure 7 The structure shown, when Figure 6 The device shown has the following characteristics: Figure 7 When the structure shown is used, Figure 7The processor in the device can perform the same or similar functions as the processing module provided in the aforementioned device embodiments. Figure 7 The transceiver in the device can perform the same or similar functions as the transceiver module provided in the aforementioned device embodiments corresponding to this device. Figure 7 The memory stores the computer programs that the processor needs to call when executing the above-described image information recognition method. (In this application embodiment) Figure 6 In the illustrated embodiment, the physical device corresponding to the transceiver module can be an input / output interface, and the physical device corresponding to the processing module can be a processor.

[0206] This application also provides a terminal device, such as... Figure 8 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal device can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0207] Figure 8 This diagram illustrates a partial structure of a mobile phone related to the terminal device provided in the embodiments of this application. (Reference) Figure 8 The mobile phone includes: a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (Wi-Fi) module 570, a processor 580, and a power supply 590, among other components. Those skilled in the art will understand that... Figure 8 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0208] The following is combined with Figure 8 A detailed introduction to each component of a mobile phone:

[0209] RF circuit 510 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 580; additionally, it transmits uplink data to the base station. Typically, RF circuit 510 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 510 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0210] The memory 520 can be used to store software programs and modules. The processor 580 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0211] The input unit 530 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 531), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 580, and can also receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 531, the input unit 530 may also include other input devices 532. Specifically, other input devices 532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0212] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. Display unit 540 may include display panel 541, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 541. Further, touch panel 531 may cover display panel 541. When touch panel 531 detects a touch operation on or near it, it transmits the information to processor 580 to determine the type of touch event. Subsequently, processor 580 provides corresponding visual output on display panel 541 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 531 and the display panel 541 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the mobile phone.

[0213] The mobile phone may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 541 according to the ambient light level, and the proximity sensor can turn off the display panel 541 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0214] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface between the user and the mobile phone. Audio circuit 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. On the other hand, microphone 562 converts collected sound signals into electrical signals, which are received by audio circuit 560, converted into audio data, and then output to processor 580 for processing. The audio data is then transmitted via RF circuit 510 to, for example, another mobile phone, or output to memory 520 for further processing.

[0215] Wi-Fi is a short-range wireless transmission technology. Through the Wi-Fi module 570, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 8 The Wi-Fi module 570 is shown, but it is understood that it is not a necessary component of the mobile phone and can be omitted as needed without changing the nature of the application.

[0216] The processor 580 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520. Optionally, the processor 580 may include one or more processing units; preferably, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0217] The mobile phone also includes a power supply 590 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0218] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0219] In this embodiment of the application, the processor 580 included in the mobile phone also has the function of controlling and executing the above-mentioned... Figure 2 The flowchart shown illustrates the object processing method.

[0220] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 620 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 622 (e.g., one or more processors) and memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 622 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the server 620.

[0221] Server 620 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658, and / or one or more operating systems 641, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0222] The steps performed by the server in the above embodiments can be based on this Figure 9 The structure of server 620 is shown. For example, in the above embodiment, it consists of... Figure 2 The steps of the server shown can be based on this Figure 9 The server architecture is shown. For example, the processor 622 performs the following operations by calling instructions in memory 632:

[0223] An initial 3D model is generated based on the target video, wherein the target video includes N objects of the same type to be modeled, and the N objects to be modeled share the initial 3D model, where N>1;

[0224] For a target video frame, determine M object images of M objects to be modeled in the target video frame on the video frame imaging plane. The M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N.

[0225] Keypoints in the initial 3D model are mapped to M object images, and the visibility probabilities of the keypoints as 2D planar points in each object image are determined. The object images in the M object images include the target object, and the target 2D planar points in the target images correspond to the keypoints. The visibility probabilities of the target 2D planar points are used to indicate the probability that the target object's 2D planar points are not occluded in the target video frame.

[0226] The target 3D model is obtained by correcting the corresponding key points in the initial 3D model based on the visibility probability.

[0227] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0228] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0229] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0230] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0231] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0232] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0233] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0234] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. An object processing method, characterized in that, include: An initial 3D model is generated based on the target video, wherein the target video includes N objects of the same type to be modeled, and the N objects to be modeled share the initial 3D model, where N>1; For the target video frame of the target video, determine M object images of M objects to be modeled in the target video frame on the video frame imaging plane, wherein the M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N; Key points in the initial 3D model are mapped to the M object images, and the visibility probability of each key point as a 2D planar point in each object image is determined. The target images in the M object images include target objects, and the target 2D planar points in the target images correspond to the key points. The visibility probability of the target 2D planar points is used to indicate the probability that the target 2D planar points of the target object are not occluded in the target video frame. Based on the visibility probability, the corresponding key points in the initial 3D model are corrected to obtain the target 3D model; The step of mapping the key points in the initial 3D model to the M object images and determining the visibility probability of the key points as two-dimensional plane points in each object image includes: mapping the key points in the initial 3D model to the M object images through a visibility prediction model and determining the visibility probability of the key points as two-dimensional plane points in each object image. The visual prediction model is trained in the following way: Obtain sample videos for training, and an initial sample 3D model generated based on Q sample objects included in the sample videos, wherein the sample videos include multiple sample video frames, and the Q sample objects are of the same object type as the N objects to be modeled, and Q>1; For a target sample video frame of the sample video, determine P sample object images of P sample objects in the target sample video frame on the video frame imaging plane, wherein the P sample object images correspond one-to-one with the P sample objects, and 2≤P≤Q; Using an initial visual prediction model, the sample keypoints in the initial sample 3D model are mapped to the P sample object images, and the visual prediction probabilities of the sample keypoints as 2D plane points in the P sample object images are determined. The target sample images in the P sample object images include target sample objects, and the target sample plane points in the target sample images correspond to the sample keypoints. The visual prediction probability of the target sample plane points is used to identify the probability that the target sample plane points of the target sample objects are not occluded in the target sample video frame. Based on the difference between the visual prediction probability and the visual label of the target sample plane point, a loss function is constructed, and the initial visual prediction model is trained based on the loss function to obtain the visual prediction model. The visual label is used to identify whether the target sample plane point is occluded.

2. The method according to claim 1, characterized in that, The step of constructing a loss function based on the difference between the visual prediction probability and the visual label of the target sample plane point includes: Based on the occlusion status indicated by the visual tag, a difference weight corresponding to the target sample plane point is determined, wherein when the visual tag indicates that the target sample plane point is not occluded, the corresponding first difference weight is a first value; when the visual tag indicates that the target sample plane point is occluded, the corresponding second difference weight is a second value, and the first value is greater than the second value; and A loss function is constructed based on the difference between the visual prediction probability and the visual label of the target sample plane point, and the corresponding difference weight.

3. The method according to claim 2, characterized in that, The first value is a hyperparameter, and the second value is used to adjust the model based on the loss function during training.

4. The method according to claim 1, characterized in that, For the target sample video frame, the visual tag is determined in the following way: Determine the semantic mask for the P sample objects in the target sample video frame, wherein the semantic mask is used to identify the pixel range of the P sample objects; as well as Based on the semantic masks of the P sample objects, visual labels for pixels are determined in the corresponding P sample object images. In the target sample image, for pixels within the semantic mask of the target sample object, the corresponding visual label is "not occluded," and for pixels not within the semantic mask of the target sample object, the corresponding visual label is "occluded." 5. The method according to claim 1, characterized in that, For the target two-dimensional planar point, the step of mapping the key points in the initial three-dimensional model to the M object images and determining the visibility probability of the key points as two-dimensional planar points in each object image includes: Based on the ray from the camera acquisition point of the target video frame to the target two-dimensional plane point, multiple sampling points for the target two-dimensional plane point are determined; and The visibility probability of the target two-dimensional plane point is determined based on the sub-visibility probabilities corresponding to the multiple sampling points.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The object to be reconstructed is determined from the M objects to be modeled in the target video frame; Determine the distance between the three-dimensional key points of the object to be reconstructed in the camera space of the target video and the three-dimensional mapping points in the camera space, wherein the three-dimensional mapping points are the mapping positions of specified key points in the target three-dimensional model in the camera space; The rotation matrix and scaling factor for adapting to differences in object size are determined based on the distance. The target 3D model is transformed into a 3D modeling model of the object to be reconstructed in the target video frame using the rotation matrix and the scaling factor.

7. The method according to claim 6, characterized in that, The designated key point is the center point of the 3D model.

8. An object processing apparatus, characterized in that, The device includes a transceiver module and a processing module: The transceiver module is used for: Acquire the target video; The processing module is used for: An initial 3D model is generated based on the target video. The target video includes N objects of the same type to be modeled. The N objects to be modeled share the initial 3D model, where N>1. For the target video frame of the target video, determine M object images of M objects to be modeled in the target video frame on the video frame imaging plane, wherein the M object images correspond one-to-one with the M objects to be modeled, and 2≤M≤N; Key points in the initial 3D model are mapped to the M object images, and the visibility probability of each key point as a 2D planar point in each object image is determined. The target images in the M object images include target objects, and the target 2D planar points in the target images correspond to the key points. The visibility probability of the target 2D planar points is used to indicate the probability that the target 2D planar points of the target object are not occluded in the target video frame. Based on the visibility probability, the corresponding key points in the initial 3D model are corrected to obtain the target 3D model; Specifically, the processing module is used for: By using a visual prediction model, key points in the initial 3D model are mapped to M object images, and the visibility probability of each key point in a 2D plane point in each object image is determined. The visual prediction model is trained in the following way: Obtain sample videos for training, and an initial sample 3D model generated based on Q sample objects included in the sample videos. The sample videos include multiple sample video frames, and the Q sample objects are of the same type as the N objects to be modeled, where Q>1. For a target sample video frame of a sample video, determine the P sample object images of P sample objects in the target sample video frame on the video frame imaging plane. The P sample object images correspond one-to-one with the P sample objects, and 2≤P≤Q. Using an initial visual prediction model, keypoints in the initial 3D model are mapped to P sample object images. The visual prediction probabilities of the keypoints in the P sample object images are determined. The target sample images in the P sample object images include target sample objects. Target sample plane points in the target sample images correspond to keypoints. The visual prediction probability of the target sample plane points is used to indicate the probability that the target sample plane points of the target sample object are not occluded in the target sample video frame. Based on the difference between the visual prediction probability and the visual label of the target sample plane point, a loss function is constructed, and the initial visual prediction model is trained based on the loss function to obtain the visual prediction model. The visual label is used to identify whether the target sample plane point is occluded.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images