Model generation method, model generation device, electronic device, storage medium

Abstract

The present disclosure provides a model generation method, a model generation device, an electronic device and a computer readable storage medium, and belongs to the technical field of artificial intelligence. The method comprises: acquiring a plurality of frames of to-be-processed images, each frame of to-be-processed image comprising a plurality of human bodies; estimating the positional relationship of the plurality of human bodies in each frame of to-be-processed image to obtain the multi-person positional relationship information of each frame of to-be-processed image; determining the position of the human body grid point in each frame of to-be-processed image, and determining the human body shape information of each frame of to-be-processed image based on the human body grid sequence formed by the position of the human body grid point in the plurality of frames of to-be-processed images; extracting the human body implicit feature of each frame of to-be-processed image; and generating a three-dimensional human body model of the human body based on the multi-person positional relationship information of each frame of to-be-processed image, the human body shape information of each frame of to-be-processed image and the human body implicit feature of each frame of to-be-processed image. The present disclosure can effectively and accurately construct a human body model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model generation method, a model generation apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] 3D human body modeling is one of the core research problems in computer vision and graphics, and single-view 3D human body reconstruction technology has already found some applications. For example, when service robots interact with humans, they need to estimate the 3D human body model of the interacting object from their own perspective; augmented reality applications also need to estimate the 3D model of the target human body from their own perspective. Since single-view images can only observe the frontal surface of the human body, reconstructing the 3D model requires prior knowledge of human geometry. A common method is to use an RGB-D (color point cloud) camera to capture color and depth images from a single viewpoint, and then fit a statistically based parametric human body model to the RGB-D image. However, when objects are occluded, the small visible area of ​​the human body leads to insufficient model fitting, and the image information of the occluded object can also cause confusion, making the above methods unreliable for single-view reconstruction. Figure 3 Human reconstruction. Therefore, the application of the above methods has limitations.

[0003] In existing 3D reconstruction techniques, the methods for solving the occlusion problem fall into two categories: parametric and non-parametric models. Parametric methods store the surface vertices of a parametric human model in a material map and simultaneously train an encoder-decoder network to repair defects in the material map caused by occlusion. Non-parametric methods, on the other hand, require pre-obtaining a 3D shape template of the human body in the scene, and then combining scene flow, segmentation map, and lighting information to simultaneously reconstruct the 3D shapes of multiple people. This method is only suitable for scenes with minimal or no mutual occlusion; when occlusion between people in the image is severe, the 3D shape reconstruction results are less than ideal.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a model generation method, a model generation device, an electronic device, and a computer-readable storage medium, thereby overcoming, to at least a certain extent, the problem of poor generation effect of 3D human body models in multi-person scenarios in the prior art.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, a model generation method is provided, comprising: acquiring multiple frames of images to be processed, each frame of the images to be processed including multiple human bodies; estimating the positional relationships of the multiple human bodies in each frame of the images to be processed to obtain positional relationship information of the multiple human bodies in each frame of the images to be processed; determining the positions of human body mesh points in each frame of the images to be processed, and determining human body morphology information of each frame of the images to be processed based on a human body mesh sequence formed by the positions of the human body mesh points in the multiple frames of the images to be processed; extracting implicit human body features of each frame of the images to be processed; and generating a three-dimensional human body model of the human body based on the positional relationship information of the multiple human bodies in each frame of the images to be processed, the human body morphology information of each frame of the images to be processed, and the implicit human body features of each frame of the images to be processed.

[0008] In one exemplary embodiment of this disclosure, estimating the positional relationships of multiple human bodies in each frame of the image to be processed to obtain the positional relationship information of multiple human bodies in each frame of the image to be processed includes: extracting image features from each frame of the image to be processed using a pre-trained residual network; and fitting the positional relationships of human bodies to the image features of each frame of the image to be processed using a pre-trained multilayer perceptron to obtain the positional relationship information of multiple human bodies in each frame of the image to be processed.

[0009] In one exemplary embodiment of this disclosure, determining the position of human grid points in each frame of the image to be processed, and determining the human morphology information of each frame of the image to be processed based on the human grid sequence formed by the positions of human grid points in the multiple frames of the image to be processed, includes: extracting human grid points and features from each frame of the image to be processed; generating the human grid map based on the human grid points and the features; processing the human grid map using a pre-trained first graph neural network to obtain the positions of the human grid points; forming a human grid sequence from the positions of the human grid points in the multiple frames of the image to be processed; and processing the human grid sequence using a pre-trained recurrent neural network to obtain the human morphology information of each frame of the image to be processed.

[0010] In one exemplary embodiment of this disclosure, the step of extracting the implicit human features of each frame of the image to be processed includes: performing human target detection on each frame of the image to be processed, extracting a local human image from each frame of the image to be processed based on the detection result, and extracting implicit features from the local human image corresponding to each frame of the image to be processed to obtain the implicit human features of each frame of the image to be processed.

[0011] In an exemplary embodiment of this disclosure, the step of extracting implicit features from the human body local image corresponding to each frame of the image to be processed includes: extracting image features, texture features, and edge features from the human body local image; fusing the image features, texture features, and edge features of the human body local image to obtain the appearance features of the human body local image; and performing fully connected processing and activation processing on the appearance features of the human body local image to obtain the implicit features of the human body local image.

[0012] In one exemplary embodiment of this disclosure, generating a three-dimensional human body model based on the multi-person positional relationship information, human body morphology information, and implicit human body features of each frame of the image to be processed includes: generating a positional relationship map based on the multi-person positional relationship information, human body morphology information, and implicit human body features of each frame of the image to be processed; processing the positional relationship map using a pre-trained second graph neural network to obtain human body pose information of each frame of the image to be processed; optimizing the human body pose information of each frame of the image to be processed according to the implicit human body features of each frame of the image to be processed; and generating a three-dimensional human body model based on the optimization results.

[0013] In an exemplary embodiment of this disclosure, generating a positional relationship map based on the multi-person positional relationship information, the human body morphology information, and the implicit human body features of each frame of the image to be processed includes: generating human body nodes and human body position nodes based on the multi-person positional relationship information and the human body morphology information of each frame of the image to be processed; determining the features of the human body nodes, the features of the human body position nodes, and the edge features between the human body nodes according to the multi-person positional relationship information, the human body morphology information, and the implicit human body features of each frame of the image to be processed; initializing the edge features between the human body nodes and the position nodes; and generating the positional relationship map.

[0014] According to one aspect of this disclosure, a model generation apparatus is provided, comprising: a to-be-processed image acquisition module, configured to acquire multiple frames of to-be-processed images, each frame of the to-be-processed image including multiple human bodies; a positional relationship estimation module, configured to estimate the positional relationship of the multiple human bodies in each frame of the to-be-processed image to obtain positional relationship information of the multiple human bodies in each frame of the to-be-processed image; a morphological information determination module, configured to determine the position of human body grid points in each frame of the to-be-processed image, and determine the human body morphological information of each frame of the to-be-processed image based on the human body grid sequence formed by the positions of the human body grid points in the multiple frames of the to-be-processed image; an implicit feature extraction module, configured to extract implicit human body features from each frame of the to-be-processed image; and a human body model generation module, configured to generate a three-dimensional human body model of the human body based on the positional relationship information of the multiple human bodies in each frame of the to-be-processed image, the human body morphological information in each frame of the to-be-processed image, and the implicit human body features in each frame of the to-be-processed image.

[0015] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods by executing the executable instructions.

[0016] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0017] The exemplary embodiments disclosed herein have the following beneficial effects:

[0018] The process involves acquiring multiple frames of images to be processed, each containing multiple human figures; estimating the positional relationships of these human figures in each frame to obtain positional relationship information for each frame; determining the positions of human body mesh points in each frame and, based on the human body mesh sequence formed by the positions of these mesh points across multiple frames, determining the human body morphology information for each frame; extracting implicit human body features from each frame; and generating a 3D human body model based on the positional relationship information for each human figure, the human body morphology information, and the implicit human body features from each frame. On the one hand, this exemplary embodiment proposes a new model generation method. Compared with the existing technology that uses human pose information from a single image to construct a model, this exemplary embodiment takes into account the positional relationship between human bodies and implicit feature information in complex scenes. It generates a three-dimensional human body model by using multiple dimensions of information, including positional relationship information, human body shape information, and implicit human features of multiple human bodies in an image, resulting in higher accuracy and precision of the generated three-dimensional human body model. On the other hand, this exemplary embodiment can generate an accurate and effective three-dimensional human body model in multi-person scenes using multiple frames of images to be processed, overcoming the problem that the existing technology does not achieve ideal results in constructing three-dimensional human body models in multi-person scenes.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 This schematically illustrates a flowchart of a model generation method in this exemplary embodiment;

[0022] Figure 2 A sub-flowchart of a model generation method in this exemplary embodiment is illustrated schematically.

[0023] Figure 3 This schematically illustrates a flowchart of implicit feature extraction in this exemplary embodiment;

[0024] Figure 4 A flowchart illustrating another model generation method in this exemplary embodiment is shown schematically.

[0025] Figure 5This schematic diagram illustrates a structural block diagram of a model generation apparatus in this exemplary embodiment.

[0026] Figure 6 An electronic device for implementing the above method is illustrated in this exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] The exemplary embodiments of this disclosure first provide a model generation method, which is described below in conjunction with the appendix. Figure 1 The exemplary embodiments will be further described as follows: Figure 1 As shown, the model generation method may include the following steps S110 to S150:

[0029] Step S110: Obtain multiple frames of images to be processed, each frame of which includes multiple human figures.

[0030] In this exemplary embodiment, the image to be processed refers to the image that needs to be processed. The image to be processed may include multiple human figures, that is, the image to be processed includes multiple human figures. For example, the image to be processed may be an image of a group photo of multiple people, or other images of multiple human figures captured in the image. The multiple human figures may be related, such as a group photo of multiple people who know each other; or they may be unrelated, such as images of multiple people randomly captured in a crowd.

[0031] In this exemplary embodiment, the image to be processed can be a series of continuous or discontinuous frames of images to be processed. Specifically, it can be obtained in various ways, such as by capturing a video stream through a camera and using each frame as the image to be processed, or by using some frames as the image to be processed, or by extracting images of multiple human bodies taken at different times from an album as the image to be processed, etc. This disclosure does not specifically limit this.

[0032] In an exemplary embodiment, step S110 may include:

[0033] Acquire a video stream containing multiple human bodies;

[0034] Frame extraction is performed on the video stream to determine multiple frames of images to be processed.

[0035] This exemplary embodiment can first acquire a video stream including multiple human bodies, for example, by capturing a video in real time using a camera, or by downloading a video from the network. Then, by performing frame extraction processing on the video stream, multiple frames of images to be processed can be determined. Frame extraction can be performed at preset time intervals, such as extracting one frame every 2 seconds as the image to be processed; or it can be performed by randomly extracting a preset number of image frames in chronological order as the images to be processed, etc. This disclosure does not specifically limit the specific methods used.

[0036] Step S120: Estimate the positional relationship of multiple human bodies in each frame of the image to be processed to obtain the positional relationship information of multiple people in each frame of the image to be processed.

[0037] In complex scenes, there are often relationships between multiple human figures and their relative positions. To leverage this characteristic and improve the accuracy of the initial estimation, this exemplary embodiment can estimate the positional relationships of multiple human figures in each frame of the image to be processed, thus determining the positional relationship information of the multiple individuals. This positional relationship information can include left-right / front-back positional relationships, coordinate relationships, occlusion relationships, and so on, between different human figures.

[0038] Specifically, this exemplary embodiment can process the image to be processed using a pre-trained positional relationship graph neural network to jointly refine the human posture and positional layout.

[0039] In an exemplary embodiment, step S120 described above may include:

[0040] Image features are extracted for each frame of the image to be processed using a pre-trained residual network;

[0041] By using a pre-trained multilayer perceptron to fit the positional relationships of human bodies to the image features of each frame of the image to be processed, the positional relationship information of multiple people in each frame of the image to be processed is obtained.

[0042] For estimating the positional relationships of multiple people, this exemplary embodiment first employs a pre-trained Residual Network (ResNet) to extract image features from the input image to be processed. These image features refer to data extracted from the image to characterize various aspects of image information, such as image structure, texture, color, and layout. This exemplary embodiment can represent the image as image features in vector or matrix form. Then, a Multilayer Perceptron (MLP) is used to process the image features of each frame of the image to be processed to fit the positional layout parameters of the 3D figures. Through the above process, this exemplary embodiment can learn the image context relationships from the image features provided by the initial estimation, and jointly refine the human pose and positional layout.

[0043] Step S130: Determine the position of the human body grid points in each frame of the image to be processed, and determine the human body morphology information of each frame of the image to be processed based on the human body grid sequence formed by the positions of the human body grid points in multiple frames of the image to be processed.

[0044] The position of human body mesh points can be characterized by the coordinates of the human body within the mesh point, or by data such as the mesh point's sequence number, label, or coordinates. By determining the position of the human body mesh points in each frame of the image to be processed, a human body mesh sequence formed by these mesh point positions can be first established. Further, based on this sequence, the human body morphology information for each frame of the image to be processed can be determined. Human body morphology information refers to indicator data that characterizes a person's form. This can include shape parameters related to the person's shape, such as height, weight, and clothing outlines, as well as pose parameters related to the person's state, such as position and posture. Human body morphology information can be used to construct three-dimensional human body models for each individual.

[0045] This exemplary embodiment can reflect the morphological state of the human body at the current frame by the position of the human body grid points in each frame of the image to be processed. Then, the human body grid sequence formed by the positions of the human body grid points in multiple frames of the image to be processed can characterize the motion change state of the human body in the corresponding time series of multiple frames of the image to be processed.

[0046] In one exemplary embodiment, such as Figure 2 As shown, step S130 above may include the following steps:

[0047] Step S210: Extract human body grid points and human body grid point features from each frame of the image to be processed, and generate a human body grid map based on the human body grid points and human body grid point features.

[0048] Step S220: Process the human body grid map using the pre-trained first graph neural network to obtain the positions of the human body grid points;

[0049] Step S230: The positions of human body grid points in multiple frames of images to be processed are used to form a human body grid sequence;

[0050] Step S240: The human body grid sequence is processed using a pre-trained recurrent neural network to obtain human body morphology information for each frame of the image to be processed.

[0051] To achieve 3D human reconstruction based on video sequences, this exemplary embodiment proposes using a combination of graph convolutional networks and recurrent neural networks, such as GRU (Gated Recurrent Unit), to regress human body mesh coordinates. This determines the human body morphology information for each frame of the image to be processed.

[0052] Specifically, this exemplary embodiment can perform meshing processing on the image to be processed or the human body region of the image to determine the human body mesh region used to represent the human body. A human body mesh map is generated by extracting human body mesh points and their features. Here, human body mesh points can be vertices of the human body mesh. Extracting human body mesh points can be the process of determining human body mesh points, such as determining the sequence number or other identification information of the human body mesh vertices, and determining the feature information of the human body mesh vertices, such as the location of the mesh point, its own characteristics, or its relationship with the human body and background region, and encoding these features. Then, a human body mesh map can be generated based on the human body mesh points and their features. This human body mesh map refers to graph data used to represent the human body mesh points and their features. Further, a pre-trained first graph neural network is used to process the human body mesh map to obtain the positions of the human body mesh points, such as the three-dimensional coordinates of each human body mesh vertex; that is, the first graph neural network first uses the mesh topology to capture complete information of the three-dimensional human body. The positions of human body grid points in multiple frames of images to be processed are organized to form a human body grid sequence according to the time series order; finally, the human body grid sequence is used as input to a pre-trained recurrent neural network model to determine the human body morphology information of each frame of the image to be processed.

[0053] To accelerate the training speed of the neural network, this exemplary embodiment incorporates residual connections and group normalization into the graph convolutional layers. A GRU network is used to process time-series data to capture the sequential nature of human motion.

[0054] Step S140: Extract the implicit human features of each frame of the image to be processed.

[0055] Implicit human features refer to feature information other than the physical characteristics of a person, such as texture information of the area where the human body is located, edge features of the area where the human body meets the background in an image, or other feature information. Implicit human features can be used to supplement human features and improve the overall features of a person.

[0056] This exemplary embodiment can process the image to be processed using a neural network model to extract implicit features of a person, or it can encode the implicit data of a person using a specific coding network to extract implicit feature information of a person, etc.

[0057] In an exemplary embodiment, step S130 described above may include:

[0058] Human target detection is performed on each frame of the image to be processed, and local human images are extracted from each frame of the image to be processed based on the detection results.

[0059] Implicit features are extracted from the local human body image corresponding to each frame of the image to be processed, thus obtaining the implicit human body features of each frame of the image to be processed.

[0060] This exemplary embodiment can first employ a target detection network to perform human target detection on each frame of the image to be processed, and obtain detection results. The detection results can include whether each frame of the image to be processed contains a person, how many people are included, the location of each person (e.g., represented by a detection box), etc. Based on the detection results, local human images can be extracted from each frame of the image to be processed. For example, multiple people in the image to be processed can be cropped according to the detection boxes to extract local human images.

[0061] Furthermore, implicit features can be extracted from the image to be processed through a pre-configured local implicit embedding network to obtain the human implicit features of each frame of the image to be processed. These implicit features can be local shape information or other implicit relational features hidden in the image to be processed.

[0062] To generate the implicit features required for local depth implicit functions from image frames and achieve more challenging 3D object reconstruction tasks, this exemplary embodiment proposes a Local Implicit Embedding Network (LIE). To provide subsequent stages with 2D image information and 3D features of the human body, this exemplary embodiment uses a LIE-based method for human body reconstruction, combining structured and implicit representations to provide high 3D representation accuracy. The LIE can use VGG (a network model) as an image encoder and a five-layer MLP to encode implicit representation vectors, i.e., implicit features, which, combined with LIE decoding, determine the human body model. To effectively learn the implicit representation, the texture and edge feature encodings of the human body are concatenated with the VGG image features in the form of embedding encoding, introducing prior information about the human body shape into the network to improve reconstruction accuracy.

[0063] In an exemplary embodiment, the extraction of implicit features from the local human body image corresponding to each frame of the image to be processed may include:

[0064] Extracting image features, texture features, and edge features from local human body images;

[0065] By fusing image features, texture features, and edge features of a local human body image, the appearance features of the local human body image are obtained.

[0066] By performing fully connected processing and activation processing on the appearance features of local human body images, implicit features of local human body images are obtained.

[0067] Image features can include color features, texture features, shape features, and spatial relationship features, etc. Texture features can be texture data of human body regions, and edge features can refer to human body contour features or features of the area where the human body meets the background, such as color changes, brightness changes, or positional changes. This exemplary embodiment can extract image features, texture features, and edge features from a local human body image using a pre-trained network model, and then fuse them to obtain the appearance features of the local human body image. Fusion can be achieved by concatenating the vectors of image features, texture features, and edge features, or by combining the vectors of image features, texture features, and edge features to generate a fusion matrix, etc. Finally, fully connected processing and activation processing are applied to the appearance features of the local human body image to obtain the implicit features of the local human body image.

[0068] Figure 3 The diagram illustrates a flowchart of implicit feature extraction in this exemplary embodiment, which may include: acquiring multiple frames of images to be processed 310; performing human target detection on the processed images; extracting human local images from each frame of images to be processed based on the detection results; for each frame of images to be processed corresponding to the human local image, extracting image features 330 of the human local image through a VGG image encoder 320, and extracting texture features and edge features 340 of the human local image; further, performing feature fusion processing 350 on the image features, texture features, and edge features of the human local image to obtain appearance features 360 of the human local image; finally, performing fully connected processing and activation processing 370 on the appearance features of the human local image to obtain implicit features 380 of the human local image.

[0069] Step S150: Based on the multi-person positional relationship information of each frame of the image to be processed, the human body morphology information of each frame of the image to be processed, and the implicit human body features of each frame of the image to be processed, a three-dimensional human body model is generated.

[0070] After determining the positional relationship information of multiple people, the human morphology information of each frame of the image to be processed, and the implicit human features of each frame of the image to be processed, a three-dimensional human body model can be generated. Specifically, the positional relationship information of multiple people, the human morphology information of each frame of the image to be processed, and the implicit human features of each frame of the image to be processed can be used together as input data for processing to generate a three-dimensional human body model. Alternatively, a preliminary three-dimensional human body model can be generated first based on the human morphology information of each frame of the image to be processed, and then supplemented and optimized by the implicit human features to generate the final three-dimensional human body model.

[0071] In an exemplary embodiment, step S150 may include:

[0072] Based on the location relationship information of multiple people in each frame of the image to be processed, the human body morphology information in each frame of the image to be processed, and the implicit human body features in each frame of the image to be processed, a location relationship map is generated.

[0073] The positional relationship map is processed using a pre-trained second graph neural network to obtain human pose information for each frame of the image to be processed.

[0074] The human pose information of each frame of the image to be processed is optimized based on the implicit human features of each frame, and a three-dimensional human model is generated based on the optimization results.

[0075] This exemplary embodiment generates a positional relationship map based on the positional relationship information of multiple people, the human morphology information of the image to be processed, and the implicit features of the human body. The positional relationship map is then processed by a second graph neural network to obtain the human pose information of each frame of the image to be processed. In other words, this exemplary embodiment uses the second graph neural network to refine and optimize the initially determined human morphology information of each frame of the image to be processed, obtaining optimized human pose information, such as the human body's position. Finally, based on the implicit features of each frame, the human pose information of each processed frame is further optimized, and a three-dimensional human body model is generated based on the optimization results.

[0076] In an exemplary embodiment, the above-mentioned generation of a positional relationship map based on the positional relationship information of multiple people in each frame of the image to be processed, the human body morphology information in each frame of the image to be processed, and the implicit human body features in each frame of the image to be processed includes:

[0077] Based on the multi-person positional relationship information and human body morphology information of each frame of the image to be processed, human body nodes and human body position nodes are generated.

[0078] Based on the multi-person positional relationship information, human body morphology information, and implicit human body features of each frame of the image to be processed, the features of human body nodes, the features of human body position nodes, and the edge features between human body nodes are determined. The edge features between human body nodes and position nodes are initialized to generate a positional relationship map.

[0079] In this exemplary embodiment, the entire input image to be processed can be modeled as a relational graph. Based on the multi-person positional relationship information and the human body morphology information of each frame of the image to be processed, human body nodes and human body position nodes in the graph can be generated. Specifically, the relational graph can be constructed from a complete graph consisting of object and layout nodes, with non-directional edges between each node, allowing information to flow between the human body and the layout of the human body position. In this exemplary embodiment, the second graph neural network can concatenate different features after flattening them into a vector, and then encode it into a representation vector of a certain length and embed it into the node. For the human body position node, its features may include features from the image encoder, camera pose parameters, and positional encoding features of the 3D human figure. For the human body node, its features may include features from image-relationship features, human 3D pose parameters, and deep implicit features; for relational nodes connecting human bodies, its features include geometric features extracted from each pair of human object detection boxes and normalized parameters of the 2D detection boxes; for relational nodes connecting human bodies and human body positions, the features are initialized to constants and change during the message update process of the second graph neural network. This exemplary embodiment can determine the features of human nodes, the features of human position nodes, and the edge features between human nodes based on the multi-person position relationship information, human morphology information, and implicit human features of each frame of the image to be processed. It can also initialize the edge features between human nodes and position nodes to generate a position relationship map.

[0080] Figure 4 This diagram illustrates another model generation method in this exemplary embodiment, which may specifically include the following steps:

[0081] Step S402: Obtain multiple frames of images to be processed; each frame of the image to be processed includes multiple human figures;

[0082] Step S404: Extract image features for each frame of the image to be processed using a pre-trained residual network.

[0083] Step S406: Use a pre-trained multilayer perceptron to fit the positional relationship of the human body to the image features of each frame of the image to be processed, and obtain the positional relationship information of multiple people in each frame of the image to be processed.

[0084] Step S408: Extract human body grid points and human body grid point features from each frame of the image to be processed, and generate a human body grid map based on the human body grid points and human body grid point features.

[0085] Step S410: Process the human body grid map using the pre-trained first graph neural network to obtain the positions of the human body grid points;

[0086] Step S412: The positions of human body grid points in multiple frames of images to be processed are used to form a human body grid sequence;

[0087] Step S414: The human body grid sequence is processed using a pre-trained recurrent neural network to obtain the human body morphology information of each frame of the image to be processed.

[0088] Step S416: Perform human target detection on each frame of the image to be processed, and extract local human images from each frame of the image to be processed based on the detection results.

[0089] Step S418: Extract implicit features from the local human body image corresponding to each frame of the image to be processed to obtain the implicit human body features of each frame of the image to be processed.

[0090] Step S420: Generate a positional relationship map based on the multi-person positional relationship information, human body morphology information, and implicit human body features of each frame of the image to be processed.

[0091] Step S422: The positional relationship map is processed using a pre-trained second graph neural network to obtain the human pose information of each frame of the image to be processed.

[0092] Step S424: Optimize the human pose information of each frame of the image to be processed based on the implicit human features of each frame, and generate a three-dimensional human model based on the optimization results.

[0093] Based on the above description, in this exemplary embodiment, multiple frames of images to be processed are acquired, each frame of the image to be processed including multiple human bodies; the positional relationship of the multiple human bodies in each frame of the image to be processed is estimated to obtain the positional relationship information of multiple human bodies in each frame of the image to be processed; the position of the human body grid points in each frame of the image to be processed is determined, and the human body morphology information of each frame of the image to be processed is determined based on the human body grid sequence formed by the position of the human body grid points in the multiple frames of the image to be processed; the implicit human body features of each frame of the image to be processed are extracted; and a three-dimensional human body model is generated based on the positional relationship information of multiple human bodies in each frame of the image to be processed, the human body morphology information of each frame of the image to be processed, and the implicit human body features of each frame of the image to be processed. On the one hand, this exemplary embodiment proposes a new model generation method. Compared with the existing technology that uses human pose information from a single image to construct a model, this exemplary embodiment takes into account the positional relationship between human bodies and implicit feature information in complex scenes. It generates a three-dimensional human body model by using multiple dimensions of information, including positional relationship information, human body shape information, and implicit human features of multiple human bodies in an image, resulting in higher accuracy and precision of the generated three-dimensional human body model. On the other hand, this exemplary embodiment can generate an accurate and effective three-dimensional human body model in multi-person scenes using multiple frames of images to be processed, overcoming the problem that the existing technology does not achieve ideal results in constructing three-dimensional human body models in multi-person scenes.

[0094] An exemplary embodiment of this disclosure also provides a model generation apparatus. (Refer to...) Figure 5 The device 500 may include: an image acquisition module 510 for acquiring multiple frames of images to be processed, each frame of the image to be processed including multiple human bodies; a positional relationship estimation module 520 for estimating the positional relationship of multiple human bodies in each frame of the image to be processed to obtain positional relationship information of multiple human bodies in each frame of the image to be processed; a morphological information determination module 530 for determining the position of human body grid points in each frame of the image to be processed, and determining the human body morphological information of each frame of the image to be processed based on the human body grid sequence formed by the position of human body grid points in multiple frames of the image to be processed; an implicit feature extraction module 540 for extracting implicit human body features in each frame of the image to be processed; and a human body model generation module 550 for generating a three-dimensional human body model based on the positional relationship information of multiple human bodies in each frame of the image to be processed, the human body morphological information in each frame of the image to be processed, and the implicit human body features in each frame of the image to be processed.

[0095] In an exemplary embodiment, the positional relationship estimation module includes: an image feature extraction unit, used to extract image features for each frame of the image to be processed using a pre-trained residual network; and a positional relationship fitting unit, used to fit the positional relationship of the human body to the image features of each frame of the image to be processed using a pre-trained multilayer perceptron, to obtain the positional relationship information of multiple people in each frame of the image to be processed.

[0096] In an exemplary embodiment, the morphological information determination module includes: a mesh map generation unit, configured to extract human mesh points and features from each frame of the image to be processed, and generate a human mesh map based on the human mesh points and features; a position determination unit, configured to process the human mesh map using a pre-trained first graph neural network to obtain the positions of the human mesh points; a sequence formation unit, configured to form a human mesh sequence from the positions of the human mesh points in multiple frames of the image to be processed; and a morphological information determination unit, configured to process the human mesh sequence using a pre-trained recurrent neural network to obtain human morphological information for each frame of the image to be processed.

[0097] In an exemplary embodiment, the implicit feature extraction module includes: a local image extraction unit, used to perform human target detection on each frame of the image to be processed, and extract human local images from each frame of the image to be processed based on the detection results; and an implicit feature acquisition unit, used to extract implicit features from the human local images corresponding to each frame of the image to be processed, and obtain the human implicit features of each frame of the image to be processed.

[0098] In an exemplary embodiment, the implicit feature acquisition unit includes: a feature extraction subunit for extracting image features, texture features, and edge features of a local human body image; a feature fusion subunit for fusing the image features, texture features, and edge features of the local human body image to obtain the appearance features of the local human body image; and a feature processing subunit for performing fully connected processing and activation processing on the appearance features of the local human body image to obtain the implicit features of the local human body image.

[0099] In an exemplary embodiment, the human body model generation module includes: a positional relationship map generation unit, configured to generate a positional relationship map based on the positional relationship information of multiple people in each frame of the image to be processed, the human body morphology information in each frame of the image to be processed, and the implicit human body features in each frame of the image to be processed; a human body pose acquisition unit, configured to process the positional relationship map using a pre-trained second graph neural network to obtain the human body pose information in each frame of the image to be processed; and a model generation unit, configured to optimize the human body pose information in each frame of the image to be processed based on the implicit human body features in each frame of the image to be processed, and generate a three-dimensional human body model based on the optimization result.

[0100] In an exemplary embodiment, the positional relationship map generation unit includes: a node determination subunit, configured to generate human nodes and human position nodes based on the positional relationship information of multiple people in each frame of the image to be processed and the human morphology information of each frame of the image to be processed; and a map generation subunit, configured to determine the features of human nodes, the features of human position nodes, and the edge features between human nodes based on the positional relationship information of multiple people in each frame of the image to be processed, the human morphology information of each frame of the image to be processed, and the implicit human features of each frame of the image to be processed, and initialize the edge features between human nodes and position nodes to generate a positional relationship map.

[0101] The specific details of each module / unit in the above-mentioned device have been described in detail in the embodiments of the method section. For any undisclosed details, please refer to the embodiments of the method section, and therefore will not be repeated here.

[0102] An exemplary embodiment of this disclosure also provides an electronic device capable of implementing the above-described method.

[0103] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0104] The following reference Figure 6 To describe an electronic device 600 according to such an exemplary embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0105] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0106] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 610 can execute... Figure 1 , Figure 2 , Figure 3 or Figure 4 The steps shown are as follows.

[0107] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0108] Storage unit 620 may also include a program / utility 624 having a set (at least one) of program modules 625, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0109] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0110] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0111] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.

[0112] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0113] Exemplary embodiments of this disclosure also provide a program product for implementing the above-described method, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0117] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0118] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0119] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0120] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0121] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A model generation method, characterized in that, include: Acquire multiple frames of images to be processed, each frame containing multiple human figures; The positional relationships of multiple human bodies in each frame of the image to be processed are estimated to obtain the positional relationship information of multiple human bodies in each frame of the image to be processed. The process involves determining the positions of human mesh points in each frame of the image to be processed, and forming a human mesh sequence based on the positions of these points to determine the human morphology information of each frame. This determination includes: extracting human mesh points and their features from each frame; generating a human mesh map based on the human mesh points and their features; processing the human mesh map using a pre-trained first graph neural network to obtain the positions of the human mesh points; forming a human mesh sequence from the positions of the human mesh points in the multiple frames; and processing the human mesh sequence using a pre-trained recurrent neural network to obtain the human morphology information of each frame. Extract the implicit human features of each frame of the image to be processed; Based on the location relationship information of multiple people in each frame of the image to be processed, the human body morphology information in each frame of the image to be processed, and the implicit human body features in each frame of the image to be processed, a three-dimensional human body model is generated.

2. The method according to claim 1, characterized in that, The step of estimating the positional relationships of multiple human figures in each frame of the image to be processed to obtain the positional relationship information of multiple human figures in each frame of the image to be processed includes: Image features are extracted from each frame of the image to be processed using a pre-trained residual network; By using a pre-trained multilayer perceptron to fit the positional relationships of human bodies to the image features of each frame of the image to be processed, the positional relationship information of multiple people in each frame of the image to be processed is obtained.

3. The method according to claim 1, characterized in that, The extraction of implicit human features from each frame of the image to be processed includes: Human target detection is performed on each frame of the image to be processed, and local human images are extracted from each frame of the image to be processed based on the detection results. Implicit features are extracted from the local human body image corresponding to each frame of the image to be processed to obtain the implicit human body features of each frame of the image to be processed.

4. The method according to claim 3, characterized in that, The step of extracting implicit features from the local human body image corresponding to each frame of the image to be processed includes: Extract image features, texture features, and edge features from the local human body image; By fusing the image features, texture features, and edge features of the local human body image, the appearance features of the local human body image are obtained. The appearance features of the local human body image are subjected to fully connected processing and activation processing to obtain the implicit features of the local human body image.

5. The method according to claim 1, characterized in that, The step of generating a three-dimensional human body model based on the multi-person positional relationship information, the human body morphology information, and the implicit human body features of each frame of the image to be processed includes: Based on the multi-person positional relationship information, the human body morphology information, and the implicit human body features of each frame of the image to be processed, a positional relationship map is generated. The positional relationship map is processed using a pre-trained second graph neural network to obtain human pose information for each frame of the image to be processed. The human pose information of each frame of the image to be processed is optimized based on the implicit human features of each frame, and a three-dimensional human model of the human body is generated based on the optimization results.

6. The method according to claim 5, characterized in that, The step of generating a positional relationship map based on the multi-person positional relationship information, the human body morphology information, and the implicit human body features of each frame of the image to be processed includes: Based on the multi-person positional relationship information and the human body morphology information of each frame of the image to be processed, human body nodes and human body position nodes are generated. Based on the multi-person positional relationship information of each frame of the image to be processed, the human body morphology information of each frame of the image to be processed, and the implicit human body features of each frame of the image to be processed, the features of the human body node, the features of the human body position node, and the edge features between the human body node and the human body node are determined, and the edge features between the human body node and the position node are initialized to generate the positional relationship map.

7. A model generation apparatus, characterized in that, include: The image acquisition module is used to acquire multiple frames of images to be processed, each frame of which includes multiple human figures; The positional relationship estimation module is used to estimate the positional relationship of multiple human bodies in each frame of the image to be processed, and to obtain the positional relationship information of multiple human bodies in each frame of the image to be processed. A morphological information determination module is used to determine the position of human body grid points in each frame of the image to be processed, and to determine the human body morphological information of each frame of the image to be processed based on the human body grid sequence formed by the positions of the human body grid points in the multiple frames of the image to be processed; the morphological information determination module is configured to: extract human body grid points and human body grid point features from each frame of the image to be processed, generate a human body grid map based on the human body grid points and the human body grid point features; and process the human body grid map using a pre-trained first graph neural network to obtain the position of the human body grid points; The positions of human body grid points in the multiple frames of images to be processed are used to form a human body grid sequence; the human body grid sequence is processed using a pre-trained recurrent neural network to obtain the human body morphology information of each frame of the image to be processed. The implicit feature extraction module is used to extract the implicit human features of each frame of the image to be processed. The human body model generation module is used to generate a three-dimensional human body model based on the multi-person positional relationship information of each frame of the image to be processed, the human body morphology information of each frame of the image to be processed, and the implicit human body features of each frame of the image to be processed.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images