Single-stage multi-person grid model construction method and system based on time series coordinate information

Abstract

The present invention discloses a single-stage multi-person grid model construction method and system based on temporal coordinate information. The method includes: obtaining a video of a real scene; extracting features from the video of the real scene through a neural network to predict a person center map, a camera parameter map, and a person parameter map; using the person center heat map to focus on the camera parameter map and the person parameter map, and performing spatial and temporal relationship modeling based on the neural network to model the spatio-temporal connection between person models, and predicting the final camera parameter map and person parameter map; finding the regions determined as people in the person center heat map, and indexing to the corresponding camera parameter map and person parameter map to generate a person model. The present invention avoids the computing power consumption caused by explicit detection and explicit tracking, realizes the simplicity, low computing power consumption, and accuracy of person model regression, and can be widely applied to the field of computer technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a single-stage multi-person mesh model construction method and system based on temporal coordinate information. Background Art

[0002] With the development and maturity of technologies such as the metaverse, virtual reality, augmented reality, and virtual fitting, there is an urgent need for a simple and general method for constructing character models to generate one's own independent image and participate in different application scenarios. In this context and challenge, the single-frame single-person mesh model regression method has achieved great results. They first use a detector to detect the area where the person is located in the picture, and then regress the corresponding 3D person mesh model on this area. However, real-world movements are continuous and consistent and exist in the form of videos. Therefore, there is a need for a video-based method to obtain a series of natural, accurate, and continuous person models while ensuring that no person models are missed due to missed detection by the detector.

[0003] Existing video-based methods usually follow the multi-stage system design specification. That is, before actually regressing the person model based on the cropped picture, a person detector and a person tracker are usually required for multi-person detection and single-person tracking for each person to obtain a sequence of cropped pictures for each person. This top-down design specification first disassembles the multi-person motion image sequence into single-person motion image sequences, and then extracts features based on single frames and fuses them with temporal information on this sequence to construct the result of the person mesh model motion sequence. However, the accuracy of this multi-stage design style depends heavily on the accuracy of the first stage and may cause error accumulation, making the first stage very sensitive to misdetection, missed detection, and overlapping of people. At the same time, since regression can only be performed based on the cropped pictures after the heavy first stage, this also greatly increases the computing power consumption of the video-based method, making it more difficult to meet the real-time requirement. Moreover, in multi-person sports videos such as basketball games and football games, the multi-stage video-based method roughly separates all people independently without considering the spatio-temporal interaction between people, which also reduces the prediction accuracy.

[0004] Therefore, in the context of the limited accuracy and speed of video-based methods, in order to obtain accurate and real multi-person motion information to generate a person mesh model, there is an urgent need for a multi-person mesh model regression system with fast running speed and considering person interaction, which uses the temporal information in the video to maintain the consistency and coordination of the person mesh.

[0005] The current human mesh model regression system can be roughly divided into three categories: self-designed, scanner-scanned, and image regression. Among them, the self-designed method mainly relies on users to model independently and refine the human model through 3D software. Scanner scanning requires the use of expensive scanners or scanning rooms and other equipment to collect human data to obtain a refined human model. Image regression is based on models such as deep neural networks to obtain human features and obtain the corresponding human model based on the human parameter model. The relevant technical solutions are as follows:

[0006] 1). MetaHuman Creator is a digital human personalized self-design and construction framework based on the Unreal Engine, which can combine mobile applications to capture facial parameters for assisting modeling.

[0007] 2). HP 3D strutured light scanner is a high-precision three-dimensional scan designed by HP. It restores the three-dimensional model of the object through the form of multi-angle scanning, but it needs to be corrected before use.

[0008] 3). PARE is a multi-stage image regression method based on a single image, that is, first obtain the bounding box of the person through a person detector, and use the image of this bounding box as the input to obtain the mesh model of the target person.

[0009] 4). ROMP is a single-stage image regression method based on a single image, that is, through a deep neural network model and an SMPL human parameter model, the human mesh model of all people can be obtained without a person detector.

[0010] 5). VIBE is a multi-stage image regression method based on video, that is, first obtain the bounding boxes of all people in each frame of the video through a person detector, then use a tracker to track the movement of the person in the video and obtain a sequence of bounding box images of the person as the input, then obtain the features of each frame of the person based on a deep neural network and construct temporal information, and finally obtain a sequence of mesh models of each person.

[0011] Regarding the above-mentioned existing technical solutions, it can be found that although the existing methods for constructing human mesh models can obtain corresponding human mesh models through different frameworks, devices, and neural network methods by inputting self-adjustment, inherent human appearance, human pictures, etc. However, the method based on self-design mainly meets the high-precision design requirements of commercialization, and the time required to obtain a human model is long; the method based on scanner scanning requires expensive scanners and laboratory environments, and the scanned models are not compatible with most mainstream software; the method based on image regression using neural network methods has a fixed model structure, and the existing single-image-based methods ignore the inherent temporal information of real movements. The existing video-based methods follow multi-stage design specifications, ignoring the spatio-temporal interaction of people and unable to meet the real-time operation requirements. To sum up, the problems of the above technical solutions are summarized as follows:

[0012] 1) It is difficult and expensive to obtain a human model; the methods based on self-design and scanner scanning have high operating costs and require expensive equipment support with a large investment in time cost;

[0013] 2) The prediction of the human model is inaccurate and the actions are uncoordinated; the single-image-based image regression method only focuses on the accuracy of a single frame and cannot utilize the temporality of real movements. In actual applications, it is easy to cause problems such as jitter and misalignment of the predicted human model;

[0014] 3) The prediction steps of the human model are numerous and it cannot be real-time; the video-based image regression method follows multi-stage design specifications, and the accuracy seriously depends on the detection and tracking in the first stage, which also causes error accumulation and high computing power consumption, and cannot model the spatio-temporal interaction between people. Summary of the Invention

[0015] In view of this, an embodiment of the present invention provides a single-stage multi-person mesh model construction method and system based on temporal coordinate information that can reduce computing power consumption.

[0016] One aspect of the embodiment of the present invention provides a single-stage multi-person mesh model construction method based on temporal coordinate information, including:

[0017] Obtain a video of a real scene;

[0018] Extract features from the real scene video through a neural network, and predict a human center map, a camera parameter map, and a human parameter map based on the extracted features;

[0019] Use the human center heat map to focus on the camera parameter map and the human parameter map, and perform spatial and temporal relationship modeling based on the neural network to model the spatio-temporal connection between human models, and predict the final camera parameter map and human parameter map;

[0020] Find the areas determined as humans in the human-centered heatmap, index them to the corresponding camera parameter map and human parameter map, and generate a human model.

[0021] Optionally, the feature extraction of the real-scene video by the neural network and the prediction of the human-centered map, camera parameter map, and human parameter map based on the extracted features include:

[0022] Perform feature extraction on each frame of the real-scene video using a deep learning-based feature extractor to obtain image features;

[0023] Add coordinate features representing the two directions of pixels to the image features to obtain enhanced image features;

[0024] Perform prediction processing on the enhanced image features through a multi-head convolutional layer to obtain a human-centered heatmap, a camera parameter map, and a human parameter map.

[0025] Optionally, in the step of using the human-centered heatmap to perform focusing processing on the camera parameter map and the human parameter map, the expression of the focusing processing is:

[0026] Camera c =Camera c ⊙Heatmap

[0027] SMPL c =SMPL c ⊙Heatmap

[0028] Where Camera c and SMPL c represent the c-dimensional features of the camera parameter map and the human parameter map respectively, ⊙ represents the Hadamard product between matrices; Heatmap represents the human-centered heatmap.

[0029] Optionally, using the human-centered heatmap to perform focusing processing on the camera parameter map and the human parameter map, and performing spatial and temporal relationship modeling based on the neural network to model the spatio-temporal connection between human models and predict the final camera parameter map and human parameter map, including:

[0030] Concatenate the camera parameter map and the human parameter map into a grid feature map;

[0031] Concatenate coordinate information representing time and coordinate height attributes to the grid feature map to obtain an enhanced grid feature map;

[0032] According to the enhanced grid feature map, obtain temporal features through Temporal Transformer modeling processing;

[0033] Based on the enhanced grid feature map, spatial features are obtained through Spatial Transformer modeling processing;

[0034] The time features and the spatial features are concatenated to obtain hybrid features;

[0035] The hybrid features are predicted through a convolutional layer to obtain the final camera parameter map and the person parameter map.

[0036] Another aspect of the embodiments of the present invention further provides a single-stage multi-person grid model construction system based on temporal coordinate information, including:

[0037] The first module is used to obtain a real-scene video;

[0038] The second module is used to extract features from the real-scene video through a neural network, and predict a person center map, a camera parameter map, and a person parameter map based on the extracted features;

[0039] The third module is used to focus on the camera parameter map and the person parameter map by using the person center heat map, and perform spatial and temporal relationship modeling based on the neural network, model the spatio-temporal connection between person models, and predict the final camera parameter map and the person parameter map;

[0040] The fourth module is used to find out the regions determined as people in the person center heat map, and index to the corresponding camera parameter map and person parameter map to generate a person model.

[0041] Optionally, the second module includes:

[0042] The first unit is used to extract features from each frame of the real-scene video through a deep learning-based feature extractor to obtain image features;

[0043] The second unit is used to add coordinate features representing the two directions of pixels to the image features to obtain enhanced image features;

[0044] The third unit is used to perform prediction processing on the enhanced image features through a multi-head convolutional layer to obtain a person center heat map, a camera parameter map, and a person parameter map.

[0045] Optionally, the third module includes:

[0046] The fourth unit is used to concatenate the camera parameter map and the person parameter map into a grid feature map;

[0047] The fifth unit is used to concatenate coordinate information representing time and coordinate height attributes to the grid feature map to obtain an enhanced grid feature map;

[0048] The sixth unit is configured to obtain temporal features through Temporal Transformer modeling processing based on the enhanced grid feature map;

[0049] The seventh unit is configured to obtain spatial features through Spatial Transformer modeling processing based on the enhanced grid feature map;

[0050] The eighth unit is configured to splice the temporal features and the spatial features to obtain hybrid features;

[0051] The ninth unit is configured to perform prediction on the hybrid features through a convolutional layer to obtain the final camera parameter map and the person parameter map.

[0052] Another aspect of the embodiments of the present invention further provides an electronic device, including a processor and a memory;

[0053] The memory is used to store a program;

[0054] The processor executes the program to implement the method as described above.

[0055] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, where the storage medium stores a program,

[0056] The program is executed by a processor to implement the method as described above.

[0057] The embodiments of the present invention also disclose a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method as described above.

[0058] The embodiments of the present invention obtain a real-scene video; perform feature extraction on the real-scene video through a neural network, and predict a person center map, a camera parameter map, and a person parameter map according to the extracted features; use the person center heat map to perform focusing processing on the camera parameter map and the person parameter map, and perform spatial and temporal relationship modeling based on the neural network to model the spatio-temporal connection between person models, and predict the final camera parameter map and the person parameter map; find out the regions determined as people in the person center heat map, and index to the corresponding camera parameter map and person parameter map to generate a person model. The present invention utilizes the inherent coordinate attributes of the detected object to achieve implicit detection in the form of feature interaction, and learns the spatio-temporal connection of different objects across frames based on the neural network, avoiding the computing power consumption brought by explicit detection and explicit tracking, and realizing the simplicity, low computing power consumption, and accuracy of person model regression. Description of the Drawings

[0059] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0060] Figure 1 It is a comparison diagram of the design specifications of a single-stage multi-person mesh model regression system based on temporal coordinate information in an embodiment of the present invention;

[0061] Figure 2 It is a schematic diagram of the architecture of a single-stage multi-person mesh model regression system based on temporal coordinate information in an embodiment of the present invention;

[0062] Figure 3 It is a schematic diagram of the working process of a single-stage multi-person mesh model regression system based on temporal coordinate information in an embodiment of the present invention;

[0063] Figure 4 It is a system framework diagram of a single-stage multi-person mesh model regression system based on temporal coordinate information in an embodiment of the present invention. Detailed implementation manners

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the following further details the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] First, the relevant technical terms that may appear in the embodiments of the present invention are introduced specifically:

[0066] 1) Human body mesh model: The human body mesh model is a model that approximately represents a human body using a series of polygons (usually triangles) with similar sizes and shapes.

[0067] 2) Human body parameter model: The human body parameter model refers to collecting and statistically analyzing a large amount of human body shape data and obtaining a mapping function based on human body characteristics, that is, only by providing partial attribute data, the vertices of the entire human body model can be obtained. Taking SMPL as an example, only by providing 72 pose parameters and 10 shape parameters, a human body model with 6890 vertices can be obtained.

[0068] 3) Camera parameters: Only one corresponding human body parameter can be obtained through the SMPL model, so camera parameters are needed to obtain the distance and rotation angle of the human body, etc.

[0069] 4) Person-centered heatmap: The person-centered heatmap is a heatmap representing whether it is person-centered. That is, each pixel on this map represents the probability of this pixel being a person center, which is equivalent to judging whether there are people in this place, forming a prediction of the existence of people without being able to give a person bounding box.

[0070] 5) Image regression method based on single image and multiple stages: Briefly speaking, they cannot predict the person models of all people at the same time. Therefore, it is necessary to first use a detector to find the bounding box where the person is located, and crop the image of the specified area as the input, and then predict the corresponding person grid model based on the neural network.

[0071] 6) Image regression method based on single image and single stage: This solution uses the person-centered heatmap as a detection method, and simultaneously predicts the corresponding camera parameters and person parameters when it is assumed that there are people on all pixels. Finally, indexing and extraction are performed through the person-centered heatmap.

[0072] 7) Image regression method based on video and multiple stages: In order to build correct temporal connections, this method requires obtaining an image sequence of a certain person. Therefore, it is necessary to first use a detector and a tracker to obtain a series of cropped image frames of a certain person, and use this as the input to build temporal information.

[0073] 6) Differences between single stage and multiple stages: Multiple stages need to rely on independent detectors and trackers, which exist as an independent task in themselves. Therefore, they have high computing power consumption and running time costs. At the same time, the multiple-stage model needs to independently train the detector and tracker in the first stage, and the accuracy being limited by the accuracy of the first stage may also lead to error accumulation. In a multi-person scenario, the multiple-stage model requires 2 to 5 times more running time cost than the single-stage model.

[0074] In view of the problems existing in the prior art, one aspect of the embodiments of the present invention provides a method for constructing a single-stage multi-person grid model based on temporal coordinate information, including:

[0075] Obtain a video of a real scene;

[0076] Extract features from the video of the real scene through a neural network, and predict a person center map, a camera parameter map, and a person parameter map according to the extracted features;

[0077] Use the person-centered heatmap to perform focusing processing on the camera parameter map and the person parameter map, and perform spatial and temporal relationship modeling based on the neural network, model the spatio-temporal connection between person models, and predict the final camera parameter map and person parameter map;

[0078] Find the area judged to be a person in the person-centered heatmap, and index to the corresponding camera parameter map and person parameter map to generate a person model.

[0079] Optionally, the feature extraction of the real scene video by the neural network and the prediction of the person center map, camera parameter map, and person parameter map according to the extracted features include:

[0080] Performing feature extraction on each frame of the real scene video by a deep learning-based feature extractor to obtain image features;

[0081] Adding coordinate features representing the coordinates of pixels in two directions to the image features to obtain enhanced image features;

[0082] Performing prediction processing on the enhanced image features through a multi-head convolutional layer to obtain a person center heatmap, a camera parameter map, and a person parameter map.

[0083] Optionally, in the step of using the person center heatmap to perform focusing processing on the camera parameter map and the person parameter map, the expression of the focusing processing is:

[0084] Camera c = Camera c ⊙ Heatmap

[0085] SMPL c = SMPL c ⊙ Heatmap

[0086] where Camera c and SMPL c represent the c-dimensional features of the camera parameter map and the person parameter map respectively, ⊙ represents the Hadamard product between matrices; Heatmap represents the person center heatmap.

[0087] Optionally, using the person center heatmap to perform focusing processing on the camera parameter map and the person parameter map, and performing spatial and temporal relationship modeling based on the neural network, modeling the spatio-temporal connection between person models, and predicting the final camera parameter map and person parameter map, including:

[0088] Concatenating the camera parameter map and the person parameter map into a grid feature map;

[0089] Concatenating coordinate information representing time and coordinate height attributes to the grid feature map to obtain an enhanced grid feature map;

[0090] According to the enhanced grid feature map, obtaining temporal features through Temporal Transformer modeling processing;

[0091] According to the enhanced grid feature map, obtaining spatial features through Spatial Transformer modeling processing;

[0092] Concatenate the time feature and the space feature to obtain a hybrid feature;

[0093] Perform prediction on the hybrid feature through a convolutional layer to obtain the final camera parameter map and the person parameter map.

[0094] Another aspect of the embodiments of the present invention further provides a single-stage multi-person mesh model construction system based on temporal coordinate information, including:

[0095] The first module is used to obtain a real-scene video;

[0096] The second module is used to extract features from the real-scene video through a neural network, and predict a person center map, a camera parameter map, and a person parameter map according to the extracted features;

[0097] The third module is used to focus on the camera parameter map and the person parameter map by using the person center heat map, and perform spatial and temporal relationship modeling based on the neural network, model the spatio-temporal connection between person models, and predict the final camera parameter map and the person parameter map;

[0098] The fourth module is used to find the regions determined as people in the person center heat map, and index to the corresponding camera parameter map and person parameter map to generate a person model.

[0099] Optionally, the second module includes:

[0100] The first unit is used to extract features from each frame of the real-scene video through a deep learning-based feature extractor to obtain image features;

[0101] The second unit is used to add coordinate features representing the two directions of pixels to the image features to obtain enhanced image features;

[0102] The third unit is used to perform prediction processing on the enhanced image features through a multi-head convolutional layer to obtain a person center heat map, a camera parameter map, and a person parameter map.

[0103] Optionally, the third module includes:

[0104] The fourth unit is used to concatenate the camera parameter map and the person parameter map into a grid feature map;

[0105] The fifth unit is used to concatenate coordinate information representing time and coordinate height attributes to the grid feature map to obtain an enhanced grid feature map;

[0106] The sixth unit is used to obtain a time feature through Temporal Transformer modeling processing according to the enhanced grid feature map;

[0107] A seventh unit, configured to obtain spatial features through Spatial Transformer modeling processing according to the enhanced grid feature map;

[0108] An eighth unit, configured to splice the temporal features and the spatial features to obtain hybrid features;

[0109] A ninth unit, configured to perform prediction on the hybrid features through a convolutional layer to obtain a final camera parameter map and a person parameter map.

[0110] Another aspect of the embodiments of the present invention further provides an electronic device, including a processor and a memory;

[0111] The memory is used to store a program;

[0112] The processor executes the program to implement the method as described above.

[0113] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, where the storage medium stores a program,

[0114] The program is executed by a processor to implement the method as described above.

[0115] The embodiments of the present invention also disclose a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method as described above.

[0116] The following combines the accompanying drawings of the specification to describe in detail the specific implementation process and implementation principle of the present invention:

[0117] The present invention proposes a single-stage multi-person grid model regression system based on temporal coordinate information, which uses the inherent coordinate attributes of the detected object to achieve implicit detection in the way of feature interaction, and based on neural network learning of the spatio-temporal relationship between different objects across frames, avoiding the computing power consumption brought by explicit detection and explicit tracking, and realizing the simplicity, low computing power consumption and accuracy of the person model regression.

[0118] Specifically, the present invention provides a single-stage multi-person grid model regression system based on temporal coordinate information, which uses the inherent coordinate attributes of the detected object as an implicit identity, and omits explicit detection and explicit tracking based on the coordinate information to implement a single-stage video-based image regression method. The single-stage multi-person grid model regression system includes but is not limited to:

[0119] The interactive terminal layer is used to complete the interaction operations between the user and the multi-person grid model regression system. It realizes the front-end device requests and back-end server response services through interface calls and data communication. The services include, but are not limited to, selecting and uploading user videos, adjusting the system time configuration, setting the system output data format, and pulling the system output results.

[0120] The feature extraction layer is used to initially extract the features of each frame of the video, and directly predict the person center heatmap, camera parameter map, and person parameter map for each frame, representing whether there may be a person on each pixel of the image, the corresponding camera parameters on that pixel, and the person parameters.

[0121] The feature extraction layer receives video data from the user, uses a deep learning-based feature extractor such as ResNet-50 or HRNet-32 to extract image features for each frame, performs coordinate enhancement on the features, and then uses a convolutional layer to predict the person center heatmap and encode the camera parameter map and person parameter map. Specifically,

[0122] The feature extraction layer first receives video data Video ∈ R H×W×T , where H, W, and T represent the length, width, and number of frames of the video frame respectively. After extracting image features Image Feature ∈ R T×H×W×C using a feature extractor Extractor such as ResNet-50 or HRNet-32, where C represents the dimension of the extracted features. To enhance the model's perception of spatial position and distance, the embodiment of the present invention adds coordinate features Coordinate Feature ∈ R T×H×W×2 representing the two directions of the pixel to the image features, that is, the two-layer feature values corresponding to the pixel position (x, y) are x and y respectively, to obtain the enhanced feature Ehanced Feature ∈ R T×H×W× ( C+2 ). On this basis, a multi-head convolutional layer is used to predict the person center heatmap Heatmap ∈ R T×H×W×1 , the camera parameter map Camera ∈ R T×H×W×3 , and the person parameter map SMPL ∈ R T×H×W×142 . The formula is:

[0123] Image Feature = Extractor(Video)

[0124] Enhanced Feature = Cat(Image Feature, Coordinate Feature)

[0125] Heatmap = ConvHead1(Enhanced Feature)

[0126] Camera = ConvHead2(Enhanced Feature)

[0127] SMPL = ConvHead3(Enhanced Feature)

[0128] Among them, Cat represents the concatenation of features, and ConvHead represents the output head of the convolutional layer.

[0129] The center guidance layer focuses on the camera parameter map and the human parameter map by using the human center heatmap, forcing subsequent modeling of the focused features, and achieving an implicit detection effect.

[0130] The center guidance layer uses the human center heatmap Heatmap ∈ R T×H×W×1 to focus on the camera parameter map Camera ∈ R T ×H×W×3 and the human parameter map SMPL ∈ R T×H×W×142 The formula for focusing is

[0131] Camera c = Camera c ⊙ Heatmap

[0132] SMPL c = SMPL c ⊙ Heatmap

[0133] Among them, Camera c and SMPL c respectively represent the c - dimensional features of the camera parameter map and the human parameter map, and ⊙ represents the Hadamard product between matrices, that is, the element - by - element dot - product operation.

[0134] The spatio - temporal modeling layer models the spatial and temporal relationships based on neural networks according to single - frame and multi - frame respectively, in order to simultaneously model the spatio - temporal connections between human models and predict the final camera parameter map and human parameter map.

[0135] The spatio - temporal modeling layer learns the consistency between features based on the Transformer temporal deep - learning model to model the spatio - temporal interaction relationship. The Temporal Transformer based on temporal information receives the features of all frames and learns the cross - frame constraints, and the Spatial Transformer based on spatial information receives the features of each frame and learns the spatial information within a single frame. For the model to be easy to learn, in the embodiments of the present invention, the camera parameter map and the human parameter map are concatenated into a grid feature map Mesh ∈ R T×H×W×145To enhance spatial information, embodiments of the present invention similarly splice coordinate information Temporal Coordinate ∈ R representing time and coordinate height attributes for the grid feature map T×H×2×145 , that is, the feature corresponding to the pixel coordinate (t, x, ~) is (t, x), where ~ represents any value, obtaining Enhanced Mesh ∈ R T×H×(W+2)×145 . After being modeled by Temporal Transformer, temporal feature Temporal Feature ∈ R is obtained T×H×W . After being modeled by Spatial Transformer, spatial feature Spatial Feature ∈ R is obtained T×H×W . They are spliced to obtain the fused feature Fuse Feature ∈ R T ×H×W×16 . Then, through the convolutional layer, the final camera parameter map and person parameter map are predicted. The formula is

[0136] Mesh = Cat(Camera, SMPL)

[0137] Enhanced Mesh = Cat(Mesh, Temporal Coordinate)

[0138] Temporal Feature = Temporal Transformer(Enhanced Mesh)

[0139] Spatial Feature t = Spatial Transformer(Enhanced Mesh t )

[0140] Spatial Feature = Cat(Spatial Feature 1～T )

[0141] Fuse Feature = Cat(Temporal Feature, Spatial Feature)

[0142] Mesh Result = [Camera, SMPL] = ConvHead(Fuse Feature)

[0143] The result sampling layer uses a threshold to find the area determined to be a person in the person center heat map, indexes the camera parameters and person parameters, generates a person model, and finally returns the result to the user.

[0144] The result sampling layer uses a threshold Thresh to find the regions Point∈R in each frame where the model determines a person exists K×2 , where K represents the number of regions in the person center heatmap Heatmap that exceed Thresh, which is also the number of people the model determines in a certain frame of the image. The formula is

[0145] Point = Where((Heatmap > Thresh) ∩ MaxPool(Heatmap))

[0146] Target Mesh = Sample(Mesh Result, Point)

[0147] Where MaxPool represents the max pooling operation and ∩ represents the union operation. So the first formula is to find the points in the Heatmap that are both greater than the threshold Thresh and the maximum value near the center point. Then, the Mesh features of the target are indexed through the pixel coordinates of the target points, and 3D human mesh results are obtained based on models such as SMPL

[0148] As Figure 1 shown is a comparison diagram of the design specifications of a single-stage multi-person mesh model regression system based on temporal coordinate information according to an embodiment of the present invention. This system does not require explicit detection and target tracking, and can simultaneously model the spatio-temporal connections between people

[0149] As Figure 2 shown is a schematic diagram of the architecture of a single-stage multi-person mesh model regression system based on temporal coordinate information provided by an embodiment of the present invention. A single-stage multi-person mesh model regression system based on temporal coordinate information includes but is not limited to

[0150] An interaction terminal layer for completing the interaction operation between the user and the multi-person mesh model regression system, and realizing the front-end device request and back-end server response services through interface calls and data communication. The services include but are not limited to selecting and uploading user videos, adjusting system time configurations, setting system output data formats, and pulling system output results

[0151] A feature extraction layer for initially extracting the features of each frame of the video in the video, and directly predicting the person center heatmap, camera parameter map, and person parameter map of each frame of the image, representing whether a person may exist at each pixel of the image, the corresponding camera parameters at that pixel, and the person parameters

[0152] A center guidance layer that focuses on the camera parameter map and person parameter map by using the person center heatmap, forcing subsequent modeling of the focused features, and realizing an implicit detection effect

[0153] The spatio-temporal modeling layer models spatial and temporal relationships based on neural networks according to single-frame and multi-frame respectively, so as to model the spatio-temporal connections between human models simultaneously and predict the final camera parameter map and human parameter map.

[0154] The result sampling layer uses a threshold to find the regions judged as human in the human center heat map, indexes the camera parameters and human parameters, generates a human model, and finally returns the result to the user.

[0155] Figure 3 It is a schematic diagram of the working process of a secure personalized travel recommendation system based on federated learning and regional avoidance according to an embodiment of the present invention, including the following steps:

[0156] S1: The service user provides the server with model configurations (such as model sampling time window, input image resolution, feature map size, etc.), data formats (visualization result angle, output file naming format, output human model format, etc.) and the video to be predicted.

[0157] S2: The system receives the video and relevant configuration information from the user, and extracts image features, feature focusing and spatio-temporal modeling through a single-stage neural network model.

[0158] S3: The system borrows the predicted results for analysis and sampling, and obtains the corresponding human mesh model by using the human parameter model specified by the user (such as implementation methods like SMPL, STAR, etc.)

[0159] S4: The system returns the specified visualization results and human model results to the user.

[0160] Figure 4 It is a system framework diagram of a single-stage multi-person mesh model regression system based on time-series coordinate information according to an embodiment of the present invention, including the following steps:

[0161] S100: The user provides the server with the specified configuration and video.

[0162] S110: The server predicts the specified result according to the neural network.

[0163] S120: The user obtains the corresponding output result.

[0164] To sum up, the present invention first obtains an effective and feasible deep neural network based on the publicly available dataset. After receiving the video and prediction configuration from the user, it extracts image features and preliminarily predicts the human center map, camera parameter map and human parameter map, saves the computing power consumption and operation time brought by explicit detection through the interaction of the human center map, and finally considers the spatio-temporal interaction between people by using spatio-temporal modeling. Finally, the time-series information is used in a single-stage manner to maintain the consistency of the human model, achieving the rapidity of the prediction process and the simplicity of obtaining a natural and accurate model.

[0165] Compared with the prior art, the present invention has the following advantages:

[0166] 1), The difficulty of obtaining the character model is small and the threshold is low; the present invention does not require heavy independent design or expensive scanning equipment. The user only needs to provide the server with the video sequence to be predicted to obtain the corresponding character mesh model result.

[0167] 2), The character model prediction is accurate and the action is natural; the present invention uses the temporal information in the video to refine the character model action, maintain the continuity and consistency of the action, and reduce jitter. Compared with the current video-based image regression method, the system utilizes the spatio-temporal interaction between character mesh models to obtain more accurate character model results.

[0168] 3), The system has low computing power consumption and short running time; the present invention is based on a single-stage design, avoiding the computing power consumption and waiting time caused by explicit detection and explicit.

[0169] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated, where the order of various operations is changed and where sub-operations described as part of a larger operation are executed independently.

[0170] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features described may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More precisely, considering the attributes, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0171] If the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs, Read-Only Memories), random access memories (RAMs, Random Access Memories), magnetic disks, or optical discs.

[0172] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a predefined sequence of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0173] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part with one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or otherwise processing it as necessary, and then storing it in a computer memory.

[0174] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0175] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0176] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0177] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for constructing a single-stage multi-person grid model based on time-series coordinate information, characterized in that Including: Obtain a video of a real - world scene; Extract features from the video of the real - world scene through a neural network, and predict a person center map, a camera parameter map, and a person parameter map based on the extracted features; Use the person center heat map to perform focusing processing on the camera parameter map and the person parameter map, and perform spatial and temporal relationship modeling based on the neural network, model the spatio - temporal connections between person models, and predict the final camera parameter map and person parameter map; Find the regions determined to be people in the person center heat map, and index to the corresponding camera parameter map and person parameter map to generate a person model; In the step of using the person center heat map to perform focusing processing on the camera parameter map and the person parameter map, the expression of the focusing processing is: Camera c = Camera c ⊙Heatmap SMPL c = SMPL c ⊙Heatmap Among them, Camera c and SMPL c respectively represent the c - dimensional features of the camera parameter map and the human parameter map, ⊙ represents the Hadamard product between matrices; Heatmap represents the human center heatmap; The step of using the person center heat map to perform focusing processing on the camera parameter map and the person parameter map, and performing spatial and temporal relationship modeling based on the neural network, model the spatio - temporal connections between person models, and predict the final camera parameter map and person parameter map, includes: Concatenate the camera parameter map and the person parameter map into a grid feature map; Attach coordinate information representing time and coordinate height attributes to the grid feature map to obtain an enhanced grid feature map; According to the enhanced grid feature map, obtain temporal features through Temporal Transformer modeling processing; According to the enhanced grid feature map, obtain spatial features through Spatial Transformer modeling processing; Concatenate the temporal features and the spatial features to obtain a mixed feature; Predict the final camera parameter map and person parameter map through a convolutional layer for the mixed feature.

2. The method for constructing a single-stage multi-person grid model based on temporal coordinate information according to claim 1, wherein The step of extracting features from the video of the real - world scene through a neural network, and predicting a person center map, a camera parameter map, and a person parameter map based on the extracted features, includes: Extract features from each frame of the video of the real - world scene based on a deep - learning feature extractor to obtain image features; Attach coordinate features representing the coordinates in two directions of the pixels to the image features to obtain enhanced image features; Perform prediction processing on the enhanced image features through a multi - head convolutional layer to obtain a person center heat map, a camera parameter map, and a person parameter map.

3. A single-stage multi-person grid model construction system based on temporal coordinate information, characterized in that, Including: The first module is used to obtain a video of a real - world scene; The second module is used to extract features from the video of the real - world scene through a neural network, and predict a person center map, a camera parameter map, and a person parameter map based on the extracted features; The third module is used to use the person center heat map to perform focusing processing on the camera parameter map and the person parameter map, and perform spatial and temporal relationship modeling based on the neural network, model the spatio - temporal connections between person models, and predict the final camera parameter map and person parameter map; The fourth module is used to find the regions determined to be people in the person center heat map, and index to the corresponding camera parameter map and person parameter map to generate a person model; The expression of the focusing processing is: Camera c = Camera c ⊙Heatmap SMPL c = SMPL c ⊙Heatmap Among them, Camera c and SMPL c respectively represent the c - dimensional features of the camera parameter map and the human parameter map, ⊙ represents the Hadamard product between matrices; Heatmap represents the human center heatmap; The third module includes: The fourth unit is used to concatenate the camera parameter map and the person parameter map into a grid feature map; The fifth unit is configured to splice coordinate information representing time and coordinate height attributes to the grid feature map to obtain an enhanced grid feature map; The sixth unit is configured to obtain temporal features through TemporalTransformer modeling processing according to the enhanced grid feature map; The seventh unit is configured to obtain spatial features through SpatialTransformer modeling processing according to the enhanced grid feature map; The eighth unit is configured to splice the temporal features and the spatial features to obtain hybrid features; The ninth unit is configured to perform prediction on the hybrid features through a convolutional layer to obtain a final camera parameter map and a person parameter map.

4. The single-stage multi-person grid model construction system based on timing coordinate information according to claim 3, wherein The second module includes: The first unit is configured to perform feature extraction on each frame of the real-scenario video through a deep learning-based feature extractor to obtain image features; The second unit is configured to add coordinate features representing pixels in two directions to the image features to obtain enhanced image features; The third unit is configured to perform prediction processing on the enhanced image features through a multi-head convolutional layer to obtain a person center heat map, a camera parameter map, and a person parameter map.

5. An electronic device, characterized in that, It includes a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method according to any one of claims 1 to 2.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 2.

Images