Multi-person 3D human pose estimation method and device based on multi-view

Abstract

The present application discloses a method and device for multi-person 3D human pose estimation based on multiple perspectives, wherein the method includes: estimating the 2D human pose of each person in each human body picture based on the transformer method; using the 2D human pose of each person, using a pre-built pedestrian re-identification model to perform personnel registration and recognition on each person in each human body picture, and determining the 2D joint points of each person; and using a multi-viewpoint triangulation method to triangulate the 2D joint points of each person in each picture, fitting them to the 3D key points in the 3D space, and generating the 3D human pose. Thus, the technical problem of low applicability in the related technology due to the large amount of computing resources occupied and the slow running time that makes it difficult to be widely used in scenes with poor computing hardware conditions is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for estimating multi-person three-dimensional human body posture based on multiple perspectives. Background Art

[0002] The task of human posture estimation is an important research branch in the current field of computer vision, and it is also a research hotspot based on application and industry needs. There are three common ways to divide the task of human posture estimation: according to the number of input perspectives provided, it can be divided into single-perspective estimation tasks and multi-perspective estimation tasks; according to the number of people detected, it can be divided into single-person scene tasks and multi-person scene tasks; according to the purpose information, it can be divided into two-dimensional estimation tasks and three-dimensional estimation tasks. According to its basic classification and application scenarios, the task of human posture estimation has a wide range of applications in the fields of holographic reality, human simulation, video surveillance, drone swarms, etc., and it also has huge development potential. Human posture estimation is also the research basis of many computer vision tasks, and its estimation accuracy has an important impact on the results of downstream tasks. Therefore, studying the problem of human posture estimation is of increasing significance.

[0003] The related technology follows the three steps of 2D human pose estimation in a single image, personnel registration and recognition in multiple images, and 3D human pose fitting. It adopts a multi-way matching algorithm. The matching algorithm first establishes a cross-matching matrix by analyzing the appearance information itself, and finds the periodic consistency correspondence of the 2D poses detected in multiple views, so as to match different people in multi-view images. The matching algorithm can trim false detections and process partial overlaps between views without knowing the actual number of people in the scene, and achieves good results in the problem of cross-linked 2D view personnel matching. At the same time, the related technology improves the conventional 3DPS (3D pictorial structure, immersive projection display system) method, and generates 2D pose clusters by matching the detected 2D poses between multiple views. Each cluster contains the 2D poses of the same person in different views, solving the correspondence problem at the body level.

[0004] However, there are still many defects in the related technology:

[0005] 1. The 2D human pose estimation method is poor. The 2D human pose designed by the related technology is a CPN (Cascaded Pyramid Network) trained on the MSCOCO 2D human pose dataset. The CPN system has two sub-networks, a global network for global human positioning and rough estimation of key points, and a local fine-tuning network for estimating the fine position of joints. This method has poor accuracy, slow running time, and occupies more computing resources.

[0006] 2. The overall system runs slowly. The processing steps of this system are basically the same as those of other similar systems, and they all follow the pipeline process of 2D posture estimation, personnel matching, and 3D fitting between single frames. Related technologies have designed too many links in each system to ensure accuracy. For example, in the personnel matching link, similarity estimation, pedestrian re-identification, cycle consistency and other technologies are designed to ensure fitting accuracy. The 3D fitting link uses a computationally intensive 3DPS method to ensure fitting accuracy, which occupies a large amount of computing resources and prolongs a lot of computing time.

[0007] In summary, the relevant technologies are difficult to be widely used in scenarios with poor computing hardware conditions due to their large occupation of computing resources and slow running time. They have low applicability and are in urgent need of improvement. Summary of the invention

[0008] The present application provides a method and device for multi-person three-dimensional human body posture estimation based on multiple perspectives to solve technical problems in related technologies such as occupying more computing resources, having a slow running time, being difficult to be widely used in scenarios with poor computing hardware conditions, having low applicability, and having poor accuracy in human body posture estimation.

[0009] The first aspect of the present application provides a method for estimating three-dimensional human body posture of multiple people based on multiple perspectives, comprising the following steps: estimating the two-dimensional human body posture of each person in each human body picture based on a transformer method; using the two-dimensional human body posture of each person, using a pre-built pedestrian re-identification model to perform person registration and identification on each person in each human body picture, and determining the two-dimensional joint points of each person; and using a multi-viewpoint triangulation method to triangulate the two-dimensional joint points of each person in each picture, fitting them to three-dimensional key points in three-dimensional space, and generating a three-dimensional human posture.

[0010] Optionally, in one embodiment of the present application, the transformer-based method for estimating the two-dimensional body pose of each person in each body image includes: using a swin-transformer skeleton to detect the two-dimensional body pose of each person in each body image; or, using the swin-transformer skeleton to estimate the joint point position of each person to determine the two-dimensional body pose.

[0011] Optionally, in one embodiment of the present application, before using the pre-built person re-identification model to perform person registration and identification on each person in each human image, it also includes: obtaining a public data set for training the model; using the public data set to train the pedestrian re-identification model built based on deep learning to generate the pre-built pedestrian re-identification model.

[0012] Optionally, in one embodiment of the present application, estimating the two-dimensional human posture of each person in each human body picture based on the transformer method includes: obtaining the two-dimensional human posture estimation of each picture based on the ViT deformation framework.

[0013] The second aspect of the present application provides a multi-perspective three-dimensional human posture estimation device for multiple people, including: a posture estimation module, which is used to estimate the two-dimensional human posture of each person in each human body picture based on a transformer method; an identification module, which is used to use the two-dimensional human posture of each person, and use a pre-built pedestrian re-identification model to perform person registration and identification on each person in each human body picture, and determine the two-dimensional joint points of each person; and a generation module, which is used to triangulate the two-dimensional joint points of each person in each picture using a multi-viewpoint triangulation method, fit them to three-dimensional key points in three-dimensional space, and generate three-dimensional human posture.

[0014] Optionally, in one embodiment of the present application, the posture estimation module is further used to use the swin-transformer skeleton to detect the two-dimensional body posture of each person in each body image; or, use the swin-transformer skeleton to estimate the joint point position of each person to determine the two-dimensional body posture.

[0015] Optionally, in one embodiment of the present application, the recognition module includes: an acquisition unit, used to acquire a public data set for training a model; and a generation unit, used to use the public data set to train a pedestrian re-identification model constructed based on deep learning to generate the pre-built pedestrian re-identification model.

[0016] Optionally, in one embodiment of the present application, the posture estimation module includes: a posture estimation unit, configured to obtain a two-dimensional human body posture estimation for each image based on a deformation framework of ViT.

[0017] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-perspective-based multi-person three-dimensional human body posture estimation method as described in the above embodiment.

[0018] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the multi-perspective based multi-person three-dimensional human body pose estimation method as described in any one of claims 1 to 4.

[0019] The embodiment of the present application can estimate the 2D human posture of each person in each human body picture based on the transformer method, and use the pedestrian re-identification model to perform personnel registration and recognition, extract the 2D joint points of each person, and triangulate the 2D key points through multi-view triangulation, and then generate 3D human posture. Without harsh hardware equipment operating conditions, it can complete the 3D human posture estimation of multiple people under the premise of ensuring a certain accuracy, and has stronger applicability. Therefore, it solves the technical problem that the related technology of realizing 3D human posture estimation of multiple people leads to more computing resources being occupied during operation, slower running time, and difficulty in being widely used in scenarios with poor computing hardware conditions, and low applicability.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a method for estimating multi-person 3D human body posture based on multiple perspectives provided according to an embodiment of the present application;

[0023] Figure 2 A flowchart of a method for estimating multi-person 3D human body posture based on multiple perspectives according to an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of a multi-view 3D human body posture estimation device provided according to an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] The following describes the method and device for multi-person 3D human pose estimation based on multiple perspectives of the embodiment of the present application with reference to the accompanying drawings. Aiming at the technical problem that the related technology mentioned in the background technology center mentioned above, in order to realize the multi-person 3D human pose estimation, it occupies more computing resources and runs slowly during operation, and it is difficult to be widely used in scenes with poor computing hardware conditions, and the applicability is low, the present application provides a multi-person 3D human pose estimation method based on multiple perspectives, in which the 2D human pose of each person in each human body picture can be estimated based on the transformer method, and the pedestrian re-identification model is used to perform personnel registration and recognition, extract the 2D joint points of each person, and triangulate the 2D key points through the multi-viewpoint triangulation method, and then generate the 3D human pose, without the need for harsh hardware equipment operating conditions, it can complete the multi-person 3D human pose estimation under the premise of ensuring a certain accuracy, and the applicability is stronger. Thus, the technical problem that the related technology, in order to realize the multi-person 3D human pose estimation, occupies more computing resources and runs slowly during operation, and it is difficult to be widely used in scenes with poor computing hardware conditions, and the applicability is low is solved.

[0028] Specifically, Figure 1 A flowchart of a multi-view based three-dimensional human body pose estimation method for multiple people provided in an embodiment of the present application.

[0029] like Figure 1 As shown, the multi-view based three-dimensional human body pose estimation method includes the following steps:

[0030] In step S101, the two-dimensional body posture of each person in each body picture is estimated based on the transformer method.

[0031] It is understandable that two-dimensional human pose estimation can be divided into single-person human pose estimation and multi-person human pose estimation according to the number of people in the image. Single-person human pose estimation can be regarded as locating the given joint points of a single person in the image under the conditions of a given number of people and determined joint points; while multi-person pose estimation usually cannot determine the number of people and the position of each person in the input (picture or video) in advance, and the probability of human occlusion and self-occlusion is relatively high. Compared with the single-person human pose estimation task, the multi-person human pose estimation task is more difficult.

[0032] During the actual implementation process, the embodiment of the present application can use the transformer framework to estimate the two-dimensional human posture of each person in each human body image based on data flow and transformation, thereby realizing the estimation of single-person human posture or multi-person human posture, and laying the foundation for subsequent three-dimensional human posture conversion.

[0033] Optionally, in one embodiment of the present application, the two-dimensional human body posture of each person in each human body picture is estimated based on the transformer method, including: obtaining the two-dimensional human body posture estimation for each picture based on the ViT deformation framework.

[0034] In some cases, the embodiments of the present application can obtain a two-dimensional human body posture estimate for each image based on the deformation framework of ViT, laying the foundation for the subsequent determination of the two-dimensional joint points of each person. It can reduce computing consumption and time while ensuring high accuracy, which is conducive to the embodiments of the present application being widely used in scenarios with low hardware equipment.

[0035] Optionally, in one embodiment of the present application, the two-dimensional body posture of each person in each body image is estimated based on the transformer method, including: using the swin-transformer skeleton to detect the two-dimensional body posture of each person in each body image; or, using the swin-transformer skeleton to estimate the joint point position of each person to determine the two-dimensional body posture.

[0036] In other cases, the embodiments of the present application may utilize a variant of the ViT model, that is, an improved swin-transformer skeleton to replace the human body detection link in the top-down CPN, or use the improved swin-transformer skeleton to completely replace the entire CPN link (human body detection + key point estimation link).

[0037] In step S102, the two-dimensional human posture of each person is used to perform person registration and recognition on each person in each human body image using a pre-built pedestrian re-identification model to determine the two-dimensional joint points of each person.

[0038] Those skilled in the art can understand that pedestrian re-identification technology is an artificial intelligence technology that uses computer vision technology to retrieve whether a specific pedestrian exists in an image or video sequence, and can be applied to scenes such as urban monitoring. The process of pre-building a pedestrian re-identification model in the embodiment of the present application can be divided into three parts: feature learning, metric learning, and ranking optimization.

[0039] Furthermore, the embodiments of the present application can perform person registration and identification on each person in each human body image based on a pre-built pedestrian re-identification model, extract the two-dimensional joint points of each person, and provide a data basis for subsequent triangulation technology, which helps to realize the pose estimation of multiple three-dimensional human bodies based on multiple perspectives.

[0040] Optionally, in one embodiment of the present application, before using a pre-built person re-identification model to perform person registration and identification on each person in each human body image, it also includes: obtaining a public data set for training the model; using the public data set to train the pedestrian re-identification model built based on deep learning to generate a pre-built pedestrian re-identification model.

[0041] Specifically, the steps for building a person re-identification model include:

[0042] 1. Data collection. The data sources usually include: raw data captured from the video, or data from surveillance cameras in the same scene, and then obtain the data set for building the pedestrian re-identification model.

[0043] 2. Pedestrian frame generation: The construction of the pedestrian re-identification model can generate pedestrian frames by cutting out pedestrians from the video data manually, through pedestrian detection, tracking, etc.

[0044] 3. Label the training data. The labeled data includes information such as camera labels and pedestrian labels.

[0045] 4. Training module.

[0046] 5. Perform model testing in a test environment.

[0047] In an embodiment of the present application, a public data set can be used to train a pedestrian re-identification model constructed by deep learning, thereby omitting the three steps of data collection, pedestrian box generation, and labeling training data, thereby simplifying the process of building a pedestrian re-identification model, thereby relaxing the requirements for hardware equipment, so that the embodiment of the present application can be widely applicable to scenarios with low hardware equipment conditions while ensuring high accuracy.

[0048] In step S103, a multi-viewpoint triangulation method is used to perform a triangulation operation on the two-dimensional joint points of each person in each picture, and the triangulation operation is performed to fit the three-dimensional key points in the three-dimensional space to generate a three-dimensional human body posture.

[0049] As a possible implementation method, the embodiment of the present application can use multi-viewpoint triangulation technology to triangulate the two-dimensional joint points determined by each person in each picture, and directly fit the two-dimensional joint points determined by each person in each picture to the three-dimensional key points in the three-dimensional space, thereby generating a three-dimensional human posture. The embodiment of the present application uses triangulation technology, which has a small calculation scale and can effectively improve the calculation rate. It can be widely used in scenarios where the actual accuracy requirements are not high and the hardware equipment conditions are poor.

[0050] Combine the following Figure 2 As shown, a multi-view three-dimensional human body posture estimation method for an embodiment of the present application is described in detail by taking an example.

[0051] like Figure 2 As shown, the embodiment of the present application includes the following steps:

[0052] Step S201: Use the transformer framework to perform 2D human posture estimation. In the actual implementation process, the embodiment of the present application can use the transformer framework to estimate the 2D human posture of each person in each human body picture based on the data flow and transformation method, thereby realizing the estimation of a single person's human posture or a multi-person human posture, and laying the foundation for the subsequent 3D human posture conversion.

[0053] Specifically, the embodiment of the present application can perform two-dimensional human body posture estimation in the following two ways:

[0054] 1. The embodiment of the present application can use a variant of the ViT model, that is, the improved swin-transformer framework to replace the human body detection link in the top-down CPN;

[0055] 2. Use the improved swin-transformer skeleton to completely replace the entire CPN link (human detection + key point estimation link).

[0056] It is understandable that the ViT model is a major breakthrough in the application of transformer technology in the field of computer vision, marking the emergence of a unified expression model in the two major fields of natural language processing and computer vision. The ViT module takes a fixed-size segmented image block as input and feeds it into the transformer module, ultimately obtaining results such as image segmentation, image detection, and image tasks, and can basically achieve good results.

[0057] However, the ViT model has certain defects. First, due to the great differences in visual entities and the uneven aggregation of effective information in the image, the ViT model will accept excessive input information. Second, the image input causes the computational complexity of the ViT model to be the square of the image scale, resulting in an excessively large amount of calculation. In summary, these two major shortcomings make the ViT technology not well adapted to most practical environments and have high requirements for computing equipment.

[0058] In order to solve the above problems, the swin-transformer model uses a sliding window operation, so that adaptive information can be input into the model. The sliding window operation scheme will obtain non-overlapping local windows through the multi-head self-attention mechanism in the model. At the same time, the module also allows cross-window and cross-scale links of information, which can not only neutralize the effective information aggregation of intermediate processes at different scales, improve the receptive field, but also obtain higher efficiency. The above structure can reduce the computational complexity of the basic image method given to the transformer from the square of the image scale to linearity, thereby expanding its practical application range.

[0059] Step S202: Use person re-identification technology to perform person registration. Furthermore, the embodiment of the present application can perform person registration and recognition on each person in each human body image based on the pre-built person re-identification model, extract the two-dimensional joint points of each person, and provide a data basis for subsequent triangulation technology, which is helpful to realize the position and posture estimation of multiple three-dimensional human bodies based on multiple perspectives.

[0060] Among them, the steps of building the pedestrian re-identification model include:

[0061] 1. Data collection. The data sources usually include: raw data captured from the video, or data from surveillance cameras in the same scene, and then obtain the data set for building the pedestrian re-identification model.

[0062] 2. Pedestrian frame generation: The construction of the pedestrian re-identification model can generate pedestrian frames by cutting out pedestrians from the video data manually, through pedestrian detection, tracking, etc.

[0063] 3. Label the training data. The labeled data includes information such as camera labels and pedestrian labels.

[0064] 4. Training module.

[0065] 5. Perform model testing in a test environment.

[0066] In an embodiment of the present application, a public data set can be used to train a pedestrian re-identification model constructed by deep learning, thereby omitting the three steps of data collection, pedestrian box generation, and labeling training data, thereby simplifying the process of building a pedestrian re-identification model, thereby relaxing the requirements for hardware equipment, so that the embodiment of the present application can be widely applicable to scenarios with low hardware equipment conditions while ensuring high accuracy.

[0067] Step S203: Use triangulation technology to complete the three-dimensional information fitting. As a possible implementation method, the embodiment of the present application can use multi-viewpoint triangulation technology to triangulate the two-dimensional joint points determined by each person in each picture, and directly fit the two-dimensional joint points determined by each person in each picture to the three-dimensional key points in the three-dimensional space, thereby generating a three-dimensional human body posture. The embodiment of the present application uses triangulation technology, which has a small calculation scale and can effectively improve the calculation rate. It can be widely used in scenes with low actual accuracy requirements and poor hardware equipment conditions.

[0068] According to the method for estimating the three-dimensional human pose of multiple people based on multiple perspectives proposed in the embodiment of the present application, the two-dimensional human pose of each person in each human body picture can be estimated based on the transformer method, and the pedestrian re-identification model can be used to perform personnel registration and recognition, extract the two-dimensional joint points of each person, and triangulate the two-dimensional key points through multi-viewpoint triangulation to generate the three-dimensional human pose. Without the need for harsh hardware equipment operating conditions, the three-dimensional human pose estimation of multiple people can be completed under the premise of ensuring a certain accuracy, and the applicability is stronger. Therefore, the technical problem of low applicability caused by the large amount of computing resources occupied and slow running time during operation in the related technology for realizing the three-dimensional human pose estimation of multiple people is solved.

[0069] Next, a multi-perspective three-dimensional human body posture estimation device for multiple persons proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0070] Figure 3 It is a block diagram of a multi-view based three-dimensional human body posture estimation device according to an embodiment of the present application.

[0071] like Figure 3 As shown, the multi-view based multi-person 3D human pose estimation device 10 includes: a pose estimation module 100 , a recognition module 200 and a generation module 300 .

[0072] Specifically, the posture estimation module 100 is used to estimate the two-dimensional human posture of each person in each human body picture based on the transformer method.

[0073] The recognition module 200 is used to use the two-dimensional human posture of each person and a pre-built pedestrian re-identification model to perform person registration and recognition on each person in each human body picture to determine the two-dimensional joint points of each person.

[0074] The generation module 300 is used to triangulate the two-dimensional joint points of each person in each picture using a multi-viewpoint triangulation method, fit them to the three-dimensional key points in the three-dimensional space, and generate a three-dimensional human body posture.

[0075] Optionally, in one embodiment of the present application, the posture estimation module is further used to detect the two-dimensional body posture of each person in each body image using the swin-transformer skeleton; or, to estimate the joint point position of each person using the swin-transformer skeleton to determine the two-dimensional body posture.

[0076] Optionally, in one embodiment of the present application, the identification module 200 includes: an acquisition unit and a generation unit.

[0077] The acquisition unit is used to acquire a public data set for training the model.

[0078] The generation unit is used to train a person re-identification model built based on deep learning using a public dataset to generate a pre-built person re-identification model.

[0079] Optionally, in one embodiment of the present application, the posture estimation module 100 includes: a posture estimation unit.

[0080] Among them, the posture estimation unit is used to obtain the two-dimensional human posture estimation of each image based on the ViT deformation framework.

[0081] It should be noted that the aforementioned explanation of the embodiment of the multi-perspective based three-dimensional human body pose estimation method is also applicable to the multi-perspective based three-dimensional human body pose estimation device of this embodiment, which will not be repeated here.

[0082] According to the device for multi-person 3D human pose estimation based on multiple perspectives proposed in the embodiment of the present application, the 2D human pose of each person in each human body picture can be estimated based on the transformer method, and the pedestrian re-identification model can be used to perform personnel registration and recognition, extract the 2D joint points of each person, and triangulate the 2D key points through multi-viewpoint triangulation to generate the 3D human pose. Without the need for demanding hardware equipment operating conditions, the 3D human pose estimation of multiple people can be completed under the premise of ensuring a certain accuracy, and the applicability is stronger. Therefore, the technical problem of low applicability caused by the large amount of computing resources occupied and slow running time during operation in order to realize the 3D human pose estimation of multiple people in the related technology is solved. It is difficult to be widely used in scenes with poor computing hardware conditions.

[0083] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0084] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0085] When the processor 402 executes the program, the multi-view based multi-person 3D human body posture estimation method provided in the above embodiment is implemented.

[0086] Furthermore, the electronic device further comprises:

[0087] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0088] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0089] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0090] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0092] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0093] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned multi-view-based three-dimensional human body posture estimation method is implemented.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0098] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for multi-person 3D human pose estimation based on multiple views, characterized in that: The following steps are involved: Estimate the 2D human pose of each person in each human image based on the transformer method; Using the two-dimensional human body posture of each person, and using a pre-built pedestrian re-identification model, perform person registration and identification on each person in each human body image to determine the two-dimensional joint points of each person; as well as Using a multi-viewpoint triangulation method to triangulate the two-dimensional joint points of each person in each image, fitting them to three-dimensional key points in a three-dimensional space, and generating a three-dimensional human body posture; The transformer-based method for estimating the two-dimensional human pose of each person in each human image includes: Detecting the two-dimensional human pose of each person in each human body image using a swin-transformer skeleton; Alternatively, the swin-transformer skeleton is used to estimate the joint position of each person to determine the two-dimensional human body posture; The transformer-based method for estimating the two-dimensional human pose of each person in each human image includes: Obtaining a two-dimensional human pose estimate for each image based on a deformation framework of ViT; Among them, the swin-transformer model adopts a sliding window operation, which obtains non-overlapping local windows through the multi-head self-attention mechanism in the swin-transformer model; the swin-transformer model allows cross-window and cross-scale links of information.

2. The method according to claim 1, characterized in that Before using the pre-built person re-identification model to perform person registration and identification on each person in each human body image, the method further includes: Get public datasets for training models; The public data set is used to train a person re-identification model built based on deep learning to generate the pre-built person re-identification model.

3. A multi-view three-dimensional human body posture estimation device, characterized in that: include: The pose estimation module is used to estimate the 2D pose of each person in each human image based on the transformer method; A recognition module, used to perform person registration and recognition on each person in each human body image by using the two-dimensional human body posture of each person and a pre-built pedestrian re-recognition model, and determine the two-dimensional joint points of each person; as well as A generation module, used for triangulating the two-dimensional joint points of each person in each picture by multi-viewpoint triangulation, fitting them to three-dimensional key points in three-dimensional space, and generating three-dimensional human body posture; The posture estimation module is further used to detect the two-dimensional human posture of each person in each human body picture using a swin-transformer skeleton; Alternatively, the swin-transformer skeleton is used to estimate the joint position of each person to determine the two-dimensional human body posture; The pose estimation module comprises: A posture estimation unit, used for obtaining a two-dimensional human posture estimation for each image based on a deformation framework of ViT; Among them, the swin-transformer model adopts a sliding window operation, which obtains non-overlapping local windows through the multi-head self-attention mechanism in the swin-transformer model; the swin-transformer model allows cross-window and cross-scale links of information.

4. The device according to claim 3, characterized in that The identification module comprises: An acquisition unit, used for acquiring a public data set for training a model; A generating unit is used to train a person re-identification model constructed based on deep learning using the public data set to generate the pre-constructed person re-identification model.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-perspective based multi-person three-dimensional human body posture estimation method as described in any one of claims 1-2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the multi-perspective based multi-person three-dimensional human body pose estimation method as described in any one of claims 1-2.

Images