Personnel twinborn body construction method and device and storage medium

Abstract

The invention provides a person twinborn body construction method and device and a storage medium. According to the embodiment of the invention, a person 3D bounding box of the same person corresponding to different cameras is generated based on a panoramic image obtained by a plurality of cameras, and coordinates of person twins are generated based on human body key point coordinates of the same person corresponding to different cameras. According to the method, the personnel positioning error can be reduced, the positioning precision can be improved, the problem of inaccurate personnel information matching result caused by relatively close personnel can be better solved, and better support is provided for subsequent application based on the personnel twinborn body position.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and storage medium for constructing a person twin. Background Art

[0002] Digital twin technology refers to the use of digital models to simulate objects, systems and processes in the real world to achieve all-round monitoring and optimization. The application scope of digital twin technology is very wide, covering many fields such as industrial manufacturing, urban planning and logistics management, and is hailed as the next Internet. For example, using technologies such as artificial intelligence technology and 3D visualization technology, digital twins can generate 3D virtual scenes that are highly similar to factory scenes, monitor the location of factory personnel in real time, warn of dangerous behaviors of factory personnel, and improve the management and control level of the factory.

[0003] At present, the existing digital twin technology mainly relies on image data from a single perspective when constructing personnel twins. However, this construction method is easily affected by factors such as environmental interference, signal interference, object occlusion and shadow, which leads to the accumulation of personnel positioning errors and reduced positioning accuracy, thereby affecting subsequent applications based on the position of personnel twins, for example, affecting the early warning effect of dangerous behaviors of personnel. Summary of the invention

[0004] At least one embodiment of the present application provides a method, device and storage medium for constructing a human twin, which can improve the position accuracy of the constructed human twin.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a person twin, comprising:

[0007] Acquire a first panoramic image in a target space synchronously photographed by each camera, wherein the target space is provided with at least two cameras, and the field of view of any camera overlaps with that of at least one adjacent camera;

[0008] Performing personnel detection on the first panoramic images synchronously taken by each camera to obtain two-dimensional 2D information of the personnel in the field of view of each camera; generating 3D bounding boxes of the personnel corresponding to each camera in the target space coordinate system according to the 2D information of the personnel in the field of view of each camera;

[0009] Generate at least one bounding box set according to the 3D bounding boxes of people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of a person in each bounding box set is the 3D bounding box of a person corresponding to any camera, and the other 3D bounding boxes of a person are the 3D bounding boxes of people corresponding to the adjacent cameras of any camera, and overlap with the first 3D bounding box of a person;

[0010] Traversing each bounding box set, if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, respectively calculating the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retaining the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and deleting the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set;

[0011] Based on the 3D bounding box of the person in each bounding box set, the coordinates of a person twin are generated; based on the coordinates of each person twin, the person twin is constructed in the target space twin corresponding to the target space.

[0012] Optionally, the person 2D information includes a person 2D bounding box and a 2D coordinate sequence of key points of the human body; and generating a person 3D bounding box corresponding to each camera in the target space coordinate system according to the person 2D information in the field of view of each camera includes:

[0013] Projecting the 2D information of the person in the field of view of each camera into the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, wherein the 3D information of the person includes a 3D bounding box of the person and a 3D coordinate sequence of key points of the human body;

[0014] The plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side of a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera.

[0015] Optionally, generating at least one bounding box set according to the 3D bounding boxes of the persons corresponding to each camera in the target space coordinate system includes:

[0016] Traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set;

[0017] When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to the adjacent cameras, and overlap with the traversed person's 3D bounding box;

[0018] When the traversed person 3D bounding box exists in the generated bounding box set, continue to traverse the next person 3D bounding box.

[0019] Optionally, generating coordinates of a person twin based on the person's 3D bounding box in each bounding box set includes:

[0020] For each bounding box set, obtain the 3D coordinate sequence of the human body key points corresponding to the 3D bounding boxes of all people in the bounding box set, and calculate the average value of the 3D coordinates of the same human body key point to obtain the coordinates of the person twin corresponding to the bounding box set.

[0021] Optionally, the method further includes: calculating the posture similarity between the human bodies pointed to by the two 3D bounding boxes of the persons in the following manner:

[0022] Generate a human key point vector corresponding to the 3D bounding box of the person based on the 3D coordinate sequence of the human key points corresponding to the 3D bounding box of the person;

[0023] The cosine similarity of the human body key point vectors corresponding to the two person 3D bounding boxes is calculated as the posture similarity between the human bodies pointed to by the two person 3D bounding boxes.

[0024] Optionally, generating a human body key point vector corresponding to the 3D bounding box of the person includes:

[0025] The coordinate value of each dimension in the 3D coordinate sequence of the key point of the human body corresponding to the 3D bounding box of the person is respectively used as an element of the intermediate vector to obtain the intermediate vector;

[0026] The intermediate vector is normalized and regularized to obtain the human body key point vector corresponding to the 3D bounding box of the person.

[0027] Optionally, the above method further includes:

[0028] Acquire a second panoramic image captured by each camera at the same time, and convert the second panoramic image captured by each camera into a local 3D model in a target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image; a group of identification points corresponding to the camera is set in the field of view of each camera, wherein when the first camera and the second camera are adjacent cameras with overlapping fields of view, a plurality of identification points corresponding to the first camera set in the overlapping field of view are the same as a plurality of identification points corresponding to the second camera set in the overlapping field of view;

[0029] Based on the external parameters of each camera, the local 3D models are spliced ​​to obtain a global 3D model corresponding to the target space in the target space coordinate system, and the 3D coordinates of each corner point of the global 3D model are obtained;

[0030] According to the 3D coordinates of each corner point of the global 3D model, a target space twin corresponding to the target space is constructed.

[0031] Optionally, the method further includes: obtaining external parameters of each camera according to the following steps:

[0032] Taking a corner point of the target space as the origin, establishing a target space coordinate system of the target space, and obtaining the coordinates of the identification point in the target space coordinate system;

[0033] Obtaining coordinates of the identification points in the second panoramic image captured by each camera;

[0034] According to the coordinates of the identification points in the second panoramic image taken by each camera and the coordinates of the identification points in the target space coordinate system, the coordinates of each camera in the target space coordinate system are determined, and according to the identification points and the coordinates of each camera in the target space coordinate system, each camera is jointly calibrated to determine the external parameters of each camera in the target space coordinate system.

[0035] Optionally, the above method further includes:

[0036] According to the coordinates of each camera in the target space coordinate system, the camera twin corresponding to each camera is constructed.

[0037] In a second aspect, an embodiment of the present application provides a device for constructing a twin of a person, comprising:

[0038] A first acquisition module is used to acquire a first panoramic image in a target space photographed synchronously by each camera, wherein at least two cameras are provided in the target space, and the field of view of any camera overlaps with that of at least one adjacent camera;

[0039] The first generation module is used to perform personnel detection on the first panoramic images synchronously taken by each camera to obtain two-dimensional 2D information of the personnel in the field of view of each camera; based on the 2D information of the personnel in the field of view of each camera, a 3D bounding box of the personnel corresponding to each camera in the target space coordinate system is generated;

[0040] The second generating module is used to generate at least one bounding box set according to the 3D bounding boxes of people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of a person in each bounding box set is the 3D bounding box of a person corresponding to any camera, and the other 3D bounding boxes of a person are the 3D bounding boxes of a person corresponding to the adjacent cameras of the any camera, and overlap with the first 3D bounding box of a person;

[0041] A traversal processing module is used to traverse each bounding box set, and if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, then respectively calculate the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retain the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and delete the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set;

[0042] The first construction module is used to generate the coordinates of a person twin based on the person's 3D bounding box in each bounding box set; based on the coordinates of each person twin, the person twin is constructed in the target space twin corresponding to the target space.

[0043] Optionally, the person 2D information includes a person 2D bounding box and a 2D coordinate sequence of human body key points; and the first generating module is further used to:

[0044] Projecting the 2D information of the person in the field of view of each camera into the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, wherein the 3D information of the person includes a 3D bounding box of the person and a 3D coordinate sequence of key points of the human body;

[0045] The plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side of a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera.

[0046] Optionally, the second generating module is further used to:

[0047] Traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set;

[0048] When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to the adjacent cameras, and overlap with the traversed person's 3D bounding box;

[0049] When the traversed person 3D bounding box exists in the generated bounding box set, continue to traverse the next person 3D bounding box.

[0050] Optionally, the first building module is further used to:

[0051] For each bounding box set, obtain the 3D coordinate sequence of the human body key points corresponding to the 3D bounding boxes of all people in the bounding box set, and calculate the average value of the 3D coordinates of the same human body key point to obtain the coordinates of the person twin corresponding to the bounding box set.

[0052] Optionally, the above device further comprises:

[0053] The similarity calculation module is used to calculate the posture similarity between the human bodies pointed to by the 3D bounding boxes of two persons in the following manner:

[0054] Generate a human key point vector corresponding to the 3D bounding box of the person based on the 3D coordinate sequence of the human key points corresponding to the 3D bounding box of the person;

[0055] The cosine similarity of the human body key point vectors corresponding to the two person 3D bounding boxes is calculated as the posture similarity between the human bodies pointed to by the two person 3D bounding boxes.

[0056] Optionally, the similarity calculation module is further used to:

[0057] The coordinate value of each dimension in the 3D coordinate sequence of the key point of the human body corresponding to the 3D bounding box of the person is respectively used as an element of the intermediate vector to obtain the intermediate vector;

[0058] The intermediate vector is normalized and regularized to obtain the human body key point vector corresponding to the 3D bounding box of the person.

[0059] Optionally, the above device further comprises:

[0060] a conversion module, used for acquiring a second panoramic image captured by each camera at the same time, and converting the second panoramic image captured by each camera into a local 3D model in a target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image; a group of identification points corresponding to the camera is set in the field of view of each camera, wherein when the first camera and the second camera are adjacent cameras with overlapping fields of view, a plurality of identification points corresponding to the first camera set in the overlapping field of view are the same as a plurality of identification points corresponding to the second camera set in the overlapping field of view;

[0061] A stitching module, used to stitch the local 3D models together based on the external parameters of each camera, to obtain a global 3D model corresponding to the target space in the target space coordinate system, and to obtain the 3D coordinates of each corner point of the global 3D model;

[0062] The second construction module is used to construct a target space twin corresponding to the target space according to the 3D coordinates of each corner point of the global 3D model.

[0063] Optionally, the above device further comprises:

[0064] External parameter determination module, used for:

[0065] Taking a corner point of the target space as the origin, establishing a target space coordinate system of the target space, and obtaining the coordinates of the identification point in the target space coordinate system;

[0066] Obtaining coordinates of the identification points in the second panoramic image captured by each camera;

[0067] According to the coordinates of the identification points in the second panoramic image taken by each camera and the coordinates of the identification points in the target space coordinate system, the coordinates of each camera in the target space coordinate system are determined, and according to the identification points and the coordinates of each camera in the target space coordinate system, each camera is jointly calibrated to determine the external parameters of each camera in the target space coordinate system.

[0068] Optionally, the above device further comprises:

[0069] The third construction module is used to construct the camera twin corresponding to each camera according to the coordinates of each camera in the target space coordinate system.

[0070] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the steps of the method described above are implemented.

[0071] Compared with the prior art, the personnel twin construction method and device provided in the embodiments of the present application generate personnel 3D bounding boxes corresponding to different cameras for the same person based on panoramic images obtained by multiple cameras, and generate the coordinates of the personnel twin based on the coordinates of the key points of the human body corresponding to different cameras for the same person. This can reduce personnel positioning errors, improve positioning accuracy, and better handle problems such as inaccurate personnel information matching results caused by close proximity of personnel, thereby providing better support for subsequent applications based on the position of personnel twins. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0073] Figure 1 A flow chart of a method for constructing a person twin according to an embodiment of the present application;

[0074] Figure 2 An example diagram for installing a camera in the target space;

[0075] Figure 3 This is an example diagram of a 3D bounding box of a person according to an embodiment of the present application;

[0076] Figure 4 An example diagram of setting the origin of the target space coordinate system in an embodiment of the present application;

[0077] Figure 5 This is an example diagram of stitching a global 3D model based on a local 3D model in an embodiment of the present application;

[0078] Figure 6 An example diagram of obtaining corner points of a global 3D model in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of the structure of a device for constructing a human twin according to an embodiment of the present application;

[0080] Figure 8 Another structural schematic diagram of the human twin construction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the technical problems, technical solutions and advantages to be solved by the application clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the application. In addition, for clarity and brevity, the description of known functions and structures has been omitted.

[0082] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. "And / or" in the specification and claims represents at least one of the connected objects.

[0083] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0084] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the functions and arrangements of the elements discussed without departing from the spirit and scope of the present disclosure. Various examples may appropriately omit, replace, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0085] Twins, also known as digital twins, are digital representations that are highly similar to physical entities in real scenes. They should reflect the important properties of physical entities in the virtual world. For example, for a spatial twin, such as a factory twin, it is given the spatial structure of the real target space, and its spatial structure is represented by a series of corner point coordinates of the target space. The mathematical definition can be:

[0086] DTE S =f1({(x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,z 2 ),…,(x n ,y n ,z n )}).

[0087] Here, DTE S represents the target space twin, f1(·) represents the functional relationship between the target space twin and each corner point, (x i ,y i ,z i ) represents the 3D coordinates of the i-th corner point.

[0088] Similarly, the camera twin will be assigned the position of the camera, which is represented by the 3D coordinates of the camera in the target space. The mathematical definition of can be: Here, x i ,y i ,z i represents the 3D coordinates of the camera. f2(·) represents the functional relationship between the camera twin and the camera coordinates.

[0089] The person twin will be assigned the person’s position, which is also represented by the person’s 3D coordinates in the target space. The mathematical definition of can be: f3(·) represents the functional relationship between the person twin and the person coordinates.

[0090] Here, the target space model DTE F The mathematical definition of can be: Here, f(·) represents the target space model DTE F The functional relationship between and each twin.

[0091] In order to improve the position accuracy of the constructed personnel twin and realize better applications based on the position of the personnel twin, the embodiment of the present application provides a personnel twin construction method, which can collect personnel images in the target space and then construct the personnel twin. The target space can be a space composed of entities such as a railway station, an airport, a hospital, a factory, a workshop, a stadium, a conference room, and a cafeteria. The embodiment of the present application pre-installs at least two cameras in the target space, and these cameras synchronously collect the first panoramic image of the target space, that is, each camera collects the first panoramic image at the same collection frequency, and each camera collects the first panoramic image at the same time point.

[0092] The camera in the embodiment of the present application generally refers to a panoramic image camera. Although the panoramic camera field of view is 360°, it is limited by the structure of the target space and the facilities in the space, and there are usually shooting occlusions, and it is difficult for a single camera to completely cover the entire target space. In order to more accurately construct the global 3D model corresponding to the target space and obtain a more accurate position of personnel, the embodiment of the present application adopts at least one camera to cooperate with each other to achieve complete coverage of the target space. The field of view of the at least two cameras completely covers the target space, and images of all corner points used to construct the global 3D model of the target space can be obtained. In addition, there is a field of view overlap between any camera and at least one adjacent camera. That is, the field of view captured by each camera has a field of view overlap with the field of view captured by at least one other camera. The field of view overlap between two adjacent cameras means that although the areas of the target space captured by the two adjacent cameras are not exactly the same, there is an overlapping part between the two. Cameras with overlapping fields of view are adjacent cameras to each other. For example, if camera A and camera B have overlapping fields of view, then camera A is an adjacent camera of camera B, and camera B is also an adjacent camera of camera A.

[0093] In addition, the specific number of cameras that need to be installed in the target space can be selected based on factors such as the structural characteristics of the target space, the size of the space, and the shooting errors of the cameras in different field of view.

[0094] In order to obtain as many first panoramic images of the same person as possible taken by different cameras at the same time, when installing multiple cameras, the embodiment of the present application can set the position of each camera according to the activity area of ​​the personnel in the target area, for example, so that any activity area of ​​the personnel in the target space exists in the field of view of at least two cameras. Optionally, when installing the camera, the embodiment of the present application can install the camera at the top of the target space and shoot downward. In order to simplify subsequent processing, the embodiment of the present application can make the installation height of each camera the same.

[0095] In addition, in this article, 2D means two-dimensional and 3D means three-dimensional.

[0096] like Figure 1As shown, the method for constructing a person twin provided in the embodiment of the present application includes:

[0097] Step 11: Acquire a first panoramic image in a target space captured synchronously by various cameras, wherein the target space is provided with at least two cameras, and the field of view of any camera overlaps with that of at least one adjacent camera.

[0098] Here, each camera synchronously captures a first panoramic image in the target space, and the first panoramic image can specifically be an equirectangular projection. The first panoramic image captured by each camera at the same time can be sent to the personnel twin construction device of the embodiment of the present application for processing by wireless and / or wired means. Here, the synchronous shooting means that each camera shoots at the same time, or the shooting time of each camera is very close. For example, the shooting time difference between each camera is usually less than a preset threshold, which can be 100 milliseconds, 20 milliseconds, 100 microseconds or even smaller.

[0099] Figure 2 An example of installing a camera in a target space is provided, where Figure 2 The left part of is a schematic diagram of the target space 21. Figure 2 The middle part of the figure shows the four cameras installed on the top of the target space in the form of a top view, namely d1~d4. Figure 2 The right part of the diagram is a schematic diagram of the topological results between cameras. In this topological structure, if there is a line connecting two cameras, it means that the fields of view of the two cameras overlap. Figure 2 In the topological structure shown, the field of view of camera d1 overlaps with cameras d2 and d3 respectively, the field of view of camera d2 overlaps with cameras d1 and d4 respectively, the field of view of camera d3 overlaps with cameras d1 and d4 respectively, and the field of view of camera d4 overlaps with cameras d2 and d3 respectively.

[0100] Step 12, perform personnel detection on the first panoramic images synchronously taken by each camera to obtain 2D information of the personnel in the field of view of each camera; based on the 2D information of the personnel in the field of view of each camera, generate a 3D bounding box of the personnel corresponding to each camera in the target space coordinate system.

[0101] Here, the personnel 2D information generally includes a personnel 2D bounding box and a 2D coordinate sequence of human key points. The embodiment of the present application can use the personnel detection algorithm / model of the prior art to detect the personnel present in the first panoramic image. For example, a deep learning model is used to detect the personnel information in each first panoramic image. Specifically, a personnel detection model (such as a YOLO series model) can be used to detect the personnel 2D bounding box of all personnel in the first panoramic image, and the personnel 2D bounding box consists of 4 2D coordinates. A human posture detection model (such as PoseNet, LitePose and other models) can be used to detect the human key points in the 2D bounding box of the personnel, thereby obtaining a 2D coordinate sequence of human key points composed of the 2D coordinates of multiple key points of the human body. The human key points are pre-selected human position points, and the embodiment of the present application does not specifically limit the number of human key points.

[0102] When generating the 3D bounding box of the person corresponding to each camera in the target space coordinate system according to the 2D information of the person in the field of view of each camera, the 2D information of the person in the field of view of each camera can be projected to the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, that is, project each 2D coordinate in the 2D information of the person to the 3D space of the target space to obtain the corresponding 3D coordinates, thereby obtaining the 3D information of the person corresponding to the 2D information of the person, and the 3D information of the person includes the 3D bounding box of the person and the 3D coordinate sequence of the key points of the human body. Among them, the 3D bounding box of the person is composed of 4 3D coordinates corresponding to the 4 2D coordinates of the 2D bounding box of the person, and the 3D coordinate sequence of the key points of the human body is a 3D coordinate sequence composed of the 3D coordinates of the key points corresponding to the 2D coordinates of each key point in the 2D coordinate sequence of the key points of the human body.

[0103] It should be noted that, before step 11, in the embodiment of the present application, it is necessary to jointly calibrate each camera to determine the external parameters of each camera in the target space coordinate system, so as to establish a mapping relationship between each camera coordinate system and the target space coordinate system, and it is also necessary to construct a global 3D model corresponding to the target space. In this way, in step 12, the 2D information of the person in the camera coordinate system can be converted into the 2D information of the person in the target space coordinate system according to the above external parameters. Then, according to the projection formula of the panoramic image to the three-dimensional space and the pre-constructed global 3D model corresponding to the target space, the 2D information of the person in the camera coordinate system is projected to the 3D space of the target space, so as to obtain the 3D information of the person corresponding to the 2D information of the person, including the 3D bounding box of the person and the 3D coordinate sequence of the key points of the human body. The joint calibration of cameras and the construction of the global 3D model corresponding to the target space will be explained later.

[0104] Then, the plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side with a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera. Figure 3 An example of a person 3D bounding box is provided, which is a rectangular parallelepiped structure, with 8 vertices 231 to 238, and a central section thereof is a plane where the person 3D bounding box 22 is located. The 4 vertices of the person 3D bounding box 22 are 221 to 224. The length of the edge between vertex 237 and vertex 238 is a preset length value, and the preset length value can be set according to the pixel width of the detected person image.

[0105] In this way, through step 12, the 3D bounding box of the person corresponding to each camera in the target space coordinate system can be obtained.

[0106] Step 13, generating at least one bounding box set according to the 3D bounding boxes of the people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of the person in each bounding box set is the 3D bounding box of the person corresponding to any camera, and the other 3D bounding boxes of the person are the 3D bounding boxes of the person corresponding to the adjacent cameras of any camera, and overlap with the first 3D bounding box of the person.

[0107] Here, the existence of overlap between two person 3D bounding boxes refers to partial overlap of the two person 3D bounding boxes, complete overlap of the two person 3D bounding boxes, or the presence of one person 3D bounding box in another person 3D bounding box. In the above step 12, the person 3D bounding boxes corresponding to each camera are obtained, that is, the person 3D bounding boxes obtained based on the first panoramic image taken by each camera. In step 13, the embodiment of the present application generates the at least one bounding box set based on the person 3D bounding boxes corresponding to each camera. In the at least one bounding box set, different bounding box sets include different person 3D bounding boxes, that is, there are no two bounding box sets with exactly the same members in the at least one bounding box set.

[0108] An implementation method for generating the at least one bounding box set is provided below:

[0109] (a) traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set;

[0110] (b) When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and then the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to adjacent cameras, and overlap with the traversed person's 3D bounding box;

[0111] (c) When the traversed person's 3D bounding box exists in the generated bounding box set, continue to traverse the next person's 3D bounding box.

[0112] For ease of understanding, the following example is used to illustrate. Here, the first panoramic image captured by camera i at time t is used for person detection to obtain the 3D bounding box of the person corresponding to camera i, where the jth 3D bounding box of the person corresponding to camera i is named as follows: (t, d i ,p j ).

[0113] Assume that in step 12, based on the first panoramic image taken by three cameras, the following bounding box sets corresponding to the cameras are obtained:

[0114] Camera d1 corresponds to three 3D bounding boxes of people, namely: (t, d1, p1), (t, d1, p2), (t, d1, p3);

[0115] Camera d2 corresponds to two 3D bounding boxes of people, namely: (t, d2, p1) and (t, d2, p2);

[0116] Camera d3 corresponds to four 3D bounding boxes of people, namely: (t,d3,p1), (t,d3,p2), (t,d3,p3), (t,d3,p4);

[0117] Assume the following overlapping relationships exist:

[0118] Camera d1 has overlapping fields of view with cameras d2 and d3, that is, the adjacent cameras of camera d1 include d2 and d3;

[0119] Camera d2 has overlapping fields of view with cameras d1 and d4, that is, the adjacent cameras of camera d2 include d1 and d4;

[0120] Camera d3 has overlapping fields of view with cameras d1 and d4, that is, the adjacent cameras of camera d3 include d1 and d4;

[0121] Camera d4 has overlapping fields of view with cameras d2 and d3, that is, the adjacent cameras of camera d2 include d2 and d3;

[0122] (t,d1,p1) overlaps with (t,d2,p2), (t,d3,p1), and (t,d3,p4) respectively;

[0123] (t,d1,p2) overlaps with (t,d3,p1) and (t,d3,p4) respectively;

[0124] (t,d1,p3) overlaps with (t,d3,p3);

[0125] (t,d2,p1) overlaps with (t,d3,p2);

[0126] According to the above implementation, it is assumed that the traversal starts from the 3D bounding boxes of each person corresponding to camera d1:

[0127] (1) First, starting from camera d1, when traversing (t, d1, p1) corresponding to camera d1, since no bounding box set has been generated yet, the Separating Axis Theorem (SAT) algorithm can be used to calculate whether there is overlap between (t, d1, p1) and the 3D bounding boxes of people corresponding to adjacent cameras d2 and d3. If there is overlap, a bounding box set can be generated for (t, d1, p1), such as:

[0128] CoarseSet1={(t,d1,p1),(t,d2,p2),(t,d3,p1),(t,d3,p4)}

[0129] The first bounding box set in the bounding box set is (t, d1, p1), and the other bounding box sets are 3D bounding boxes of people corresponding to adjacent cameras and having overlapping fields of view with (t, d1, p1).

[0130] (2) By analogy, continue to traverse (t, d1, p2) corresponding to camera d1. Since (t, d1, p2) does not appear in the generated bounding box set, a bounding box set can be generated for (t, d1, p2), such as:

[0131] CoarseSet2={(t,d1,p2),(t,d3,p1),(t,d3,p4)}

[0132] (3) Similarly, continue to traverse (t, d1, p3) corresponding to camera d1. Since (t, d1, p3) does not appear in the generated bounding box set, a bounding box set can be generated for (t, d1, p3), such as:

[0133] CoarseSet3={(t,d1,p3),(t,d3,p3)}

[0134] (4) Continue to traverse the 3D bounding box (t, d2, p1) of the person corresponding to camera d2. Since (t, d2, p1) does not appear in the generated bounding box set, a bounding box set can be generated for (t, d2, p1), such as:

[0135] CoarseSet4={(t,d2,p1),(t,d3,p2)}

[0136] (5) Continue to traverse the 3D bounding box (t, d2, p2) of the person corresponding to camera d2. Since (t, d2, p2) already appears in the generated bounding box set CoarseSet1, there is no need to generate a bounding box set for (t, d2, p2). Continue traversal at this time.

[0137] (6) Continue to traverse the 3D bounding box of the person (t, d3, p1) corresponding to camera d3. Since (t, d3, p1) has appeared in the generated bounding box set CoarseSet1, there is no need to generate a bounding box set for (t, d3, p1). At this time, continue to traverse the 3D bounding boxes of the person (t, d3, p2), (t, d3, p3), and (t, d3, p4) corresponding to camera d3. Since the above 3D bounding boxes of the person (t, d3, p2), (t, d3, p3), and (t, d3, p4) have all appeared in the generated bounding box set, the traversal ends.

[0138] So far, a total of 4 bounding box sets are obtained, namely CoarseSet1, CoarseSet2, CoatseSet3, and CoatseSet4.

[0139] The SAT algorithm determines whether there is a collision by determining whether the projections of any two convex polygons at any angle overlap. In the embodiment of the present application, the SAT algorithm can be used to perform collision detection of 3D bounding boxes to determine whether there is overlap in the 3D bounding boxes of people.

[0140] Step 14, traverse each bounding box set, if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, then respectively calculate the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retain the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and delete the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set.

[0141] Here, each bounding box set generated in step 13 is traversed to update the 3D bounding boxes of the people in the bounding box set. If there are no at least two 3D bounding boxes of the people corresponding to the same camera in the currently traversed bounding box set, the 3D bounding boxes of the people in the traversed bounding box set are kept unchanged. If there are at least two 3D bounding boxes of the people corresponding to the same camera in the currently traversed bounding box set, one 3D bounding box of the people is selected from the at least two 3D bounding boxes of the people and retained in the bounding box set, and the unselected 3D bounding boxes of the people in the at least two 3D bounding boxes of the people are deleted from the bounding box set, thereby updating the members in the bounding box set.

[0142] Specifically, when selecting a person 3D bounding box for retention from the at least two person 3D bounding boxes, the posture similarities between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set can be calculated respectively, and then the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes is retained in the traversed bounding box set, and the other person 3D bounding boxes in the at least two person 3D bounding boxes are deleted from the traversed bounding box set.

[0143] Posture similarity between the human bodies pointed to by the two 3D bounding boxes of the two persons. For example, the first similarity between the 3D coordinate sequences of the key points of the human body corresponding to the two 3D bounding boxes of the two persons can be calculated as the posture similarity between the human bodies pointed to by the two 3D bounding boxes of the two persons. For another example, the second similarity between the 3D bounding boxes of the two persons corresponding to the 3D bounding boxes of the two persons can be calculated as the posture similarity between the human bodies pointed to by the two 3D bounding boxes of the two persons. The similarity between the 3D bounding boxes of the persons can be the similarity between the coordinate sequences composed of the 3D coordinates of the 8 vertices of the 3D bounding boxes of the persons. For another example, the above-mentioned first similarity and second similarity can be calculated separately, and then weighted summed, as the posture similarity between the human bodies pointed to by the two 3D bounding boxes of the two persons.

[0144] A specific calculation method for the first similarity is provided below. It should be noted that the following calculation method is only an example of a calculation method that can be used in this application and is not intended to limit this application. The calculation method includes:

[0145] (1) Based on the 3D coordinate sequence of the human body key points corresponding to the 3D bounding box of the person, generate the human body key point vector corresponding to the 3D bounding box of the person.

[0146] Specifically, the coordinate values ​​of each dimension in the 3D coordinate sequence of the key points of the human body corresponding to the 3D bounding box of the person can be used as an element of the intermediate vector to obtain the intermediate vector. Then, the intermediate vector is normalized and regularized to obtain the key point vector of the human body corresponding to the 3D bounding box of the person.

[0147] For example, the 3D coordinate sequence of the key points of the human body can be flattened to obtain the intermediate vector PV′, which is expressed as follows:

[0148] PV′=Flatten((t,d i ,k l ))=[x1,y1,z1,x2,y2,z2,…x 17 ,y 17 ,z 17 ]

[0149] Here, (t,d i ,k l ) indicates camera d i The 3D coordinate sequence of the key points of the human body corresponding to the 3D bounding box of the lth person in the first panoramic image taken at time t. Flatten(·) indicates the flattening process. Assume that there are 17 key points of the human body, and the 3D coordinate of the key point m of the human body is (x m ,y m ,z m ), where m ranges from 1 to 17. Then, the intermediate vector PV′ is subjected to min-max normalization and L2 regularization to obtain the human key point vector PV corresponding to the 3D bounding box of the person.

[0150] (2) Calculate the cosine similarity of the human body key point vectors corresponding to the two 3D bounding boxes of the two persons as the posture similarity between the human bodies pointed to by the two 3D bounding boxes of the two persons. This calculation method can be expressed by the formula:

[0151] Here, the cosine similarity calculation formula is used to calculate the similarity (PV1, PV2) between the two human key point vectors PV1 and PV2, which is used as the posture similarity between the human bodies pointed to by the 3D bounding box of the person. The calculation formula is as follows:

[0152]

[0153] For example, continuing the above example, traversing each bounding box set, when traversing the bounding box set CoarseSet1, since there are two people 3D bounding boxes (t,d3,p1) and (t,d3,p4) corresponding to the same camera d3 in the bounding box set CoarseSet1 = {(t,d1,p1),(t,d2,p2),(t,d3,p1),(t,d3,p4)}, therefore, according to the above method, calculate (t,d1, The similarity between (t,s1,p1) and (t,d3,p4) is calculated, and then the 3D bounding box of the person with high similarity to (t,d1,p1) (assuming it is (t,d3,p4)) is retained. In this way, the bounding box set CosrseSet1 is updated to {(t,d1,p1),(t,d2,p2),(t,d3,p4)}.

[0154] Similarly, when traversing the bounding box set CoarseSet2, since the bounding box set CoarseSet2 = {(t,d1,p2), (t,d3,p1), (t,d3,p4)}, there are two person 3D bounding boxes (t,d3,p1) and (t,d3,p4) corresponding to the same camera d3, therefore, according to the above method, the similarity between (t,d1,p2) and (t,d3,p1) is calculated, and the similarity between (t,d1,p2) and (t,d3,p4) is calculated, and then, the person 3D bounding box (assumed to be (t,d3,p1)) with a high similarity to (t,d1,p2) of the two person 3D bounding boxes is retained, so that the bounding box set CoarseSet2 is updated to {(t,d1,p2), (t,d3,p1)}.

[0155] Next, we continue to traverse CoarseSet3 and oarseSet4. Since there are no at least two 3D bounding boxes of people corresponding to the same camera in these two bounding box sets, the two bounding box sets remain unchanged. Through the above processing, one bounding box set includes the 3D bounding boxes of the same person in different camera fields of view.

[0156] Step 15, based on the 3D bounding box of the person in each bounding box set, generate the coordinates of a person twin; based on the coordinates of each person twin, construct the person twin in the target space twin corresponding to the target space.

[0157] Here, based on the 3D bounding box of the person in each bounding box set obtained in step 14, the coordinates of a person twin corresponding to the bounding box set are generated. In this way, the coordinates of at least one person twin can be obtained, and then based on the coordinates of each person twin obtained, the person twin can be constructed in the target space twin corresponding to the target space.

[0158] When generating the coordinates of the person twin corresponding to a certain bounding box set, the 3D coordinate sequence of the human key points corresponding to all the 3D bounding boxes of the person in the bounding box set can be obtained, and then the average value of the 3D coordinates of the same human key point can be calculated to obtain the coordinates of the person twin corresponding to the bounding box set. When calculating the average value of the 3D coordinates of the same human key point, a certain key point, such as the head key point, the foot key point or other key point, can be selected, and the average value of the 3D coordinates of the key point can be calculated as the coordinates of the person twin.

[0159] Through the above steps, the embodiment of the present application generates a 3D bounding box of the same person corresponding to different cameras based on the panoramic images obtained by multiple cameras, and generates the coordinates of the person's twin based on the coordinates of the key points of the human body of the same person corresponding to different cameras, thereby reducing the person positioning error, improving the positioning accuracy, and better handling the problem of inaccurate personnel information matching results caused by the close distance of people, and can provide better support for subsequent applications based on the position of person twins.

[0160] Before the above step 11, the embodiment of the present application may also pre-construct a target space twin corresponding to the target space. The construction of the target space twin specifically includes:

[0161] Step A, obtaining the second panoramic image taken by each camera at the same time, and converting the second panoramic image taken by each camera into a local 3D model in the target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image.

[0162] Here, a group of identification points corresponding to the camera is pre-set in the field of view of each camera, wherein, when the first camera and the second camera are adjacent cameras with overlapping fields of view, the multiple identification points corresponding to the first camera set in the overlapping field of view are the same as the multiple identification points corresponding to the second camera set in the overlapping field of view. For example, assuming that camera 1 and camera 2 are adjacent cameras, there is an overlapping field of view between the two; a first group of identification points 11, 12, 13, 14, 15, 16, 17, 18 corresponding to camera 1 is set in the field of view of camera 1, and a second group of identification points 21, 22, 23, 24, 15, 16, 17, 18 corresponding to camera 2 is set in the field of view of camera 2. Among them, identification points 15, 16, 17, 18 are set in the overlapping field of view of camera 1 and camera 2, and these four identification points correspond to camera 1 and camera 2. In other words, the multiple identification points corresponding to camera 1 set in the overlapping field of view are exactly the same as the multiple identification points corresponding to camera 2 set in the overlapping field of view.

[0163] The second panoramic image is usually an equirectangular projection. The embodiment of the present application determines multiple identification points set for a camera from the second panoramic image taken by a camera as multiple corner points of the local 3D model. According to the 2D coordinates of the multiple identification points in the second panoramic image and the projection formula of the first panoramic image into the three-dimensional space, the second panoramic image is converted into a local 3D model.

[0164] Step B, based on the external parameters of each camera, stitching the obtained local 3D models to obtain a global 3D model corresponding to the target space in the target space coordinate system, and obtaining the 3D coordinates of each corner point of the global 3D model.

[0165] Camera extrinsics usually include rotation matrix and translation matrix, which can be obtained in the following ways:

[0166] Firstly, a target space coordinate system of the target space is established with a corner point of the target space as the origin, and the coordinates of the identification point in the target space coordinate system are obtained. Figure 4 An example of setting the origin 400 of the target space coordinate system is shown. Figure 4 In the target space, four cameras are set in the top area. In addition, the direction of the XYZ axis of the target space coordinate system can be determined according to the position of the corner point in the target space. The coordinates of the marker point in the target space coordinate system can be measured by the distance measurement tool.

[0167] Then, the coordinates of the identification point in the second panoramic image taken by each camera are obtained; according to the coordinates of the identification point in the second panoramic image taken by each camera and the coordinates of the identification point in the target space coordinate system, the coordinates of each camera in the target space coordinate system are determined.

[0168] Then, each camera is jointly calibrated according to the identification points and the coordinates of each camera in the target space coordinate system to determine the extrinsic parameters of each camera in the target space coordinate system.

[0169] Figure 5 An example of stitching a global 3D model corresponding to a target space using four local 3D models converted from a first panoramic image taken by four cameras is provided. Figure 4 8 identification points are selected from the first panoramic image taken by each camera in the image as 8 corner points of the local 3D model, thereby generating 4 local 3D models in the shape of rectangular blocks. Then, based on the external parameters of the camera, the positions of these local 3D models in the target space coordinate system are determined, and these local 3D models are spliced ​​to obtain the global 3D model. Figure 6 An example of obtaining eight corner points of the global 3D model based on the global 3D model obtained by stitching is shown.

[0170] Step C: constructing a target space twin corresponding to the target space according to the 3D coordinates of each corner point of the global 3D model.

[0171] Furthermore, the embodiment of the present application can also construct a camera twin corresponding to each camera according to the coordinates of each camera in the target space coordinate system.

[0172] The following provides an implementation method for constructing related twins (such as target space twins, camera twins, and personnel twins, etc.), including:

[0173] 1) Drawing of physical entity images.

[0174] Here, drawing software can be used to generate texture images of physical entities, where the physical entities are entities that need to build twins, such as target space, camera, personnel, etc. Drawing software can be used to draw images of target space and camera according to the real scene of the target space and the camera used; and drawing software can be used to draw a general character image of a person. The drawing software can be one of CAD, Maya and other software.

[0175] 2) Modeling of physical entities;

[0176] Here, 3D modeling software can be used to build 3D models of the target space, the camera, and the person. For example, the 3D modeling software is used to build a three-dimensional model of the target space according to the spatial structure information of the target space (such as the 3D coordinates of each corner point of the global 3D model of the target space); the 3D modeling software is used to build a three-dimensional model of the camera in 1:1 according to the camera used; the 3D modeling software is used to build a three-dimensional model of the person. The 3D modeling software can be one of software such as Three.js, Blender, and 3DMax.

[0177] 3) Model mapping;

[0178] Attach corresponding texture images to the 3D models of the target space, camera, and personnel. The model mapping process can be achieved through software such as Three.js, Blender, and 3DMax.

[0179] 4) Generation of twins;

[0180] The 3D model of the target space is placed in the web interface, ensuring that the target space coordinate system coincides with the coordinate system in the interface, and the target space twin is obtained. According to the coordinates of the camera, the instance obtains the corresponding camera twin and presents it in the target space twin. According to the coordinates of the person twin, the instance obtains the corresponding person twin and presents it in the target space twin, thereby obtaining the target space twin that contains the camera twin and the person twin.

[0181] Through the above steps, the embodiment of the present application uses image projection calculation instead of manual measurement to obtain the spatial structure required to construct the target space twin. First, the first panoramic image is mapped to the 3D space through the projection formula to obtain a local 3D model, and then the local 3D models are spliced ​​to obtain a global 3D model. Finally, the 3D coordinates of each corner point of the target space are obtained from the global 3D model to guide the construction of the target space twin.

[0182] Based on the above method, the present application also provides a device for implementing the above method. Figure 7 , the embodiment of the present application provides a person twin construction device, including:

[0183] A first acquisition module 71 is used to acquire a first panoramic image in a target space photographed synchronously by each camera, wherein the target space is provided with at least two cameras, and the field of view of any camera overlaps with that of at least one adjacent camera;

[0184] The first generating module 72 is used to perform personnel detection on the first panoramic images synchronously taken by each camera to obtain two-dimensional 2D information of the personnel in the field of view of each camera; and generate a 3D bounding box of the personnel corresponding to each camera in the target space coordinate system according to the 2D information of the personnel in the field of view of each camera;

[0185] The second generating module 73 is used to generate at least one bounding box set according to the 3D bounding boxes of people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of a person in each bounding box set is the 3D bounding box of a person corresponding to any camera, and the other 3D bounding boxes of a person are the 3D bounding boxes of a person corresponding to the adjacent cameras of the any camera, and overlap with the first 3D bounding box of a person;

[0186] The traversal processing module 74 is used to traverse each bounding box set, and if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, then respectively calculate the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retain the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and delete the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set;

[0187] The first construction module 75 is used to generate the coordinates of a person twin based on the person's 3D bounding box in each bounding box set; and construct the person twin in the target space twin corresponding to the target space based on the coordinates of each person twin.

[0188] Through the above modules, the embodiment of the present application can improve the position accuracy of the constructed person twin.

[0189] Optionally, the person 2D information includes a person 2D bounding box and a 2D coordinate sequence of human body key points; and the first generating module is further used to:

[0190] Projecting the 2D information of the person in the field of view of each camera into the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, wherein the 3D information of the person includes a 3D bounding box of the person and a 3D coordinate sequence of key points of the human body;

[0191] The plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side of a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera.

[0192] Optionally, the second generating module is further used to:

[0193] Traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set;

[0194] When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to the adjacent cameras, and overlap with the traversed person's 3D bounding box;

[0195] When the traversed person 3D bounding box exists in the generated bounding box set, continue to traverse the next person 3D bounding box.

[0196] Optionally, the first building module is further used to:

[0197] For each bounding box set, obtain the 3D coordinate sequence of the human body key points corresponding to the 3D bounding boxes of all people in the bounding box set, and calculate the average value of the 3D coordinates of the same human body key point to obtain the coordinates of the person twin corresponding to the bounding box set.

[0198] Optionally, the above device further comprises:

[0199] The similarity calculation module is used to calculate the posture similarity between the human bodies pointed to by the 3D bounding boxes of two persons in the following manner:

[0200] Generate a human key point vector corresponding to the 3D bounding box of the person based on the 3D coordinate sequence of the human key points corresponding to the 3D bounding box of the person;

[0201] The cosine similarity of the human body key point vectors corresponding to the two person 3D bounding boxes is calculated as the posture similarity between the human bodies pointed to by the two person 3D bounding boxes.

[0202] Optionally, the similarity calculation module is further used to:

[0203] The coordinate value of each dimension in the 3D coordinate sequence of the key point of the human body corresponding to the 3D bounding box of the person is respectively used as an element of the intermediate vector to obtain the intermediate vector;

[0204] The intermediate vector is normalized and regularized to obtain the human body key point vector corresponding to the 3D bounding box of the person.

[0205] Optionally, the above device further comprises:

[0206] a conversion module, used for acquiring a second panoramic image captured by each camera at the same time, and converting the second panoramic image captured by each camera into a local 3D model in a target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image; a group of identification points corresponding to the camera is set in the field of view of each camera, wherein when the first camera and the second camera are adjacent cameras with overlapping fields of view, a plurality of identification points corresponding to the first camera set in the overlapping field of view are the same as a plurality of identification points corresponding to the second camera set in the overlapping field of view;

[0207] A stitching module, used to stitch the local 3D models together based on the external parameters of each camera, to obtain a global 3D model corresponding to the target space in the target space coordinate system, and to obtain the 3D coordinates of each corner point of the global 3D model;

[0208] The second construction module is used to construct a target space twin corresponding to the target space according to the 3D coordinates of each corner point of the global 3D model.

[0209] Optionally, the above device further comprises:

[0210] External parameter determination module, used for:

[0211] Taking a corner point of the target space as the origin, establishing a target space coordinate system of the target space, and obtaining the coordinates of the identification point in the target space coordinate system;

[0212] Obtaining coordinates of the identification points in the second panoramic image captured by each camera;

[0213] According to the coordinates of the identification points in the second panoramic image taken by each camera and the coordinates of the identification points in the target space coordinate system, the coordinates of each camera in the target space coordinate system are determined, and according to the identification points and the coordinates of each camera in the target space coordinate system, each camera is jointly calibrated to determine the external parameters of each camera in the target space coordinate system.

[0214] Optionally, the above device further comprises:

[0215] The third construction module is used to construct the camera twin corresponding to each camera according to the coordinates of each camera in the target space coordinate system.

[0216] It should be noted that the various devices / systems provided in the above embodiments are devices corresponding to the above-mentioned personnel twin construction method, and the implementation methods in the above-mentioned embodiments are applicable to the embodiments of the device, and can also achieve the same technical effects. The above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment in this embodiment will not be specifically repeated here.

[0217] Please refer to Figure 8 , the embodiment of the present application also provides a hardware structure block diagram of a person twin construction device, such as Figure 8 As shown, the personnel twin construction device 800 includes:

[0218] Processor 802; and

[0219] a memory 804 in which computer program instructions are stored,

[0220] When the computer program instructions are executed by the processor, the processor 802 is caused to perform the following steps:

[0221] Acquire a first panoramic image in a target space synchronously photographed by each camera, wherein the target space is provided with at least two cameras, and the field of view of any camera overlaps with that of at least one adjacent camera;

[0222] Performing personnel detection on the first panoramic images synchronously taken by each camera to obtain two-dimensional 2D information of the personnel in the field of view of each camera; generating 3D bounding boxes of the personnel corresponding to each camera in the target space coordinate system according to the 2D information of the personnel in the field of view of each camera;

[0223] Generate at least one bounding box set according to the 3D bounding boxes of people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of a person in each bounding box set is the 3D bounding box of a person corresponding to any camera, and the other 3D bounding boxes of a person are the 3D bounding boxes of people corresponding to the adjacent cameras of any camera, and overlap with the first 3D bounding box of a person;

[0224] Traversing each bounding box set, if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, respectively calculating the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retaining the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and deleting the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set;

[0225] Based on the 3D bounding box of the person in each bounding box set, the coordinates of a person twin are generated; based on the coordinates of each person twin, the person twin is constructed in the target space twin corresponding to the target space.

[0226] It should be noted that the various systems provided in the above embodiments are devices corresponding to the above-mentioned personnel twin construction method, and the implementation methods in the above-mentioned embodiments are applicable to the embodiments of the device, and can also achieve the same technical effects. The above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment in this embodiment will not be specifically described here.

[0227] Furthermore, if Figure 8 As shown, the personnel twin construction device 800 also includes a network interface 801, an input device 803, a hard disk 805, and a display device 806.

[0228] The above-mentioned interfaces and devices can be interconnected through a bus architecture. The bus architecture can be a bus and bridge that can include any number of interconnected buses. Specifically, one or more central processing units (CPUs) and / or graphics processing units (GPUs) represented by processor 802, and various circuits of one or more memories represented by memory 804 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together. It can be understood that the bus architecture is used to achieve connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are all well known in the art, so they are not described in detail herein.

[0229] The network interface 801 can be connected to each camera via a wired or invisible connection, receive data such as panoramic images from the camera, and save the received data in the hard disk 805 .

[0230] The input device 803 can receive various instructions input by the operator and send them to the processor 802 for execution. The input device 803 can include a keyboard or a pointing device (for example, a mouse, a trackball, a touch pad or a touch screen, etc.).

[0231] The display device 806 can display the results obtained by the processor 802 executing instructions, such as displaying the model training progress.

[0232] The memory 804 is used to store programs and data necessary for the operation of the operating system, as well as data such as intermediate results during the calculation process of the processor 802.

[0233] It is understood that the memory 804 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. The memory 804 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0234] In some implementations, the memory 804 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 8041 and an application program 8042 .

[0235] The operating system 8041 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program 8042 includes various application programs, such as a browser, etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application program 8042.

[0236] The method disclosed in the above embodiment of the present application can be applied to the processor 802, or implemented by the processor 802. The processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 802. The above processor 802 can be a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 804, and the processor 802 reads the information in the memory 804 and completes the steps of the above method in combination with its hardware.

[0237] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0238] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0239] Optionally, the person 2D information includes a person 2D bounding box and a 2D coordinate sequence of key points of the human body; when the computer program is executed by the processor 802, the following steps may also be implemented:

[0240] Projecting the 2D information of the person in the field of view of each camera into the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, wherein the 3D information of the person includes a 3D bounding box of the person and a 3D coordinate sequence of key points of the human body;

[0241] The plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side of a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera.

[0242] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0243] Traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set;

[0244] When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to the adjacent cameras, and overlap with the traversed person's 3D bounding box;

[0245] When the traversed person 3D bounding box exists in the generated bounding box set, continue to traverse the next person 3D bounding box.

[0246] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0247] For each bounding box set, obtain the 3D coordinate sequence of the human body key points corresponding to the 3D bounding boxes of all people in the bounding box set, and calculate the average value of the 3D coordinates of the same human body key point to obtain the coordinates of the person twin corresponding to the bounding box set.

[0248] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0249] The pose similarity between the two human bodies pointed to by the 3D bounding boxes of the two persons is calculated as follows:

[0250] Generate a human key point vector corresponding to the 3D bounding box of the person based on the 3D coordinate sequence of the human key points corresponding to the 3D bounding box of the person;

[0251] The cosine similarity of the human body key point vectors corresponding to the two person 3D bounding boxes is calculated as the posture similarity between the human bodies pointed to by the two person 3D bounding boxes.

[0252] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0253] The coordinate value of each dimension in the 3D coordinate sequence of the key point of the human body corresponding to the 3D bounding box of the person is respectively used as an element of the intermediate vector to obtain the intermediate vector;

[0254] The intermediate vector is normalized and regularized to obtain the human body key point vector corresponding to the 3D bounding box of the person.

[0255] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0256] Acquire a second panoramic image captured by each camera at the same time, and convert the second panoramic image captured by each camera into a local 3D model in a target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image; a group of identification points corresponding to the camera is set in the field of view of each camera, wherein when the first camera and the second camera are adjacent cameras with overlapping fields of view, a plurality of identification points corresponding to the first camera set in the overlapping field of view are the same as a plurality of identification points corresponding to the second camera set in the overlapping field of view;

[0257] Based on the external parameters of each camera, the local 3D models are spliced ​​to obtain a global 3D model corresponding to the target space in the target space coordinate system, and the 3D coordinates of each corner point of the global 3D model are obtained;

[0258] According to the 3D coordinates of each corner point of the global 3D model, a target space twin corresponding to the target space is constructed.

[0259] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0260] Follow the steps below to obtain the external parameters of each camera:

[0261] Taking a corner point of the target space as the origin, establishing a target space coordinate system of the target space, and obtaining the coordinates of the identification point in the target space coordinate system;

[0262] Obtaining coordinates of the identification points in the second panoramic image captured by each camera;

[0263] According to the coordinates of the identification points in the second panoramic image taken by each camera and the coordinates of the identification points in the target space coordinate system, the coordinates of each camera in the target space coordinate system are determined, and according to the identification points and the coordinates of each camera in the target space coordinate system, each camera is jointly calibrated to determine the external parameters of each camera in the target space coordinate system.

[0264] Optionally, when the computer program is executed by the processor 802, the following steps may also be implemented:

[0265] According to the coordinates of each camera in the target space coordinate system, the camera twin corresponding to each camera is constructed.

[0266] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0267] In some embodiments of the present application, a computer-readable storage medium is further provided, on which a program is stored, and when the program is executed by a processor, the following steps are implemented:

[0268] Acquire a first panoramic image in a target space synchronously photographed by each camera, wherein the target space is provided with at least two cameras, and the field of view of any camera overlaps with that of at least one adjacent camera;

[0269] Performing personnel detection on the first panoramic images synchronously taken by each camera to obtain 2D information of the personnel in the field of view of each camera; generating a 3D bounding box of the personnel in the field of view of each camera according to the 2D information of the personnel;

[0270] Generate at least one bounding box set according to each person 3D bounding box in each camera field of view, wherein the first person 3D bounding box in each bounding box set is a person 3D bounding box in any camera field of view, and the other person 3D bounding boxes are people 3D bounding boxes in adjacent camera fields of view that overlap with the first person 3D bounding box;

[0271] Traverse each bounding box set, and if there are at least two person 3D bounding boxes in the same camera field of view in the traversed bounding box set, only retain one of the at least two person 3D bounding boxes in the traversed bounding box set, and the retained person 3D bounding box is the person 3D bounding box among the at least two person 3D bounding boxes that has the highest similarity with the first person 3D bounding box in the traversed bounding box set;

[0272] Based on the 3D bounding box of the person in each bounding box set, the coordinates of a person twin are generated; based on the coordinates of each person twin, the person twin is constructed in the target space twin corresponding to the target space.

[0273] When the program is executed by the processor, it can implement all the implementation methods in the above-mentioned personnel twin construction method and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0274] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0275] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0276] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0277] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0278] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0279] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0280] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for constructing a personnel twin, characterized in that: include: Acquire a first panoramic image in a target space synchronously photographed by each camera, wherein the target space is provided with at least two cameras, and the field of view of any camera overlaps with that of at least one adjacent camera; Performing personnel detection on the first panoramic images synchronously taken by each camera to obtain two-dimensional 2D information of the personnel in the field of view of each camera; generating 3D bounding boxes of the personnel corresponding to each camera in the target space coordinate system according to the 2D information of the personnel in the field of view of each camera; Generate at least one bounding box set according to the 3D bounding boxes of people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of a person in each bounding box set is the 3D bounding box of a person corresponding to any camera, and the other 3D bounding boxes of a person are the 3D bounding boxes of people corresponding to the adjacent cameras of any camera, and overlap with the first 3D bounding box of a person; Traversing each bounding box set, if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, respectively calculating the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retaining the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and deleting the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set; Based on the 3D bounding box of the person in each bounding box set, the coordinates of a person twin are generated; based on the coordinates of each person twin, the person twin is constructed in the target space twin corresponding to the target space.

2. The method according to claim 1, characterized in that The 2D information of the person includes a 2D bounding box of the person and a 2D coordinate sequence of key points of the human body; the 3D bounding box of the person corresponding to each camera in the target space coordinate system is generated according to the 2D information of the person in the field of view of each camera, including: Projecting the 2D information of the person in the field of view of each camera into the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, wherein the 3D information of the person includes a 3D bounding box of the person and a 3D coordinate sequence of key points of the human body; The plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side of a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera.

3. The method according to claim 1, characterized in that The generating at least one bounding box set according to the 3D bounding boxes of the personnel corresponding to each camera in the target space coordinate system includes: Traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set; When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to the adjacent cameras, and overlap with the traversed person's 3D bounding box; When the traversed person 3D bounding box exists in the generated bounding box set, continue to traverse the next person 3D bounding box.

4. The method according to claim 2, characterized in that The coordinates of a person twin body are generated based on the person 3D bounding box in each bounding box set, including: For each bounding box set, obtain the 3D coordinate sequence of the human body key points corresponding to the 3D bounding boxes of all people in the bounding box set, and calculate the average value of the 3D coordinates of the same human body key point to obtain the coordinates of the person twin corresponding to the bounding box set.

5. The method according to claim 2, characterized in that Also includes: The pose similarity between the two human bodies pointed to by the 3D bounding boxes of the two persons is calculated as follows: Generate a human key point vector corresponding to the 3D bounding box of the person based on the 3D coordinate sequence of the human key points corresponding to the 3D bounding box of the person; The cosine similarity of the human body key point vectors corresponding to the two person 3D bounding boxes is calculated as the posture similarity between the human bodies pointed to by the two person 3D bounding boxes.

6. The method according to claim 5, characterized in that The generating of the human body key point vector corresponding to the 3D bounding box of the person includes: The coordinate value of each dimension in the 3D coordinate sequence of the key point of the human body corresponding to the 3D bounding box of the person is respectively used as an element of the intermediate vector to obtain the intermediate vector; The intermediate vector is normalized and regularized to obtain the human body key point vector corresponding to the 3D bounding box of the person.

7. The method according to claim 1, characterized in that Also includes: Acquire a second panoramic image captured by each camera at the same time, and convert the second panoramic image captured by each camera into a local 3D model in a target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image; a group of identification points corresponding to the camera is set in the field of view of each camera, wherein when the first camera and the second camera are adjacent cameras with overlapping fields of view, a plurality of identification points corresponding to the first camera set in the overlapping field of view are the same as a plurality of identification points corresponding to the second camera set in the overlapping field of view; Based on the external parameters of each camera, the local 3D models are spliced ​​to obtain a global 3D model corresponding to the target space in the target space coordinate system, and the 3D coordinates of each corner point of the global 3D model are obtained; According to the 3D coordinates of each corner point of the global 3D model, a target space twin corresponding to the target space is constructed.

8. The method according to claim 7, characterized in that Also includes: Follow the steps below to obtain the external parameters of each camera: Taking a corner point of the target space as the origin, establishing a target space coordinate system of the target space, and obtaining the coordinates of the identification point in the target space coordinate system; Obtaining coordinates of the identification points in the second panoramic image captured by each camera; According to the coordinates of the identification points in the second panoramic image taken by each camera and the coordinates of the identification points in the target space coordinate system, the coordinates of each camera in the target space coordinate system are determined, and according to the identification points and the coordinates of each camera in the target space coordinate system, each camera is jointly calibrated to determine the external parameters of each camera in the target space coordinate system.

9. The method according to claim 8, characterized in that Also includes: According to the coordinates of each camera in the target space coordinate system, the camera twin corresponding to each camera is constructed.

10. A device for constructing a twin of a person, characterized in that: include: A first acquisition module is used to acquire a first panoramic image in a target space photographed synchronously by each camera, wherein at least two cameras are provided in the target space, and the field of view of any camera overlaps with that of at least one adjacent camera; The first generation module is used to perform personnel detection on the first panoramic images synchronously taken by each camera to obtain two-dimensional 2D information of the personnel in the field of view of each camera; based on the 2D information of the personnel in the field of view of each camera, a 3D bounding box of the personnel corresponding to each camera in the target space coordinate system is generated; The second generating module is used to generate at least one bounding box set according to the 3D bounding boxes of people corresponding to each camera in the target space coordinate system, wherein the first 3D bounding box of a person in each bounding box set is the 3D bounding box of a person corresponding to any camera, and the other 3D bounding boxes of a person are the 3D bounding boxes of a person corresponding to the adjacent cameras of the any camera, and overlap with the first 3D bounding box of a person; A traversal processing module is used to traverse each bounding box set, and if there are at least two person 3D bounding boxes corresponding to the same camera in the traversed bounding box set, then respectively calculate the posture similarity between the human body pointed to by the at least two person 3D bounding boxes and the human body pointed to by the first person 3D bounding box in the traversed bounding box set, retain the person 3D bounding box with the highest posture similarity among the at least two person 3D bounding boxes in the traversed bounding box set, and delete the other person 3D bounding boxes in the at least two person 3D bounding boxes from the traversed bounding box set; The first construction module is used to generate the coordinates of a person twin based on the person's 3D bounding box in each bounding box set; based on the coordinates of each person twin, the person twin is constructed in the target space twin corresponding to the target space.

11. The device according to claim 10, characterized in that The person 2D information includes a person 2D bounding box and a 2D coordinate sequence of key points of the human body; the first generating module is further used for: Projecting the 2D information of the person in the field of view of each camera into the 3D space of the target space to obtain the 3D information of the person corresponding to the 2D information of the person, wherein the 3D information of the person includes a 3D bounding box of the person and a 3D coordinate sequence of key points of the human body; The plane where the 3D bounding box of the person is located is used as the central section of the 3D bounding box of the person, and a side of a preset length and perpendicular to the central section is generated to obtain the 3D bounding box of the person corresponding to each camera.

12. The device according to claim 10, characterized in that The second generating module is further used for: Traverse the 3D bounding boxes of each person corresponding to each camera, and determine whether the traversed 3D bounding box of the person exists in the generated bounding box set; When the traversed person's 3D bounding box does not exist in the generated bounding box set, a bounding box set is generated for the traversed person's 3D bounding box, and the next person's 3D bounding box is traversed, wherein the first person's 3D bounding box in the bounding box set is the traversed person's 3D bounding box, and the other bounding boxes are the person's 3D bounding boxes corresponding to the adjacent cameras, and overlap with the traversed person's 3D bounding box; When the traversed person 3D bounding box exists in the generated bounding box set, continue to traverse the next person 3D bounding box.

13. The device according to claim 11, characterized in that The first building block is further used for: For each bounding box set, obtain the 3D coordinate sequence of the human body key points corresponding to the 3D bounding boxes of all people in the bounding box set, and calculate the average value of the 3D coordinates of the same human body key point to obtain the coordinates of the person twin corresponding to the bounding box set.

14. The device according to claim 11, characterized in that Also includes: The similarity calculation module is used to calculate the posture similarity between the human bodies pointed to by the 3D bounding boxes of two persons in the following manner: Generate a human key point vector corresponding to the 3D bounding box of the person based on the 3D coordinate sequence of the human key points corresponding to the 3D bounding box of the person; The cosine similarity of the human body key point vectors corresponding to the two person 3D bounding boxes is calculated as the posture similarity between the human bodies pointed to by the two person 3D bounding boxes.

15. The device according to claim 10, characterized in that Also includes: a conversion module, used for acquiring a second panoramic image captured by each camera at the same time, and converting the second panoramic image captured by each camera into a local 3D model in a target space coordinate system, wherein the corner points of the local 3D model correspond to the identification points set for the camera in the second panoramic image; a group of identification points corresponding to the camera is set in the field of view of each camera, wherein when the first camera and the second camera are adjacent cameras with overlapping fields of view, a plurality of identification points corresponding to the first camera set in the overlapping field of view are the same as a plurality of identification points corresponding to the second camera set in the overlapping field of view; A stitching module, used to stitch the local 3D models together based on the external parameters of each camera, to obtain a global 3D model corresponding to the target space in the target space coordinate system, and to obtain the 3D coordinates of each corner point of the global 3D model; The second construction module is used to construct a target space twin corresponding to the target space according to the 3D coordinates of each corner point of the global 3D model.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a human twin as described in any one of claims 1 to 9 are implemented.

Images