Overlapping activity area detection method, medium, equipment and product

Abstract

The invention relates to the technical field of visual computing, and particularly provides an overlapping activity area detection method, medium, equipment and product, and the method can comprise the steps: projecting a plane polygon area which is in a world calibration coordinate system and is parallel to the ground plane into images collected by each camera in a multi-camera stereoscopic vision system, obtaining each projection polygon area; obtaining each overlapping area of each projection polygon area and the image acquired by each camera; performing back projection mapping on each overlapping region to the world calibration coordinate system to obtain each mapping polygon region; and performing fusion calculation on each mapping polygon region to obtain an overlapping activity region which is used for providing a standing reference for the motion capture actor. According to some embodiments of the invention, efficient and accurate detection of the overlapping area of the multi-camera stereoscopic vision system can be realized.

Description

Technical Field

[0001] This application relates to the field of visual computing technology. Specifically, it relates to a method, medium, device and product for detecting overlapping activity areas. Background Art

[0002] A multi-camera stereo vision system restores the 3D positional relationship in a scene through image acquisition from different perspectives of the same scene and triangulation calculation. In the field of optical human motion capture, a motion capture actor can better reconstruct the motion skeleton only when captured by as many motion capture cameras as possible simultaneously.

[0003] Currently, in the prior art, usually before the motion capture starts, a person or an object is placed at different position points, and it is observed how many cameras can capture the person or the object at this moment, and then their positions are marked respectively, so as to obtain an effective overlapping activity area that can be perceived by the subject. Obviously, this manual marking method requires the cooperation of more than two people and cannot be completed in one step. As a result, the efficiency of detecting the overlapping activity area in the motion capture scene is relatively low.

[0004] Therefore, how to provide a technical solution for an efficient method for detecting overlapping activity areas has become a technical problem that urgently needs to be solved. Summary of the Invention

[0005] Some embodiments of this application aim to provide a method, medium, device and product for detecting overlapping activity areas. Through the technical solutions of the embodiments of this application, automatic detection of the overlapping activity area can be achieved without manual participation, improving the detection accuracy and efficiency.

[0006] In a first aspect, some embodiments of this application provide a method for detecting an overlapping activity area, including: projecting a planar polygon area parallel to the ground plane in a world calibration coordinate system onto the images collected by each camera in a multi-camera stereo vision system to obtain each projected polygon area; obtaining each overlapping area between each projected polygon area and the images collected by each camera; back-projecting and mapping each overlapping area into the world calibration coordinate system to obtain each mapped polygon area; performing a fusion calculation on each mapped polygon area to obtain an overlapping activity area, where the overlapping activity area is used to provide a standing position reference for a motion capture actor.

[0007] Some embodiments of the present application solve the overlapping regions by projecting a planar polygon region parallel to the ground plane in the world calibration coordinate system onto the images collected by each camera, and then back-projecting and mapping each overlapping region onto the world calibration coordinate system to obtain each mapped polygon region; finally, a fusion calculation is performed to obtain the overlapping activity region. The embodiments of the present application convert the complex calculation problem in 3D space to a 2D plane. Through the homography characteristic between the ground plane and the camera imaging plane in the world calibration coordinate system, the polygon region parallel to the ground plane in the world calibration coordinate system is transformed into another corresponding camera image to obtain the overlapping region, and then back-projecting and mapping it onto the world calibration coordinate system on the ground to calculate the overlapping activity region. It has the characteristics of high efficiency, automation and convenience, and can give a clear indication feedback to the person being photographed during picture capture. The accuracy and efficiency of the whole solution are relatively high.

[0008] In some embodiments, projecting the planar polygon region parallel to the ground plane in the world calibration coordinate system onto the images collected by each camera in a multi-camera stereo vision system to obtain each projected polygon region includes: obtaining the projection transformation equation between the world calibration coordinate system and each camera; using the projection transformation equation to obtain each projected polygon region of the planar polygon region parallel to the ground plane in the image coordinate system of each camera.

[0009] Some embodiments of the present application can achieve accurate projection of the polygon region by implementing the projection of the polygon region in the world calibration coordinate system onto the camera through the projection transformation equation.

[0010] In some embodiments, the projection transformation equation is obtained by the following method: constructing the world calibration coordinate system, wherein the horizontal plane of the world calibration coordinate system is parallel to the ground plane; obtaining the calibration parameters of each camera relative to the world calibration coordinate system, wherein the calibration parameters include: internal parameter matrix, camera distortion coefficient, rotation matrix and translation matrix; constructing the projection transformation equation based on the world coordinates of the plane polygon region to be projected in the world calibration coordinate system and the calibration parameters.

[0011] Some embodiments of the present application can accurately project the polygon region parallel to the ground in the world calibration coordinate system into the camera image coordinate system by constructing the world calibration coordinate system, determining the calibration parameters with the camera, and then constructing the projection transformation equation, providing effective data support for subsequent calculations.

[0012] In some embodiments, obtaining the overlapping regions between each projected polygon region and the images collected by each camera includes: using the image connected component method, grid judgment method or pairwise line intersection method to calculate the projected polygon regions and the images collected by each camera to obtain each overlapping region.

[0013] Some embodiments of the present application calculate the overlapping area between the projected polygon area and the image captured by the camera through different methods, which is convenient to calculate and has a relatively low complexity.

[0014] In some embodiments, the step of back-projecting and mapping each of the overlapping areas into the world calibration coordinate system to obtain each mapped polygon area includes: using the back-projection transformation equation to calculate each of the overlapping areas to obtain each of the mapped polygon areas in the world calibration coordinate system.

[0015] Some embodiments of the present application realize the mapping of the overlapping area in the camera image plane to the world calibration coordinate system through the back-projection transformation equation, which is both efficient and accurate.

[0016] In some embodiments, the step of performing a fusion calculation on each of the mapped polygon areas to obtain an overlapping activity area, where the overlapping activity area is used to provide a standing position reference for the motion capture actor, includes: dividing all the cameras in the multi-camera stereo vision system into multiple camera combinations; solving the intersection of each of the mapped polygon areas of each camera in each camera combination among the multiple camera combinations to obtain multiple target polygons; solving the union of the multiple target polygons to obtain the overlapping activity area.

[0017] Some embodiments of the present application divide all the cameras into groups to obtain multiple target polygons, and finally perform a union calculation to determine the overlapping activity area, with a low calculation complexity, improving the detection efficiency of the overlapping activity area, and providing an accurate activity area reference guide for the person being filmed.

[0018] In some embodiments, when the vertical distance between the planar polygon area and the ground plane is h, and h is a positive integer, the step of performing a fusion calculation on each of the mapped polygon areas to obtain an overlapping activity area includes: performing a fusion calculation on each of the mapped polygon areas to obtain an overlapping polygon area at a height of h; vertically projecting the overlapping polygon area onto the ground plane to obtain a projected polygon; solving the intersection of the projected polygon and the ground overlapping area to obtain a target polygon; using the target polygon as a cross-section and a three-dimensional columnar area with a vertical height of h as the three-dimensional overlapping activity area.

[0019] Some embodiments of the present application can realize the determination of the overlapping activity area in the world calibration coordinate system at a height of h from the ground plane, with relatively high flexibility.

[0020] In a second aspect, some embodiments of the present application provide a device for detecting overlapping activity areas, including: a projection module configured to project a planar polygon area parallel to the ground plane in a world calibration coordinate system onto images acquired by each camera in a multi-camera stereo vision system to obtain respective projected polygon areas; an acquisition module configured to acquire respective overlapping areas between the respective projected polygon areas and the images acquired by each camera; a mapping module configured to back-project and map the respective overlapping areas into the world calibration coordinate system to obtain respective mapped polygon areas; and a fusion module configured to perform a fusion calculation on the respective mapped polygon areas to obtain an overlapping activity area, where the overlapping activity area is used to provide a standing position reference for a motion capture actor.

[0021] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, where the program, when executed by a processor, can implement the method described in any one of the embodiments of the first aspect.

[0022] In a fourth aspect, some embodiments of the present application provide an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor, when executing the program, can implement the method described in any one of the embodiments of the first aspect.

[0023] In a fifth aspect, some embodiments of the present application provide a computer program product, where the computer program product includes a computer program, and where the computer program, when executed by a processor, can implement the method described in any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To more clearly illustrate the technical solutions of some embodiments of the present application, the following briefly introduces the drawings required for use in some embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 A schematic diagram of a motion capture scene provided by some embodiments of the present application;

[0026] Figure 2 One of the flowcharts of the method for detecting overlapping activity areas provided by some embodiments of the present application;

[0027] Figure 3 A schematic diagram of the projection of a rectangular ground grid onto the imaging of a camera view provided by some embodiments of the present application;

[0028] Figure 4Flowchart II of the method for detecting overlapping activity areas provided in some embodiments of the present application;

[0029] Figure 5 Schematic diagram of the intersection or union of the ground visible areas provided in some embodiments of the present application;

[0030] Figure 6 Schematic diagram I of the ground visible area corresponding to the camera provided in some embodiments of the present application;

[0031] Figure 7 Schematic diagram II of the ground visible area corresponding to the camera provided in some embodiments of the present application;

[0032] Figure 8 Block diagram of the device for detecting overlapping activity areas provided in some embodiments of the present application;

[0033] Figure 9 Schematic diagram of an electronic device provided in some embodiments of the present application. Detailed implementation manners

[0034] Next, the technical solutions in some embodiments of the present application will be described with reference to the accompanying drawings in some embodiments of the present application.

[0035] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0036] In the related art, in the field of optical human motion capture, since the field of view that each camera can capture is limited, the motion capture actor has to focus on his own actions and cannot pay attention to whether his own motion capture image is captured in the capture screen of each camera. At the same time, from the requirement of continuous and accurate action acquisition, the motion capture actor needs to place his own activity position range as much as possible within the area captured by at least two or more cameras. This requires a powerful feedback for indicating the position that is convenient for the action actor to perceive, that is, a ground effective activity area indication mark. A good activity area indication mark should be similar to the area described by a series of contour lines, and each contour line indicates how many cameras can simultaneously capture the internal area.

[0037] The spatial capture field of view of the camera can be regarded as a tetrahedron extending outward perpendicular to the camera plane from the optical center of the camera. The tetrahedron light beams associated with cameras in different directions meet in space, and their intersecting polyhedrons are the spatial areas that can be captured by different cameras. The detection of the overlapping areas on the ground must be completed by first calculating the intersecting polyhedrons and then projecting them onto the ground. The difficulty of spatial calculation becomes particularly huge and complex with each increase in the number of cameras. In real applications, people or objects are usually placed at different locations before the action is captured to observe how many cameras capture them at the same time and mark their positions respectively. Obviously, this method of detecting overlapping activity areas requires the collaboration of multiple people, which is costly and time-consuming, and the integrity of the detection is difficult to guarantee.

[0038] In view of this, some embodiments of the present application provide a method for detecting overlapping activity areas, which can define a world coordinate system on the ground, complete the connection between the world calibration coordinate system and the camera's image coordinate system through camera internal and external parameter calibration and ground calibration, and determine each overlapping area between the projected polygonal area and the image captured by the camera after projecting the planar polygonal area under the world calibration coordinate system to each camera in the multi-camera stereo vision system; then, each overlapping area is reversely projected and mapped to the world calibration coordinate system, and finally, all mapped polygonal areas are fused and calculated to obtain overlapping activity areas. Some embodiments of the present application can automatically detect overlapping activity areas through the ground calibration technology in the external parameters of the multi-camera stereo vision system, without the need for multiple people to collaborate and try back and forth multiple times, and can draw corresponding overlapping contour areas according to the requirements of overlapping different numbers of cameras. By utilizing the camera geometry projection transformation and introducing the ground calibration coordinate system, the difficulty of spatial 3D calculation is converted to the calculation of the overlapping area on the 2D plane, and then the overlapping area on the 2D plane is mapped back to the 3D space, so as to directly obtain the ground physical position area indicator curve (that is, the overlapping activity area). The entire solution achieves efficient, automatic and accurate results.

[0039] The following is combined with Figure 1 A schematic diagram of a motion capture scene of a multi-camera stereo vision system for overlapping activity area detection provided by some embodiments of the present application is exemplified.

[0040] like Figure 1 As shown, some embodiments of the present application provide a motion capture scene including multiple cameras in a multi-camera stereo vision system and a motion capture actor (or subject). Different numbers of cameras are set up around the motion capture site and pointed at the capture area of the site. The visible ground range of cameras at different positions is different, and the motion capture actor moves in the motion capture area. Different color labels can be attached to the ground to draw areas with different overlapping degrees according to the detection results of the overlapping activity area to indicate the motion capture actor to facilitate planning of his action position.

[0041] The following will combine the appended Figure 2 Exemplarily illustrate the implementation process of overlapping activity area detection provided by some embodiments of the present application. It should be noted that the implementation process of this overlapping activity area detection can be executed by a processor inside the multi-camera stereo vision system, or by a terminal device associated with the multi-camera stereo vision system (for example, a mobile terminal or a non-portable computer terminal), and the embodiments of the present application do not make specific limitations here.

[0042] Please refer to the appended Figure 2 , Figure 2 which is a flowchart of a method for detecting an overlapping activity area provided by some embodiments of the present application. The method for detecting the overlapping activity area may include:

[0043] S210, project a planar polygon area parallel to the ground plane in the world calibration coordinate system onto the images collected by each camera in the multi-camera stereo vision system to obtain each projected polygon area.

[0044] For example, in some embodiments of the present application, the terminal device can obtain the ground polygon area in the world calibration coordinates (as a specific example of the planar polygon area), and project the measured coordinates of the ground polygon area onto the images collected by each camera to obtain the projected polygon areas of each camera.

[0045] To achieve accurate projection of the polygon area, in some embodiments of the present application, a projection transformation equation needs to be constructed first. Specifically, the projection transformation equation is obtained through the following method: construct the world calibration coordinate system, where the horizontal plane of the world calibration coordinate system is parallel to the ground plane; obtain the calibration parameters of each camera relative to the world calibration coordinate system, where the calibration parameters include: the internal parameter matrix, the camera distortion coefficient, the rotation matrix, and the translation matrix; based on the world coordinates of the plane polygon area to be projected in the world calibration coordinate system and the calibration parameters, construct the projection transformation equation.

[0046] For example, in some embodiments of the present application, the horizontal plane of the world calibration coordinate system is the xoy plane (or xy plane) in the coordinate system, and O is the origin of the world coordinate system. Through ground calibration, the visible overlapping areas of different overlapping numbers of cameras for the ground standing position reference of the motion capture actor can be detected (as a specific example of the overlapping activity area). The task of ground calibration is to establish a ground calibration plane coordinate system (as a specific example of the world calibration coordinate system, abbreviated as the ground coordinate system or the world coordinate system), that is, to obtain the rotation matrix Rg and the translation matrix Tg from the ground coordinate system to the main image coordinate system in the multi-camera stereo vision system. The ground calibration coordinate system means that the origin of the world coordinate system is on the plane representing the ground (i.e., the vertical distance between the horizontal plane formed by xy of the world coordinate system and the ground is zero), and the plane formed by the x and y axes of the world coordinate system is parallel to the plane representing the ground. By taking out 3 image points representing the ground coordinates (such as the origin, the x-axis and y-axis points) and the measured lengths corresponding to the x-axis and y-axis on the image collected by the main camera, and adding an image point indicating the z-axis direction, the ground calibration can be completed through the P3P operation. In other scenarios, under the condition that the xy plane of the world coordinate system is parallel to the ground, the horizontal plane formed by xy of the world coordinate system can also be higher or lower than the ground, and the embodiments of the present application are not limited thereto.

[0047] Using the rotation matrix Rg and the translation matrix Tg obtained through the ground calibration in the above text, plus the external parameter matrix obtained in the camera stereo calibration, the rotation matrix Ri and the translation matrix Ti from the world coordinate system to the image coordinate system of each camera in the camera stereo vision system can be determined in sequence (as a specific example of the calibration parameters). In the ground coordinate system, the 3D coordinate representation of the motion capture site polygon can be easily obtained. Taking a rectangular motion capture site with length W and width H as an example, if the center of the motion capture site is selected as the origin and the x-axis and y-axis are parallel to the rectangular site respectively, the 3D coordinates of the four corners can be represented as (-W / 2, H / 2, 0), (W / 2, H / 2, 0), (W / 2, -H / 2, 0), (-W / 2, -H / 2, 0) respectively. According to the projection transformation equation from the physical space point to the camera pixel point obtained by the pinhole imaging principle, each corner point of the ground polygon can be projected onto the camera image plane, and the projection transformation equation is as follows:

[0048]

[0049] In the formula, (Xw, Yw, Zw) is the point in the world coordinate system, that is, the coordinates of the four corners of the rectangular site, (u, v) is the position of the corresponding image point of this point in the camera image, Si is the scale factor, and Ki is the internal parameter matrix of the camera. Through this projection transformation equation, the plane polygon area to be projected in the ground coordinate system can be conveniently converted into the image of the i-th camera. Wherein, the i-th camera is any one in the multi-camera stereo vision system. For exampleFigure 3 As shown, the rectangular ground grid in the physical space can be projected and transformed to calculate its position in the coordinate system of the image captured by the camera.

[0050] In an actual scenario, the field of view captured by each camera is limited, which causes not all of the projected polygon regions to be shown in the image captured by the camera, and only part of them can be imaged. Figure 3 The black rectangular frame in the figure schematically shows the maximum imaging field-of-view rectangular frame of the camera. The part outside this black frame is invisible under the corresponding camera.

[0051] In addition to constructing the ground coordinate system, it is also necessary to calibrate the camera. The Zhang Zhengyou calibration method can be used for the calibration method of the multi-camera stereo vision system. By sampling the calibration board at different positions or capturing the waving of the calibration rod, the internal and external parameters of each camera can be calculated. The internal parameters of the camera include the internal parameter matrix Ki composed of the focal length and the principal point position of the camera, the camera image distortion coefficient Di, and the rotation matrix R(j) and translation matrix T(j) from the main camera to the j-th remaining camera, which characterize the relative positions of the cameras. Further, through the above-constructed ground coordinate system, the relative positions of the cameras can be transformed from the main camera coordinate system to the ground coordinate system, and the rotation matrix Ri and translation matrix Ti from the world coordinate system to the image coordinate system of each camera can be obtained, so as to construct the projection transformation equation.

[0052] In some embodiments of the present application, S210 may include: obtaining the projection transformation equations between the world calibration coordinate system and each camera; and obtaining each of the projected polygon regions of the planar polygon region in the image coordinate systems of each camera by using the projection transformation equations.

[0053] For example, in some embodiments of the present application, through the rotation matrix Ri and translation matrix Ti from the world coordinate system to the i-th image coordinate system, combined with the above projection transformation equation, the position of the planar polygon region in the ground coordinate system can be projected onto the image captured by the i-th camera (i.e., substituting the relevant parameters into the projection transformation equation), thereby obtaining the projected polygon region in the image.

[0054] S220, obtaining each overlapping region between each of the projected polygon regions and the images captured by each camera.

[0055] For example, in some embodiments of the present application, the overlapping region between the projected polygon region in each camera and the captured image is calculated.

[0056] In some embodiments of the present application, S220 may include: calculating each of the overlapping regions by using the image connected component method, the grid judgment method, or the pairwise line intersection method for each of the projected polygon regions and the images captured by each camera.

[0057] S230, reverse-project and map each of the overlapping regions into the world calibration coordinate system to obtain each mapped polygon region.

[0058] For example, in some embodiments of the present application, reverse-projecting and mapping each of the overlapping regions in each camera into the ground coordinate system is equivalent to the inverse process of S210, thereby obtaining the mapped polygon regions in the ground coordinate system.

[0059] In some embodiments of the present application, S230 may include: using the reverse projection transformation equation to calculate each of the overlapping regions to obtain each of the mapped polygon regions in the world calibration coordinate system.

[0060] For example, in some embodiments of the present application, performing an inverse operation using the projection transformation equation (i.e., the reverse projection transformation equation) to calculate the position coordinates of the mapped polygon region of the overlapping region in the ground coordinate system. That is, given (u, v), solve for (Xw, Yw, Zw) in the ground coordinate system, where the Zw of each point coordinate of the mapped polygon region is the same, and thus the physical polygon within the visible range of each camera (as a specific example of the mapped polygon region obtained by reverse projection) can be obtained.

[0061] S240, perform a fusion calculation on each of the mapped polygon regions to obtain an overlapping activity region, where the overlapping activity region is used to provide a standing position reference for the motion capture actor, and the overlapping activity region is used to provide a standing position reference for the motion capture actor.

[0062] For example, in some embodiments of the present application, by fusing all the mapped polygon regions, an overlapping activity region under the multi-camera stereo vision system is obtained. For example, when fusing the physical polygons of the visible fields of each camera, mutual fusion can be performed according to the combination requirements of different numbers m (m≥2) to form overlapping activity regions under different specified numbers m.

[0063] In the action capture application scenario, the number of cameras constituting the multi-camera stereo vision system is usually more than four, and even up to twenty or thirty; the motion data of the action actor can be well collected when the number of camera overlaps is 4 or more. Therefore, we only need to calculate the regions with overlapping numbers of m = 2, m = 3, or m = 4 to meet the requirements in reality. Specifically, it can be selected according to the actual scenario, and the embodiments of the present application do not make specific limitations here.

[0064] In some embodiments of the present application, S240 may include:

[0065] S241, divide all the cameras in the multi-camera stereo vision system into multiple camera combinations.

[0066] For example, in some embodiments of the present application, all cameras are numbered, and n different combinations (as a specific example of multiple camera combinations) are formed according to m different overlapping quantities specified, that is, each combination contains m cameras.

[0067] S242. Solve the intersection of the respective mapping polygon regions of each camera in each of the multiple camera combinations to obtain a plurality of target polygons.

[0068] For example, in some embodiments of the present application, the intersection of all physical polygons of the visible fields corresponding to the m cameras in each combination is calculated to obtain a new polygon nPPi corresponding to each combination (as a specific example of the target polygon), that is, n new polygons nPPi can be obtained.

[0069] S243. Solve the union of the plurality of target polygons to obtain the overlapping activity region.

[0070] For example, in some embodiments of the present application, the result obtained by calculating the union of the n new polygons nPPi is used as the overlapping region polygon (as a specific example of the overlapping activity region).

[0071] In some other embodiments of the present application, when the vertical distance between the planar polygon region and the ground plane is h, and h is a positive integer, S240 may further include: performing a fusion calculation on the respective mapping polygon regions to obtain an overlapping polygon region at a height of h; vertically projecting the overlapping polygon region onto the ground plane to obtain a projected polygon; solving the intersection of the projected polygon and the ground overlapping region to obtain a target polygon; using the target polygon as a cross-section and a three-dimensional columnar region with a vertical height of h as the three-dimensional overlapping activity region.

[0072] For example, in some other embodiments, the method for obtaining the ground overlapping activity region described above can be applied to a spatial plane perpendicular to the ground at a height of h (h is a positive integer), to obtain an overlapping polygon region that can be captured by multiple cameras on this spatial plane, and then project the overlapping polygon region at a height of h back onto the ground, and solve the intersection of the polygon of the overlapping region with the ground. The intersection polygon (i.e., the target polygon) is used as the bottom surface, and the overlapping polygon region with a height of h is used as the top surface to form a three-dimensional columnar region. This three-dimensional columnar region can cover people and objects at a height of h from the ground, so that people and objects can all appear completely within the visible range of m cameras with a specified overlapping quantity. It can be understood that the polygon of the ground overlapping region is obtained by solving the intersection of the polygons projected by each camera onto the ground.

[0073] For example, uniformly taking the world coordinate system established based on the ground as the standard. Therefore, when finding the visible overlapping polygons for a plane at a height h from the ground, the 3D coordinate points of the four corners of the rectangular site mentioned above can be simply set as (-W / 2, H / 2, h), (W / 2, H / 2, h), (W / 2, -H / 2, h), (-W / 2, -H / 2, h). Then, following the aforementioned processing flow, project them into the camera image, obtain the overlapping polygons in the image, and then project them back to the world coordinate system in reverse to find the overlapping polygons in the physical world at this moment.

[0074] In some embodiments, the final overlapping activity area can also be projected into the cameras in the multi-camera stereo vision system, so that the overlapping activity area can be visually seen from the images collected by the cameras, and the images captured by the cameras can be used to guide the ground to paste and mark the corresponding area.

[0075] The following combines the attached Figure 4 Exemplarily illustrate the specific process of detecting the overlapping activity area provided by some embodiments of the present application.

[0076] Please refer to the attached Figure 4 , Figure 4 which is a flowchart of a method for detecting an overlapping activity area provided by some embodiments of the present application.

[0077] The following exemplarily illustrates the above process.

[0078] S410, calculate the internal and external parameters of each camera in the multi-camera stereo vision system relative to the ground coordinate system, obtain the calibration parameters, and construct a projection transformation equation based on this.

[0079] Among them, the calibration parameters include: the internal parameter matrix of each camera, the distortion coefficient, and the rotation matrix and translation matrix between each image coordinate system and the ground coordinate system.

[0080] S420, use the projection transformation equation to obtain the respective projected polygon regions of the planar polygon region in the ground coordinate system in the image coordinate systems of each camera.

[0081] For example, the above projection transformation equation describes the correspondence relationship between points on the ground plane and the camera image plane, and it belongs to the homography transformation corresponding to two planar point sets. Therefore, the polygon region characterized by the physical size in the ground coordinate system can be obtained as the projected pixel polygon region in the image coordinate system of each camera according to the projection transformation equation.

[0082] S430, obtain the respective overlapping regions of the respective projected polygon regions and the images collected by each camera.

[0083] For example, if the outer frame of the image captured by the camera is regarded as a polygon, by solving the overlapping part of two polygon frames, a new polygon can be obtained, that is, the imaging polygon of the ground in the i-th camera (as a specific example of the overlapping area).

[0084] Specifically, there are various ways to obtain the overlapping polygon or the union polygon of two planar polygon frames (i.e., the image captured by a camera and the projected polygon area), such as the image connected component method, the grid judgment method, and the line intersection method.

[0085] The image connected component method is to fill the polygon with gray values, stack two images together, obtain the contour line of the target connected area according to the requirements of intersection and union, and then vectorize the contour line into a polygon to obtain the overlapping area.

[0086] The grid method converts the gray values filled in the polygon into a discrete point set with uniform internal values to judge intersection or union to obtain a new target point set, and then converts the target point set into a polygon to obtain the overlapping area.

[0087] The line intersection method finds the intersection points or endpoints according to the positional relationship between the line segments forming the polygon and another polygon, obtains the corner points of the target area according to the requirements of intersection and union, and then connects these corner points in sequence to complete the calculation of the overlapping area. Specifically, it can be done through the following steps:

[0088] 1) Take out a side line segment in polygon Pi and judge its positional relationship with another polygon Pj; if the line segment is inside or on the other polygon frame, record the 2 endpoints where the overlapping part of the line segment starts; if there is 1 intersection point, record the intersection point and the point inside the frame; if there are 2 intersection points, record the 2 intersection points.

[0089] 2) Take out the next side of polygon Pi in clockwise order;

[0090] Repeat the above operations until all sides in polygon Pi have been processed.

[0091] 3) Connect the intersection points in the above steps in clockwise order to obtain the new polygon area Pij generated after the overlapping of the two polygons.

[0092] S440, reverse-project and map each overlapping area into the ground physical coordinate system to obtain each mapped polygon area.

[0093] For example, reverse-project and map the overlapping area obtained above from the camera image plane into the ground coordinate system to obtain the physical size polygon area visible to each camera.

[0094] S450, perform fusion calculation on each mapped polygon area to obtain the overlapping activity area.

[0095] For example, according to the requirement of the number of camera overlaps, such as 2 or more, possible camera combinations can be enumerated. In a multi-camera stereo vision system, it is required that at least 2 cameras simultaneously capture the object to be measured so that the distance and size of the object in the 3D space can be deduced through the triangulation formula.

[0096] Taking a multi-camera stereo vision system composed of 4 cameras (numbered 1, 2, 3, and 4 respectively) as an example, according to the requirement of at least 2 cameras overlapping (m = 2), the number of combinations is 6, that is, there are 6 possible camera overlap combinations, namely (1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4); find the intersection of the ground visible polygons (i.e., the mapped polygon regions) corresponding to the respective cameras in each combination to obtain the overlapping polygon region Pij = Pi ∩ Pj for each combination; find the union of all the Pij obtained in all combinations to get the final ground overlapping polygon covered by at least 2 cameras; draw the boundary line at the corresponding position on the ground along the edges of the ground overlapping polygon, which is the safe and effective overlapping activity area.

[0097] As Figure 5 shown, a schematic diagram of finding the intersection or union of the visible areas (i.e., the mapped polygon regions) on the ground of a multi-camera stereo vision system composed of two cameras is given. Figure 5 In it, A, B, C, and D are the 4 corners of the rectangular ground site, and the polygon AEFGH is Figure 6 the ground visible area (i.e., the mapped polygon region) corresponding to the camera, and the polygon IBLKJ is Figure 7 the ground visible area corresponding to the camera. The triangle OXY represents the world coordinate system established on the ground through the checkerboard calibration board, O is the origin, OX represents the X-axis, and OY is the Y-axis. The polygon IEFMJ is the overlapping polygon where the fields of view of the two cameras overlap, and the polygon ABLKMGH is the polygon of the union of the field-of-view polygons of the two cameras.

[0098] It can be understood that the specific implementation process of S410 to S450 can refer to the method embodiments provided above. To avoid repetition, appropriate descriptions are omitted here.

[0099] Through some of the embodiments provided by the present application, it can be seen that in the original multi-camera stereo vision system calibration process, this solution does not add new human and material resources, but only uses the calibration results to perform geometric calculations to achieve the technical effect of accurately calculating the overlapping activity area. The innovation of this application lies in the introduction of ground plane calibration, which converts the originally complex 3D space overlap calculation problem into a 2D plane, and calculates the corresponding overlapping area according to different overlap numbers in the overlapping area calculation, thereby forming different degrees of preferred overlapping activity areas for reference of motion capture actors. This application avoids the original complex 3D space operation, and also has the characteristics of high efficiency, automaticity and convenience. It can give the subject a clear instruction feedback when capturing the picture, and promptly guide the subject to converge the activity range within the effective overlapping activity area.

[0100] Please refer to Figure 8 , Figure 8 The block diagram of the apparatus for detecting overlapping activity areas provided by some embodiments of the present application is shown. It should be understood that the apparatus for detecting overlapping activity areas corresponds to the above method embodiment and can perform each step involved in the above method embodiment. The specific functions of the apparatus for detecting overlapping activity areas can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.

[0101] Figure 8 The device for overlapping activity area detection includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the device for overlapping activity area detection, and the device for overlapping activity area detection includes: a projection module 710, which is used to project a planar polygonal area parallel to the ground plane in a world calibration coordinate system to an image captured by each camera in a multi-camera stereo vision system to obtain each projection polygonal area; an acquisition module 720, which is used to obtain each overlapping area of the image captured by each camera and each projection polygonal area; a mapping module 730, which is used to reversely project and map the each overlapping area to the world calibration coordinate system to obtain each mapped polygonal area; a fusion module 740, which is used to perform fusion calculation on the each mapped polygonal area to obtain an overlapping activity area, wherein the overlapping activity area is used to provide a position reference for the motion capture actor.

[0102] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0103] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the operations of the method corresponding to any of the above methods provided in the above embodiments.

[0104] Some embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the operations corresponding to any of the methods provided in the above embodiments.

[0105] As Figure 9 As shown, some embodiments of the present application provide an electronic device 800, which includes: a memory 810, a processor 820, and a computer program stored in the memory 810 and executable on the processor 820. When the processor 820 reads the program from the memory 810 through a bus 830 and executes the program, it can implement the method of any of the above embodiments.

[0106] The processor 820 can process digital signals and can include various computing architectures. For example, a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, the processor 820 can be a microprocessor.

[0107] The memory 810 can be used to store instructions executed by the processor 820 or data related to the execution of the instructions. These instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 820 of the present disclosure embodiment can be used to execute the instructions in the memory 810 to implement the method shown above. The memory 810 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memories well known to those skilled in the art.

[0108] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0109] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0110] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

Claims

1. A method for detecting overlapping activity regions, characterized in that Including: Projecting a planar polygon region parallel to the ground plane in the world calibration coordinate system onto the images acquired by each camera in the multi-camera stereo vision system to obtain respective projected polygon regions; Obtaining respective overlapping regions between the respective projected polygon regions and the images acquired by each camera; Back-projecting and mapping the respective overlapping regions into the world calibration coordinate system to obtain respective mapped polygon regions; Performing a fusion calculation on the respective mapped polygon regions to obtain an overlapping activity region, where the overlapping activity region is used to provide a standing position reference for the motion capture actor.

2. The method according to claim 1, characterized in that, The step of projecting a planar polygon region parallel to the ground plane in the world calibration coordinate system onto the images acquired by each camera in the multi-camera stereo vision system to obtain respective projected polygon regions includes: Obtaining a projection transformation equation between the world calibration coordinate system and each camera; Using the projection transformation equation to obtain the respective projected polygon regions of the planar polygon region in the image coordinate systems of each camera.

3. The method according to claim 2, wherein The projection transformation equation is obtained by the following method: Constructing the world calibration coordinate system, where the horizontal plane of the world calibration coordinate system is parallel to the ground plane; Obtaining the calibration parameters of each camera relative to the world calibration coordinate system, where the calibration parameters include: an internal parameter matrix, a camera distortion coefficient, a rotation matrix, and a translation matrix; Based on the world coordinates of the planar polygon region to be projected in the world calibration coordinate system and the calibration parameters, constructing the projection transformation equation.

4. The method according to any one of claims 1-3, characterized in that, The step of obtaining respective overlapping regions between the respective projected polygon regions and the images acquired by each camera includes: Using an image connected component method, a grid judgment method, or a pairwise line intersection method to calculate the respective projected polygon regions and the images acquired by each camera to obtain the respective overlapping regions.

5. The method according to any one of claims 2-3, characterized in that, The step of back-projecting and mapping the respective overlapping regions into the world calibration coordinate system to obtain respective mapped polygon regions includes: using a back-projection transformation equation to calculate the respective mapped polygon regions in the world calibration coordinate system.

6. The method according to any one of claims 1 to 3, characterized in that The step of performing a fusion calculation on the respective mapped polygon regions to obtain an overlapping activity region includes: Dividing all the cameras in the multi-camera stereo vision system into multiple camera combinations; Solving the intersection of the respective mapped polygon regions of each camera in each camera combination among the multiple camera combinations to obtain multiple target polygons; Solving the union of the multiple target polygons to obtain the overlapping activity region.

7. The method according to any one of claims 1 to 3, characterized in that, When the vertical distance between the planar polygon region and the ground plane is h, and h is a positive integer, the step of performing a fusion calculation on the respective mapped polygon regions to obtain an overlapping activity region includes: Performing a fusion calculation on the respective mapped polygon regions to obtain an overlapping polygon region at height h; Vertically projecting the overlapping polygon region onto the ground plane to obtain a projected polygon; Solving the intersection of the projected polygon and the overlapping region of the ground to obtain a target polygon; Taking the target polygon as a cross-section and a three-dimensional columnar region with a vertical height of h as the three-dimensional overlapping activity region.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, wherein the computer program, when run by a processor, executes the method according to any one of claims 1-7.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program, when run by the processor, executes the method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program, wherein the computer program, when run by a processor, executes the method according to any one of claims 1-7.

Images