Method, device, medium, equipment and product for detecting overlapping activity area of camera images under multiple visual angles

Abstract

The invention relates to the technical field of visual inspection, and particularly provides a method, a device, a medium, equipment and a product for detecting a camera image overlapping activity area under multiple visual angles. Acquiring a plurality of overlapped image areas between the target camera and each camera in the other cameras; wherein the target camera is any one in the multi-camera stereoscopic vision system, and the other cameras are cameras, except the target camera, in the multi-camera stereoscopic vision system; and performing fusion calculation on the plurality of overlapped image areas to obtain an overlapped image overlapping activity area of the at least two cameras, the image overlapping activity area being used for feeding back to a photographed person to adjust position and attitude reference. According to some embodiments of the invention, the method can achieve the accurate detection of the overlapping activity region among multiple cameras in a man-machine interaction system, and is higher in practicality.

Description

Technical Field

[0001] The present application relates to the technical field of visual detection, and more particularly, to a method, apparatus, medium, device and product for detecting overlapping activity regions of camera images from multiple perspectives. Background Art

[0002] A multi-camera stereo vision system restores the 3D positional relationship in a scene through image acquisition from different perspectives of the same scene and triangulation calculation. Whether in the field of optical human motion capture or in many applications focusing on close-proximity human-computer interaction, the actor needs to place the effective moving parts in the area captured by more than 2 cameras simultaneously to better enter the calculation and deduction. In the field of human motion capture, to capture and restore the full-body motion of the motion capture actor, usually the motion capture site is large enough, and the motion capture actor consciously selects his own standing position and posture according to the effective activity area markers pre-drawn on the ground. In the case of close-proximity human-computer interaction applications, such as gesture interaction control, it is impossible to mark the effective activity area in the suspended air in advance. Only by showing the captured multiple pictures to the subject for him to adjust his position according to the real-time image picture, or the system calculates and gives a voice prompt for standing position adjustment according to the real-time picture. Whether it is real-time image feedback or voice prompt, it is necessary to know in advance the overlapping image area between multiple cameras and visually display it intuitively. At present, the overlapping area of two images is detected by feature point matching. Stable feature points and descriptors are found on two different images, such as SIFT feature points and SURF feature points, and then consistency matching is performed between the feature point sets obtained from different images to determine the association of the two images. However, this method can only handle well the planar images with simple translation, scaling and rotation transformation, and the processing effect on the imaging images of multi-perspective cameras is not good. Especially for a single empty background image, almost no feature points can be detected, and at the same time, the amount of feature point calculation is relatively large. Therefore, how to provide a technical solution for efficient detection of overlapping activity regions of camera images from multiple perspectives has become an urgent technical problem to be solved. Summary of the Invention

[0003] Some embodiments of the present application aim to provide a method, apparatus, medium, device and product for detecting overlapping activity regions of camera images from multiple perspectives. Through the technical solutions of the embodiments of the present application, automatic detection of overlapping activity regions in a multi-perspective stereo camera system can be achieved without manual participation, improving the detection accuracy and efficiency.

[0004] In a first aspect, some embodiments of the present application provide a method for detecting overlapping activity regions of camera images from multiple perspectives, including: when the calibration between a target camera and other cameras in a multi-camera stereo vision system is completed, obtaining a plurality of overlapping image regions between the target camera and each of the other cameras; wherein, the target camera is any one of the multi-camera stereo vision system, and the other cameras are the cameras in the multi-camera stereo vision system except the target camera; performing fusion calculation on the plurality of overlapping image regions to obtain an image overlapping activity region where at least two cameras overlap, and the image overlapping activity region is used to feed back to the subject for adjusting the position and posture reference.

[0005] In some embodiments of the present application, after the calibration between the target camera and each of the other cameras is completed, a plurality of overlapping image regions between the target camera and each camera are obtained, and finally, fusion calculation is performed on the plurality of overlapping image regions to obtain an image overlapping activity region between at least two cameras. By converting the 3D space calculation problem to a 2D plane and performing image transformation between cameras, the present application can detect the overlapping activity region of multiple cameras in a human-computer interaction system, and the practicability is relatively high.

[0006] In some embodiments, the obtaining a plurality of overlapping image regions between the target camera and each of the other cameras includes: obtaining a homography transformation matrix between the target camera and each camera; using the homography transformation matrix to transform the image outer frame on each camera into the image frame of the target camera to obtain a mapping region of each camera; solving an overlapping image region where the mapping region of each camera and the image frame overlap, and the overlapping image region is any one of the plurality of overlapping image regions.

[0007] In some embodiments of the present application, through the homography transformation matrix between the target camera and each camera, the image outer frame on each camera is transformed into the target camera to obtain a mapping region, and then the overlapping image region between the mapping region and the image frame is solved, providing effective data support for the accurate detection of the subsequent overlapping activity region.

[0008] In some embodiments, the solving an overlapping image region where the mapping region of each camera and the image frame overlap includes: using an image connected component method or a line intersection method to calculate the mapping region and the image frame to obtain the overlapping image region.

[0009] In some embodiments of the present application, the overlapping image region is calculated by different methods, and the calculation method is simple and the flexibility is relatively high.

[0010] In some embodiments, calculating the overlapping image regions to obtain the image overlapping active region between at least two cameras includes: performing a union calculation on the multiple overlapping image regions to obtain the image overlapping active region between the at least two cameras.

[0011] In some embodiments of the present application, the overlapping active region is obtained by taking the union of the overlapping image regions, and the detection method is simple and easy to operate.

[0012] In some embodiments, calculating the multiple overlapping image regions to obtain the image overlapping active region between at least two cameras includes: performing an intersection calculation on the multiple overlapping image regions to obtain the image overlapping active region where n cameras in the multi-camera stereo vision system overlap simultaneously, where n is a positive integer greater than 2.

[0013] In some embodiments of the present application, the overlapping active region under multiple cameras is obtained by taking the intersection of the overlapping image regions, and the detection method is simple and easy to operate.

[0014] In some embodiments, the homography transformation matrix is obtained by the following method: jointly calibrating each of the target camera and the other cameras with a calibration board to determine the homography transformation matrix between the target camera and each camera, where the homography transformation matrix represents the mapping relationship of the pixel position coordinates of the calibration board image between cameras.

[0015] In some embodiments of the present application, the cameras are jointly calibrated with a calibration board to determine the homography transformation matrix, providing a basis for image transformation between cameras, which is both efficient and accurate.

[0016] In a second aspect, some embodiments of the present application provide a device for detecting the image overlapping active region of cameras from multiple perspectives, including: an overlapping module, configured to obtain multiple overlapping image regions between the target camera and each of the other cameras in a multi-camera stereo vision system when the calibration between the target camera and the other cameras is completed; where the target camera is any one of the multi-camera stereo vision system, and the other cameras are the cameras in the multi-camera stereo vision system except the target camera; a fusion module, configured to perform a fusion calculation on the multiple overlapping image regions to obtain the image overlapping active region where at least two cameras overlap, where the image overlapping active region is used to feed back to the person being photographed for adjusting the position and posture reference.

[0017] In a third aspect, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment of the first aspect can be implemented.

[0018] Fourth aspect, some embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, when the processor executes the program, the method described in any embodiment of the first aspect can be implemented.

[0019] Fifth aspect, some embodiments of the present application provide a computer program product, the computer program product includes a computer program, wherein, when the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented. Description of the Drawings

[0020] To more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings required to be used in some embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application, and therefore should not be regarded as a limitation of the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0021] Figure 1 One of the method flowcharts for detecting the overlapping activity area of camera images from multiple perspectives provided by some embodiments of the present application;

[0022] Figure 2 One of the example diagrams of the overlapping image area calculation process provided by some embodiments of the present application;

[0023] Figure 3 Another example diagram of the overlapping image area calculation process provided by some embodiments of the present application;

[0024] Figure 4 Another example diagram of the overlapping image area calculation process provided by some embodiments of the present application;

[0025] Figure 5 Another example diagram of the overlapping image area calculation process provided by some embodiments of the present application;

[0026] Figure 6 Another example diagram of the overlapping image area calculation process provided by some embodiments of the present application;

[0027] Figure 7 Schematic diagram of the outer frame of the image of camera b provided by some embodiments of the present application;

[0028] Figure 8 Schematic diagram of the overlapping image area calculation of camera a and camera b provided by some embodiments of the present application;

[0029] Figure 9Detection result diagram of the image overlapping activity area of camera a and camera b provided in some embodiments of the present application;

[0030] Figure 10 One of the schematic diagrams of the image overlapping activity area provided in some embodiments of the present application;

[0031] Figure 11 Another schematic diagram of the image overlapping activity area provided in some embodiments of the present application;

[0032] Figure 12 The second method flowchart for detecting the image overlapping activity area of cameras from multiple perspectives provided in some embodiments of the present application;

[0033] Figure 13 The block diagram of the device for detecting the image overlapping activity area of cameras from multiple perspectives provided in some embodiments of the present application;

[0034] Figure 14 A schematic diagram of an electronic device provided in some embodiments of the present application. Detailed implementation manners

[0035] Next, the technical solutions in some embodiments of the present application will be described in conjunction with the accompanying drawings in some embodiments of the present application.

[0036] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0037] Generally, the ways to characterize the overlapping space of a multi - perspective stereo system to guide the subject to adjust the standing position and posture can be divided into: setting visual markers in the physical space to indicate the limited activity overlapping area of people and objects, which is more applied in the field of human motion capture; or highlighting the boundary of the overlapping activity area on the multi - perspective images captured by cameras as a reference, and adjusting the activity through real - time interaction feedback between people and images. The present invention focuses on the latter, which is especially suitable for gesture interaction scenarios.

[0038] Theoretically, the spatial field of view of a camera can be regarded as a four - sided pyramid extending perpendicularly outward from the camera optical center to the camera plane. The four - sided pyramid beams associated with cameras in different orientations meet in space, and the intersecting polyhedron is the spatial area that can be captured by different cameras. It is relatively difficult to highlight the activity range of the overlapping area on the multi - angle real - time screen by first calculating the polyhedron intersecting in 3D space and then projecting the polyhedron onto each camera plane.

[0039] In view of this, the present invention provides a method and device for automatically calculating the overlapping camera images by using the detection information of the calibration plate images used in stereo calibration. The core idea is to transform the 3D space calculation problem into a 2D plane. Specifically, it uses the characteristics of the homography transformation in multi-view geometry, and calculates the homography transformation matrix through the paired point information provided by the detection patterns of the calibration plates of two cameras. At this time, the camera images are no longer regarded as the projections of the real 3D space, but as the infinite extension of the calibration plate plane in the 3D space. The extended calibration plate image in one camera is transformed into the corresponding camera image of another camera through the homography matrix, and the overlapping area is obtained with the corresponding camera image to realize the projection of the 3D overlapping space on the camera plane.

[0040] The following will Figure 1 exemplarily illustrate the implementation process of detecting the overlapping active area of camera images from multiple perspectives provided by some embodiments of the present application. It should be noted that the implementation process of detecting the overlapping active area of camera images from multiple perspectives can be executed by a processor inside the multi-camera stereo vision system, or by a terminal device associated with the multi-camera stereo vision system (for example, a mobile terminal or a non-portable computer terminal), and the embodiments of the present application do not specifically limit this here.

[0041] Please refer to the Figure 1 , Figure 1 which is a flowchart of a method for detecting the overlapping active area of camera images from multiple perspectives provided by some embodiments of the present application. Before detecting the overlapping active area of camera images from multiple perspectives, it is first necessary to perform joint calibration between the cameras. Therefore, the following will first introduce the method of joint calibration between the cameras and the process of obtaining the homography transformation matrix. Before performing the calibration, it is also necessary to select any one camera from the multi-camera stereo vision system as the target camera, and perform joint calibration on the target camera and each of the other cameras. It should be noted that the multi-camera stereo vision system can be a binocular camera, a three-camera or a four-camera, etc., and the embodiments of the present application do not specifically limit the number of cameras here.

[0042] In some embodiments of the present application, the homography transformation matrix between the target camera and each camera is obtained by the following method: respectively performing joint calibration on the target camera and each of the other cameras through the calibration plate, and determining the homography transformation matrix between the target camera and each camera, where the homography transformation matrix represents the mapping relationship of the pixel position coordinates of the calibration plate images between the cameras.

[0043] For example, in some embodiments of the present application, a planar calibration board chain calibration method is adopted to calibrate and calculate the internal parameters of each camera in a multi-camera stereo vision system and the external parameters between the target camera and each camera. Taking the joint calibration of the target camera and any one camera as an example, when using the Zhang-Zhengyou calibration method based on planar calibration board detection for camera joint calibration, camera a (as a specific example of the target camera) and camera b (as a specific example of any one camera) simultaneously sample images of the checkerboard calibration board (which can also be an asymmetric circle calibration board or a QR code checkerboard calibration board), and detect the pixel coordinates of the calibration points on the calibration board. For each pair of calibration board images, if more than 4 pairs of matching points can be detected, the 3x3 homography transformation matrix H between this pair of calibration board images can be obtained by solving the system of equations or calling the OpenCV function cv::findHomography. The homography matrix (i.e., the homography transformation matrix) here is as follows:

[0044]

[0045] This homography transformation matrix H describes the position mapping relationship (i.e., the image pixel position coordinate mapping relationship) of the corner point coordinates of the planar calibration board pattern between the pixel coordinate systems in the images of two different cameras, and is also called the perspective transformation matrix. The pixel point (u b , v b ) in camera b in the above formula can be transformed into camera a through the H matrix, and the corresponding pixel point coordinates in the image of camera a are (u a , v a ).

[0046] It can be understood that in the embodiments of the present application, the entire image captured by camera b is regarded as the imaging image after the spatial infinite extension of the calibration board plane. Therefore, the 4 vertices of the outer frame of the camera b image can be perspectively transformed according to the above formula to obtain the corresponding new positions in camera a, which means mapping the field of view of camera b into camera a. However, in some special cases, this homography perspective transformation H will produce distortion, and the 4 transformed points cannot be simply connected in sequence to obtain the field of view polygon of camera b in camera a. A specific processing (for example, discarding the area formed by abnormal points) is required to obtain the overlapping polygon of the visible fields of camera b and camera a, and this overlapping polygon means the common area that camera a and b can capture in their respective fields of view.

[0047] Taking the binocular stereo vision system as an example, Figure 2 and Figure 3 respectively show the example images of the synchronous images obtained by the binocular cameras for the same scene. The green shadow is the area that cannot be captured simultaneously by the two cameras (such as camera a and camera b) finally calculated by the present application. Figure 4Exemplarily, the case where the image of camera b is transformed to the image coordinates in camera a through the homography matrix H is given; a remarkable feature is that Figure 3 the checkerboard calibration pattern in Figure 2 is close to the position of the checkerboard calibration pattern in Figure 5 After the image of camera a is transformed to the image size of camera b through the inverse transformation of the homography transformation H, it is close to the Figure 3 checkerboard pattern in

[0048] When the angular relationship between the two cameras is large, or the inclination angle of the checkerboard pattern in space is relatively large, the perspective transformation at this moment will cause certain distortion, thus transforming the outer frame of the image of camera b into camera a to form one or two connected regions. Figure 6 An example of the image after the H transformation in this case is given. At this moment, the image is distorted, resulting in the transformation of the pattern in the lower left corner of the original image to the upper right corner. In this case, it is necessary to select the area where the checkerboard pattern is located as the overlapping image area, such as the polygon surrounded by the green line in the figure (that is, the area formed by discarding the points in the upper right corner).

[0049] The above process is described below by way of example.

[0050] In some embodiments of the present application, the method for detecting the overlapping activity area may include:

[0051] S110. When the calibration between the target camera and other cameras in the multi-camera stereo vision system is completed, obtain multiple overlapping image areas between the target camera and each of the other cameras.

[0052] For example, in some embodiments of the present application, after calibration, the overlapping situation between the image frame collected by the target camera and the outer frame of the image collected by each camera is determined to obtain multiple overlapping image areas. One overlapping image area can be obtained between the target camera and one other camera.

[0053] In some embodiments of the present application, S110 may include: obtaining the homography transformation matrix between the target camera and each camera; using the homography transformation matrix to transform the outer frame of the image on each camera into the image frame of the target camera to obtain the mapping area of each camera; and solving the overlapping image area where the mapping area of each camera overlaps with the image frame. Wherein the overlapping image area is any one of the multiple overlapping image areas.

[0054] For example, in some embodiments of the present application, after obtaining the homography transformation matrix H between the target camera and each camera through the above method, the coordinates of the four points of the image frame on each camera can be transformed into the target camera using H to obtain the mapping area of the image frame of each camera in the target camera. The overlapping area between the mapping area of each camera and the graphic frame is an overlapping image area.

[0055] In some embodiments of the present application, S110 may include: calculating the overlapping image area by using the image connected domain method or the line intersection method for the mapping area and the image frame.

[0056] For example, in some embodiments of the present application, the method for obtaining the visible overlapping area (as a specific example of the overlapping image area) after the transformation of camera b in the view of camera a can be through the image connection method and the line intersection method. The image connected domain method fills the entire image b of camera b with white, generates a new image (the white part in the target image is the mapping area) according to the size of camera a through image perspective transformation (i.e., homography matrix transformation), searches for the white connected areas in the new image, and simply selects the one with the largest area from the found multiple connected areas and converts it into a polygon to obtain the overlapping image area.

[0057] The line intersection method calculates the intersection points of the transformed sides of the image frame of camera b and the image frame of camera a to complete the calculation of the final overlapping area polygon (i.e., the overlapping image area). Specifically, it is carried out according to the following steps (taking the Figure 6 example of the result after the image transformation as an example):

[0058] 1) The four corner points of the image frame imgBoxB of camera b (as Figure 7 shown) are transformed into vTransBox in clockwise order through the homography matrix H. The 4 points shown in Figure 8 are bA, bB, bC, and bD; it is sequentially determined whether each point falls within the image frame imgBoxA of camera a. At the same time, the center point of the image frame of camera b is also transformed into camera a through H, which is the Figure 8 point pc within.

[0059] 2) The points in the transformed vTransBox are divided into 2 groups, named group 0 and group 1. The first point bA in vTransBox is classified into group 0, and the division criteria for the remaining points are determined according to steps 3)-5).

[0060] 3) Take the next point bB from vTransBox in sequence, form a line segment LAB with bA as the starting point and bB as the ending point, and determine whether the line segment LAB is on the right side of the point pc.

[0061] 4) If the LAB line segment is to the right of point pc, then the bB point is grouped into the group where the bA point is located. As long as there is one point of the line segment LAB not within the imgBoxA of camera a, the intersections of the line to which the line segment LAB belongs and the four surrounding borders of the imgBoxA of camera a need to be calculated, and the intersections are grouped into the group of the bA point.

[0062] 5) If the LAB line segment is to the left of point pc, the bB point is grouped into group 1. At the same time, all intersections of the line to which the line segment LAB belongs and the border of the imgBoxA of camera a are calculated. According to the positional relationship of each intersection between the line segments LAB, it is divided into the following 3 cases, and the intersections are grouped into different groups respectively.

[0063] Case 1: The intersection is outside bA (i.e., in the BA direction), and the intersection is grouped into the group where the bA point is located;

[0064] Case 2: The intersection is outside the bB point (i.e., in the AB direction), then it is grouped into the group where the bA point is not located;

[0065] Case 3: If the intersection is between LAB, the following 2 cases need to be considered for operation; that is: if the bA point is visible in the imgBoxA, and the line segment from bA to the intersection is to the right of point pc, then the intersection is grouped into the group where the bA point is located; if the bB point is visible in the imgBoxA, and the line segment from bB to the intersection is to the left of point pc, then the intersection is grouped into the group where the A point is not located.

[0066] 6) Successively take out the next point bC from vTransBox. The bC and bB points form a line segment (bB is the starting point and bC is the ending point), and repeat the above steps 3)-5) to complete the operation.

[0067] Repeat the point-taking operation in step 6) until the line segment formed by bD and bA is processed, and then enter step 7).

[0068] 7) Analyze each point P in groups 0 and 1: If the point P is visible within the image frame imgBoxA of camera a, it is inversely transformed to Pinv through the homography matrix H. If the point Pinv is not visible within the image frame imgBoxB of camera b, that is, invisible, then the point P is removed from the corresponding group. If Pinv is visible within the outer image frame imgBoxB of camera b, and the point P is on the four surrounding borders of the imgBoxA, it is recorded in the border point set F, and the corresponding group number of the point is also recorded.

[0069] 8) Map the four corner points of the image frame imgBoxA of camera a into the image of camera b. If the mapped points are visible in the imgBoxB, the corner point on the imgBoxA is also recorded in the border point set F, but at this time the corresponding group number of the point is -1, indicating uncertainty.

[0070] 9) Organize the points in the border point set F into an ordered point set in a clockwise direction.

[0071] 10) Traverse each point F(i) in the border point set F in a clockwise direction. If the group number corresponding to F(i) is not -1, but the combination corresponding to the next point F(i + 1) is -1, then determine whether the direction of the line segment formed by these two points is on the right side of the point pc; if it is on the right side, then classify the point F(i + 1) into the group corresponding to this group number, and at the same time change the group number corresponding to F(i + 1) to the group number of point F(i); if the direction of the line segment formed by these two points is on the left side of the point pc, then classify the point F(i + 1) into the group where point F(i) is not located, and at the same time also change the group number corresponding to F(i + 1).

[0072] 11) Calculate the convex hull polygons in the point sets of group numbers 0 and 1 respectively, and take the convex polygon where the point pc is located as the final overlapping region polygon, and its effect is equivalent to the convex polygon with the largest area.

[0073] Next, the implementation process of the above line intersection method is further elaborated with Figures 7 - 9 exemplary illustrations.

[0074] Figure 7 Denote the outer frame imgBoxB of the camera b image, which is obtained by transforming it through the homography matrix H calculated between the two images according to the image size of the camera a Figure 8 in the schematic diagram. Figure 7 The four corner points ABCD in Figure 8 correspond to bA, bB, bC, bD in Figure 7 ; the center point of Figure 8 corresponds to the pc position in

[0075] According to the above steps 3)-6), it can be successively determined that the pc point is on the right side of the line segments bA->bB and bB->bC. Therefore, bA, bB, and bC are directly classified into the set of group 0. Since the pc point is on the left side of the line segments bC->bD and bD->bA, bD is classified into the set of group 1. Calculate the intersection points of the lines of the line segments bC->bD and bD->bA with the border ImgBoxA to obtain the points j1, j2, j3, and j4. According to the positional relationship judgment in step 5), j2 and j3 can be classified into group 0; j1 and j4 are classified into group 1. According to step 8), the Figure 8 four corner points aA, aB, aC, and aD of imgBoxA in Figure 7As can be seen, the border F set {j1, j2, j3, j4, aB, aD} is obtained. After sorting the F set in the clockwise direction, {j1, aB, j4, j2, aD, j3} is obtained. Only the two points aB and aD in this set currently have no corresponding group number information. According to step 10), point j1 is found in F, and the corresponding group number is 1; the next point aB is classified into group 1 corresponding to point j1. Similarly, aD enters group 0 where j2 is located. Therefore, the final set of group 0 has {bA, bB, bC, j2, j3, aD}; the set of group 1 has {bD, j1, j4, aB}. Calculate the convex hull polygons of their respective sets according to step 11) to obtain Figure 9 The two polygons in. According to the position of the pc point, the polygon surrounded by j3 bA bB bC j2 aD can be obtained as the overlapping area polygon of the final camera b and camera a after imaging in camera a.

[0076] S120, perform fusion calculation on the multiple overlapping image regions, and obtain the image overlapping activity regions where at least two cameras overlap, wherein the image overlapping activity regions are used to feed back to the photographed person for adjusting the position and posture reference.

[0077] For example, in some embodiments of the present application, by performing intersection or union calculation on the multiple overlapping image regions obtained above, the image overlapping activity regions that can be covered by different numbers of cameras can be obtained.

[0078] In some embodiments of the present application, S120 may include: performing union calculation on the multiple overlapping image regions to obtain the image overlapping activity regions between the at least two cameras.

[0079] For example, in some embodiments of the present application, for the case where there are only two cameras in the binocular stereo vision system, the union of the overlapping image regions of the two is the largest overlapping polygon between the two cameras. If there are multiple cameras in the multi-camera stereo vision system, such as three cameras being camera a, camera b, and camera c, where camera a is the target camera, the overlapping image region Rab between camera a and camera b and the overlapping image region Rac between camera a and camera c can be obtained through the above operations. Taking the union of Rab and Rac can obtain the image overlapping activity regions including at least two cameras.

[0080] In other embodiments of the present application, S120 may include: performing intersection calculation on the multiple overlapping image regions to obtain the image overlapping activity regions where n cameras in the multi-camera stereo vision system overlap simultaneously, where n is a positive integer greater than 2.

[0081] For example, in some embodiments of the present application, for the case where there are only two cameras in a binocular stereo vision system, the intersection of the overlapping image regions of the two cameras is the overlapping activity region that can be captured simultaneously between the two cameras. If a multi-camera stereo vision system contains multiple cameras, such as 3 cameras (i.e., n = 3), namely camera a, camera b, and camera c, where camera a is the target camera, through the above operations, the overlapping image region Rab between camera a and camera b, and the overlapping image region Rac between camera a and camera c can be obtained. Taking the intersection of Rab and Rac can obtain the image overlapping activity region that can be captured simultaneously by the 3 cameras. The number of cameras can be set according to the actual application scenario, and the principle of determining the overlapping activity region is the same as above, which will not be elaborated in this embodiment of the present application.

[0082] As can be seen from the above embodiments of the present application, the present application completes the joint external parameter calibration between cameras through multiple pairs of images obtained after the cameras collect calibration board images, and one pair of calibration board images corresponds to one homography matrix. Through one homography matrix, the overlapping image region between two cameras can be calculated. For M pairs of calibration board images, M overlapping image regions can be obtained. Only by taking the union of the M overlapping regions jointly can the largest overlapping activity region between the two cameras be obtained; by taking the intersection of the M overlapping regions jointly, the smallest image overlapping activity region between the two cameras can be obtained. As Figure 10 and Figure 11 shown, the overlapping image region calculated through the calibration board image (i.e., the gray polygon region in the figure) and the finally merged largest overlapping activity region (i.e., the green polygon in the figure), superimposing this overlapping region on the real-time screen of the camera has the effect as Figure 2 and Figure 3 shown. Only one camera can illuminate the green shaded area, so the subject should place their actions in the non-shaded area to be effectively captured by the multi-camera stereo vision system to perform the expected operations.

[0083] The following will exemplarily elaborate on the specific process of detecting the overlapping activity region provided by some embodiments of the present application in conjunction with the attached Figure 12 drawings.

[0084] Please refer to the attached Figure 12 , Figure 12 which is a flowchart of a method for detecting an overlapping activity region provided by some embodiments of the present application.

[0085] The following takes the case where there are three cameras in a multi-camera stereo vision system as an example to exemplarily elaborate on the above process.

[0086] S210, the first camera a, the second camera b, and the third camera c respectively collect images of the calibration board to obtain two pairs of calibration images.

[0087] Among them, the first camera is equivalent to camera a in the above text, the second camera is equivalent to camera b in the above text, the third camera is equivalent to camera c in the above text, and camera a is the target camera. A pair of calibration images can be obtained between the first camera a and the second camera b, and another pair of calibration images can be obtained between the first camera a and the third camera c.

[0088] S220. Based on the two pairs of calibration images, calculate the homography transformation matrix between the first camera a and the second camera b, and the homography transformation matrix between the first camera a and the third camera c.

[0089] S230. Using the homography transformation matrix, transform the image frames of the second camera b and the third camera c to the image frame of the first camera a respectively to obtain the first mapping area and the second mapping area.

[0090] S240. Obtain Rab by intersecting the first mapping area and the image frame of the first camera a, and obtain Rac by intersecting the second mapping area and the image frame of the first camera a.

[0091] S250. Solve the union of Rab and Rac to obtain the overlapping activity area containing at least two cameras.

[0092] S260. Solve the intersection of Rab and Rac to obtain the overlapping activity area that overlaps three cameras simultaneously.

[0093] It should be noted that the specific implementation process of S210 to S260 can refer to the method embodiments above. To avoid repetition, the detailed description is appropriately omitted here. In addition, in actual application scenarios, at least one of the steps of S250 and S260 can be selected and executed according to actual needs, and the embodiments of the present application do not make specific limitations here.

[0094] Please refer to Figure 13 , Figure 13 shows a block diagram of the composition of the device for detecting the overlapping activity area of camera images from multiple perspectives provided by some embodiments of the present application. It should be understood that the device for detecting the overlapping activity area of camera images from multiple perspectives corresponds to the above method embodiments and can execute each step involved in the above method embodiments. The specific functions of the device for detecting the overlapping activity area of camera images from multiple perspectives can be seen in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0095] Figure 13The device for detecting overlapping active areas of camera images under multi-viewing angles includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the device for detecting overlapping active areas of camera images under multi-viewing angles. The device for detecting overlapping active areas of camera images under multi-viewing angles includes: an overlapping module 1310, which is used to obtain multiple overlapping image areas between the target camera and each of the other cameras when the calibration of the target camera and other cameras in the multi-camera stereo vision system is completed; wherein the target camera is any one of the multi-camera stereo vision system, and the other cameras are cameras other than the target camera in the multi-camera stereo vision system; a fusion module 1320, which is used to fuse and calculate the multiple overlapping image areas to obtain an image overlapping active area where at least two cameras overlap, wherein the image overlapping active area is used to feed back to the subject for reference for adjusting the position and posture.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0097] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the operations of the method corresponding to any of the above methods provided in the above embodiments.

[0098] Some embodiments of the present application further provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any of the above methods provided in the above embodiments.

[0099] like Figure 14 As shown, some embodiments of the present application provide an electronic device 600, which includes: a memory 610, a processor 620, and a computer program stored in the memory 610 and executable on the processor 620, wherein the processor 620 can implement a method as described in any of the above embodiments when reading the program from the memory 610 through a bus 630 and executing the program.

[0100] Processor 620 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 620 can be a microprocessor.

[0101] The memory 610 can be used to store instructions executed by the processor 620 or data related to the instruction execution process. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 620 of the embodiments of the present disclosure can be used to execute the instructions in the memory 610 to implement the method shown above. The memory 610 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memories well known to those skilled in the art.

[0102] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0103] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0104] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

Claims

1. A method for detecting overlapping activity regions in camera images from multiple perspectives, characterized in that Including: When the calibration between the target camera and other cameras in a multi-camera stereo vision system is completed, obtaining a plurality of overlapping image regions between the target camera and each of the other cameras; wherein, the target camera is any one of the multi-camera stereo vision system, and the other cameras are the cameras in the multi-camera stereo vision system except the target camera; Performing fusion calculation on the plurality of overlapping image regions to obtain an image overlapping active region where at least two cameras overlap, wherein the image overlapping active region is used to feed back to the subject for adjusting the position and posture reference.

2. The method according to claim 1, wherein The obtaining a plurality of overlapping image regions between the target camera and each of the other cameras includes: Obtaining the homography transformation matrix between the target camera and each camera; Using the homography transformation matrix to transform the image outer frame on each camera into the image frame of the target camera to obtain the mapping region of each camera; Solving the overlapping image regions where the mapping region of each camera and the image frame overlap, wherein the overlapping image region is any one of the plurality of overlapping image regions.

3. The method according to claim 2, wherein The solving the overlapping image regions where the mapping region of each camera and the image frame overlap includes: calculating the mapping region and the image frame using the image connected component method or the line intersection method to obtain the overlapping image region.

4. The method according to any one of claims 1 to 3, characterized in that The performing fusion calculation on the plurality of overlapping image regions to obtain an image overlapping active region between at least two cameras includes: Performing union calculation on the plurality of overlapping image regions to obtain the image overlapping active region between at least two cameras.

5. The method according to any one of claims 1-3, characterized in that, The performing fusion calculation on the plurality of overlapping image regions to obtain an image overlapping active region between at least two cameras includes: Performing intersection calculation on the plurality of overlapping image regions to obtain the image overlapping active region where n cameras in the multi-camera stereo vision system overlap simultaneously, where n is a positive integer greater than 2.

6. The method according to any one of claims 2-3, characterized in that, The homography transformation matrix is obtained by the following method: Performing joint calibration on the target camera and each of the other cameras respectively through a calibration board to determine the homography transformation matrix between the target camera and each camera, wherein the homography transformation matrix represents the mapping relationship of the pixel position coordinates of the calibration board image between cameras.

7. An apparatus for detecting overlapping activity regions in camera images from multiple perspectives, characterized in that, Including: An overlapping module, configured to obtain a plurality of overlapping image regions between the target camera and each of the other cameras when the calibration between the target camera and other cameras in a multi-camera stereo vision system is completed; wherein, the target camera is any one of the multi-camera stereo vision system, and the other cameras are the cameras in the multi-camera stereo vision system except the target camera; A fusion module, configured to perform fusion calculation on the plurality of overlapping image regions to obtain an image overlapping active region where at least two cameras overlap, wherein the image overlapping active region is used to feed back to the subject for adjusting the position and posture reference.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, wherein the computer program, when run by a processor, executes the method according to any one of claims 1-6.

9. An electronic device, characterized in that, Comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein when the computer program is run by the processor, it executes the method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product comprises a computer program, wherein when the computer program is run by a processor, it executes the method according to any one of claims 1-6.

Images