Video desktop high-angle shot correction method, system and equipment and medium

Abstract

A video desktop high-angle shooting correction method comprises the following steps: S1, acquiring a first frame image shot by an RGBD camera, and calculating three-dimensional point cloud information of pixels in a region of interest in an RGB image in combination with an internal reference matrix of the RGBD camera; s2, on the basis of the three-dimensional point cloud information, filtering out point clouds outside a desktop, fitting out a desktop plane, and further obtaining a normal vector of the desktop plane and a distance between the RGBD camera and the mass center of the desktop; s3, calculating a perspective transformation matrix H, and performing perspective transformation on the image to obtain a desktop area after high-angle shooting correction; and S4, performing proportional amplification on a region corresponding to the desktop region in the RGB image to obtain a region of interest of a next frame. The method has the advantages of low cost, simplicity and convenience in operation, high precision and rapidness.

Description

Background Art

[0002] In recent years, with the rise of online video communication, online education, online meetings, online live broadcasts and other online video communication methods have become increasingly popular. However, the camera used for online video usually needs to face the person in front of the screen. Therefore, when people need to write and draw manually during communication, the desktop in the picture will appear slanted or reversed due to the camera's viewing angle, making communication very inconvenient.

[0003] Currently, the main solution to this problem is to use a tablet or other device to connect to the online conference software at the same time, use the tablet as an electronic writing board, and then project the electronic writing board generated in real time on the tablet onto the screen, thereby obtaining a calibrated writing board that is more in line with the human eye.

[0004] However, this solution has two main problems: 1. The solution is expensive. In addition to the conference camera, a special electronic writing board is required. 2. The operation is inconvenient. Users need to manually connect the electronic writing board and the video conferencing software. The complexity depends on the support of the conference software for the electronic writing board. For ordinary video communication users, they hope to bypass these two problems. It would be best if they can directly use the most convenient paper and pen to write and draw to assist communication, just like face-to-face communication.

[0005] In other separate image processing schemes, a camera is used to obtain an image of the object to be converted, such as A4 paper. This scheme is only feasible for specific objects (such as only A4 paper), and has high requirements on the relative relationship between the camera and the object to be converted. The A4 paper needs to fill the camera's field of view to a large extent, and the angle between the camera and the object to be converted needs to be close to 90 degrees.

[0006] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0007] To this end, the present invention uses an RGBD camera to obtain three-dimensional point cloud information including the desktop, and uses the region of interest provided by the previous frame image to narrow the range, and then fits the desktop plane, and obtains the perspective transformation matrix of the RGBD camera and the desktop, thereby obtaining a desktop image after overhead correction, which has the advantages of low cost, simple operation, high precision and fast speed.

[0008] In a first aspect, the present invention provides a method for correcting a video desktop overhead shot, characterized in that it comprises the following steps:

[0009] Step S1: obtaining a first frame image taken by an RGBD camera, and then calculating the three-dimensional point cloud information of pixels in a region of interest in the RGB image in combination with an internal parameter matrix of the RGBD camera; wherein the region of interest is obtained from a frame image preceding the first frame image;

[0010] Step S2: based on the three-dimensional point cloud information, filter out the point clouds outside the desktop, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop;

[0011] Step S3: Calculate the perspective transformation matrix H, perform perspective transformation on the image, and thus obtain the desktop area after overhead correction;

[0012] Step S4: proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the area of ​​interest of the next frame.

[0013] Optionally, the video desktop overhead shooting correction method is characterized in that step S2 comprises:

[0014] Step S21: filtering out pixel points whose depth values ​​are too large to obtain a core area;

[0015] Step S22: identifying a plane area in the core area and performing fitting to obtain a desktop plane;

[0016] Step S23: Calculate the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop.

[0017] Optionally, the video desktop overhead shooting correction method is characterized in that step S22 comprises:

[0018] Step S221: identifying a plane region in the core region; wherein the plane region includes a plurality of planes;

[0019] Step S222: obtaining a first plane with the largest area and a second plane connected to the largest number of the planes;

[0020] Step S223: if the first plane and the second plane are different planes, execute step S224; if the first plane and the second plane are the same plane, mark the first plane as a desktop plane;

[0021] Step S224: identify the target object on the second plane and calculate the area. If the target object exists and the area is greater than the first threshold, mark the second plane as the desktop plane; otherwise, mark the first plane as the desktop plane.

[0022] Optionally, the video desktop overhead shooting correction method is characterized in that step S3 comprises:

[0023] Step S31: Calculate the rotation matrix R and translation vector T required to convert the oblique viewing angle of the desktop into a top-down viewing angle;

[0024] Step S32: Based on the rotation matrix R and the translation vector T, calculate the perspective transformation matrix H.

[0025] Optionally, the video desktop overhead shooting correction method is characterized in that step S31 comprises:

[0026] Step S311: Based on the normal vector a of the desktop and the target normal vector b=(0,0,-1), obtain the rotation matrix R:

[0027]

[0028] Where I is the unit matrix, v = a × b, s = ||v||sin(α), α is the angle between a and b, c = a·b,

[0029]

[0030] Step S312: According to the formula T = -(R·P center_above ) to obtain the translation vector T; where R is the rotation matrix, P center_above =P center +h*a,P center is the coordinate of the center of mass of the desktop, and h is the overlooking height specified by the user.

[0031] Optionally, the video desktop overhead correction method is characterized in that in step S32, the perspective transformation matrix H is obtained according to the following formula:

[0032]

[0033] Among them, the first item C int is the camera internal parameter, and the r in the second term is the corresponding element in the 3x3 rotation matrix, T x 、T y 、T z are the corresponding elements in the translation vector T respectively.

[0034] Optionally, the video desktop overhead shooting correction method is characterized by further comprising:

[0035] Step S5: performing target recognition on a preset area on the desktop, and scaling the target object after removing the background area to obtain the best display effect of the target object.

[0036] In a second aspect, the present invention provides a video desktop overhead correction system, which is used to implement any of the above-mentioned video desktop overhead correction methods, and is characterized by comprising:

[0037] A point cloud module is used to obtain a first frame of image taken by an RGBD camera, and then calculate the three-dimensional point cloud information of pixels in the region of interest in the RGB image in combination with the internal parameter matrix of the RGBD camera; wherein the region of interest is obtained from a frame of image previous to the first frame of image;

[0038] A first calculation module is used to filter out point clouds outside the desktop based on the three-dimensional point cloud information, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop;

[0039] The second calculation module is used to calculate the perspective transformation matrix H and perform perspective transformation on the image to obtain the desktop area after overhead correction;

[0040] The region of interest module is used to proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the region of interest of the next frame.

[0041] In a third aspect, the present invention provides a video desktop overhead correction device, characterized in that it includes:

[0042] processor;

[0043] a memory storing executable instructions of the processor;

[0044] The processor is configured to execute any one of the steps of the above-mentioned video desktop overhead shooting correction method by executing the executable instructions.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned methods for correcting overhead video desktop shots are implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention obtains information including the desktop by using an RGBD camera, and then uses the three-dimensional point cloud information to identify and calculate the desktop. Compared with using a two-dimensional plane for calculation, the present invention can better utilize the three-dimensional space to segment the target object, and can use a larger range for transformation, thus avoiding distortion of the edge area of ​​the target object due to transformation, and has higher quality.

[0048] The present invention uses the desktop as the conversion object, does not rely on the recognition of the target object, can be applicable to the conversion of target objects of various shapes and sizes, and has very strong adaptability and versatility.

[0049] The present invention only requires an RGBD camera to achieve the conversion of the target object. Compared with relying on devices such as tablets, users can display various types of content more independently without being restricted by the size of hardware devices. At the same time, users can also use a variety of tools for display, which has higher freedom and convenience.

[0050] The present invention utilizes the region of interest provided by the previous frame to greatly reduce the number of pixels that need to be calculated, and utilizes the identified desktop to provide the region of interest to the next frame. By transferring the region of interest, it is possible to reduce the amount of calculation and increase the recognition conversion speed, and to adjust the region of interest for each frame, thereby achieving adaptive adjustment of the target object and fully displaying the target object.

[0051] The present invention can intelligently correct the squinting or reverse-viewing desktop into a top-down view that conforms to the human eye's senses with one click, facilitating video users to communicate more effectively, and can also solve the problems of high cost and inconvenient operation caused by external electronic writing boards in the current mainstream solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0053] Figure 1 A flowchart of a method for correcting a video desktop overhead shot in accordance with an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a region of interest in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a correction effect in an embodiment of the present invention;

[0056] Figure 4 A flowchart of steps for obtaining the distance between a normal vector and a center of mass in an embodiment of the present invention;

[0057] Figure 5 A flowchart of steps for obtaining a desktop plane in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of a core area in an embodiment of the present invention;

[0059] Figure 7 This is another schematic diagram of a core area in an embodiment of the present invention;

[0060] Figure 8 A flowchart of the steps of obtaining a desktop area after overhead correction in an embodiment of the present invention;

[0061] Fig. 9 A flowchart of the steps of calculating a rotation matrix and a translation vector in an embodiment of the present invention;

[0062] Fig.10 A flowchart of another method for correcting a video desktop overhead shot in accordance with an embodiment of the present invention;

[0063] Fig.11 It is a structural schematic diagram of a video desktop overhead shooting correction system in an embodiment of the present invention;

[0064] Fig.12 is a structural schematic diagram of a video desktop overhead shooting correction device in an embodiment of the present invention; and

[0065] Fig.13 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0067] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] The embodiment of the present invention provides a method for correcting a video desktop overhead shot, aiming to solve the problems existing in the prior art.

[0069] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0070] The present invention uses an RGBD camera to obtain three-dimensional point cloud information including the desktop, and uses the area of ​​interest provided by the previous frame image to narrow the range, and then fits the desktop plane, and obtains the perspective transformation matrix of the RGBD camera and the desktop, thereby obtaining a desktop image after overhead correction. The present invention has the advantages of low cost, simple operation, high precision and high speed.

[0071] Figure 1 FIG. 1 is a flowchart of a method for correcting a video desktop overhead shot in an embodiment of the present invention. Figure 1 As shown, the steps of a video desktop overhead shooting correction method in an embodiment of the present invention include:

[0072] Step S1: Obtain a first frame of image taken by an RGBD camera, and then calculate the three-dimensional point cloud information of pixels in the region of interest in the RGB image in combination with the intrinsic parameter matrix of the RGBD camera.

[0073] In this step, the region of interest is obtained from the previous frame of the first frame. The region of interest is a region including the target object. For example, for an online meeting, the presenter needs to display the content written on the desktop. At this time, the target object is the content written on the desktop, and the region of interest is a region including the content. The region of interest is a set of pixels on the RGB image, representing an area on the desktop plane. In the present invention, the RGBD camera can include various depth cameras with depth information. Combined with the internal parameters of the RGBD camera, three-dimensional reconstruction can be performed to obtain three-dimensional point cloud information of each pixel. Three-dimensional reconstruction is to reproduce the scene in three-dimensional space using the depth information of each pixel in the camera coordinate system, which can achieve the replication of the target space, so that various types of processing can be performed.

[0074] Step S2: Based on the three-dimensional point cloud information, filter out the point clouds outside the desktop, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop.

[0075] In this step, when filtering out point clouds outside the desktop, it can be done based on the characteristics of the desktop, such as using point cloud density, point cloud quantity, point cloud distribution and other features for screening. When using point cloud density for screening, point clouds with point cloud density below a preset threshold are removed, leaving only point clouds with higher density. It is also possible to remove areas with smaller numbers based on the number of point clouds, and only retain faces with larger areas. It is also possible to retain only faces that can be well fitted to a plane based on the distribution of point clouds. Of course, screening can also be performed using two or more methods. Then, the plane where the desktop is located is fitted using RANSAC, ICP algorithm or other similar algorithms.

[0076] Based on the fitted plane, the normal vector of the desktop plane can be calculated. The normal vector points to the center of the target space, not away from it. Then the center of mass of the desktop is calculated, and the distance between the center of mass and the RGBD camera can be obtained.

[0077] Step S3: Calculate the perspective transformation matrix H and perform perspective transformation on the image to obtain the desktop area after overhead correction.

[0078] In this step, the perspective transformation matrix H is a transformation matrix in three-dimensional space. The perspective transformation matrix H is used to perform perspective transformation on the desktop part of the RGB image to obtain the corrected desktop. When performing the transformation, the perspective transformation matrix H can be used to perform perspective transformation on the entire image, and then the corrected desktop can be obtained based on the correspondence between the desktop pixel points; or the desktop pixel points can be obtained first, and then the perspective transformation matrix H can be used to perform perspective transformation on the desktop pixel points. Figure 3 As shown, this step can transform the numbers in the figure, that is, from Figure 3 The left picture is converted to the right picture, which is suitable for human eye observation and has no distortion or other problems.

[0079] Step S4: proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the area of ​​interest of the next frame.

[0080] In this step, the region of interest is enlarged based on the desktop region identified in step S3. Figure 2 As shown, on the RGB image, the above steps obtain the desktop area. The desktop area contains the content to be converted. The area corresponding to the desktop area is enlarged, such as 1.1 times, 1.2 times, 1.3 times, 1.4 times or any other multiple greater than 1, to obtain the region of interest of the next frame.

[0081] In some embodiments, the desktop area is subjected to target recognition to obtain the target object area, and then the target object area is proportionally enlarged as the region of interest of the next frame. Compared with the aforementioned embodiment, this embodiment does not enlarge the desktop area, but enlarges the target object area in the desktop area, which can further reduce the amount of calculation and achieve adaptive adjustment of the target object. When performing target recognition, this embodiment recognizes various graphics, texts and other contents on the desktop, regards various symbols as target objects, and takes the smallest rectangle that can contain each target object as the target object area.

[0082] Figure 4 FIG. 1 is a flow chart of steps for obtaining the distance between the normal vector and the center of mass in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining the distance between the normal vector and the center of mass includes:

[0083] Step S21: filtering out pixels whose depth values ​​are too large to obtain a core area.

[0084] In this step, the depth value is used to filter out distant areas, such as filtering out areas outside 2m, 2.5m, 3m or any other depth value, to obtain all pixels within the close distance as the core area. The core area can be the effective measurement range of the RGBD camera or the working range set by the user.

[0085] In some embodiments, in addition to filtering using depth values, filtering is also performed based on the area where the pixel points are located in the image. For example, an area in the middle of the image is set as a valid area, and the intersection of this area and the depth filtered area constitutes the core area. This embodiment can minimize the number of pixels that need to be processed, thereby reducing the amount of calculation, lowering the requirements for computing power, and reducing calculation time.

[0086] Step S22: identifying a plane area in the core area and performing fitting to obtain a desktop plane.

[0087] In this step, an average area is identified in the core area, and then a plane fitting is performed to obtain a desktop plane. When fitting a plane, a variety of fitting methods can be used. In this embodiment, the RANSAC algorithm is preferably used because the point cloud distribution of the plane measurement points presents a certain randomness, and the RANSAC algorithm can quickly, efficiently and accurately obtain the plane.

[0088] Step S23: Calculate the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop.

[0089] In this step, the normal vector of the desktop plane can be calculated according to the desktop plane equation. The distance between the center of the RGBD camera lens and the center of mass of the desktop can also be calculated. In the camera coordinate system, the distance between the RGBD camera and the center of mass of the desktop can be directly expressed by the center of mass coordinates of the desktop.

[0090] This embodiment filters the pixels by depth values, reduces the amount of data processing, improves data accuracy, and better fits the desktop plane, so that the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop can be calculated.

[0091] Figure 5 FIG. 1 is a flow chart of steps for obtaining a desktop plane in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of obtaining a desktop plane includes:

[0092] Step S221: Identify a plane area within the core area.

[0093] In this step, the plane area includes multiple planes, such as 2, 3, 4, 5 or any other number. Figure 6 and Figure 7 As shown, there are often various items placed on the desktop, such as books, stands, express boxes, exhibits, etc., and these items have flat surfaces and need to be distinguished from the desktop.

[0094] Step S222: Obtain the first plane with the largest area and the second plane connected to the largest number of the planes.

[0095] In this step, the first plane and the second plane may be different planes, such as Figure 6 As shown. Figure 6 In the table, the tabletop is the plane with the largest area, and a supporting surface is the plane connected to the most planes. When there are many objects on the table and they are scattered, usually the plane with the largest area and the plane connected to the most planes are the same plane.

[0096] Step S223: If the first plane and the second plane are different planes, execute step S224; if the first plane and the second plane are the same plane, mark the first plane as a desktop plane.

[0097] In this step, the first plane and the second plane are compared to see if they are the same plane. The comparison can be performed by the overlap of the plane point clouds or the plane equation. Through the comparison in this step, the final desktop plane can be determined, providing a benchmark for the subsequent calculation steps.

[0098] Step S224: identify the target object on the second plane and calculate the area. If the target object exists and the area is greater than the first threshold, mark the second plane as the desktop plane; otherwise, mark the first plane as the desktop plane.

[0099] In this step, when the first plane is different from the second plane, the second plane is determined. Since desktop display usually requires a larger area for display, the first plane is assumed to be the desktop. Only when the second plane meets the requirements in this step, the second plane is marked as the desktop plane.

[0100] This embodiment identifies different planes, obtains a first plane and a second plane, and then processes them. That is, by splitting, comparing and processing multiple planes, various situations existing on the desktop are processed separately, so that an accurate desktop can be obtained in various display scenarios.

[0101] Figure 8 FIG. 1 is a flowchart of a step of obtaining a desktop area after overhead correction in an embodiment of the present invention. Figure 8 As shown, in an embodiment of the present invention, a step of obtaining a desktop area after overhead correction includes:

[0102] Step S31: Calculate the rotation matrix R and translation vector T required to convert the oblique viewing angle of the desktop into a top-down viewing angle.

[0103] In this step, the rotation matrix R can be calculated based on the angle between the desktop normal vector and the optical axis of the RGBD camera. The rotation matrix R transforms the desktop into a fixed orientation, that is, the orientation of the desktop normal vector changes. According to the center of mass position of the desktop, the translation vector T can be calculated. The translation vector T moves the center of mass position to a preset position. The rotation matrix R and the translation vector T are both data in three-dimensional space.

[0104] Step S32: Based on the rotation matrix R and the translation vector T, calculate the perspective transformation matrix H.

[0105] In this step, the perspective transformation matrix H is:

[0106]

[0107] Among them, the first item C int is the camera internal parameter, and the r in the second term is the corresponding element in the 3x3 rotation matrix, T x 、T y 、T z are the corresponding elements in the translation vector T. The perspective transformation matrix H does not rely on a fixed reference point and can achieve the transformation of any position on the desktop without any reference object, without the need for target recognition, and has a wide range of applications.

[0108] Step S33: using the perspective transformation matrix H, the image is perspective transformed to obtain the desktop area after overhead correction.

[0109] In this step, the desktop part in the RGB image is perspective transformed using the perspective transformation matrix H to obtain the corrected desktop. When performing the transformation, the perspective transformation matrix H can be used to perform perspective transformation on the entire image, and then the corrected desktop can be obtained based on the correspondence between the desktop pixels; or the desktop pixels can be obtained first, and then the perspective transformation matrix H can be used to perform perspective transformation on the desktop pixels.

[0110] This embodiment first calculates the rotation matrix R and the translation vector T, and then calculates the perspective transformation matrix H, and then corrects the RGB image. Through the transformation in the three-dimensional space, the display content is accurately transformed without distortion and other phenomena, with very high accuracy.

[0111] Fig. 9 FIG. 1 is a flow chart of the steps of calculating a rotation matrix and a translation vector in an embodiment of the present invention. Fig. 9 As shown, in an embodiment of the present invention, a step of calculating a rotation matrix and a translation vector includes:

[0112] Step S311: Based on the normal vector a of the desktop and the target normal vector b=(0,0,-1), obtain the rotation matrix R:

[0113]

[0114] In this step, I is the unit matrix, v = a × b, s = ||v||sin(α), α is the angle between a and b, c = a·b,

[0115]

[0116] Step S312: According to the formula T = -(R·P center_above ) to find the translation vector T.

[0117] In this step, R is the rotation matrix, P center_above =P center +h*a,P center is the coordinate of the center of mass of the desktop, and h is the overlooking height specified by the user.

[0118] This embodiment realizes the calculation of the rotation matrix and the translation vector in three-dimensional space, and can realize image conversion in three-dimensional space with higher accuracy than image conversion in two-dimensional space. At the same time, the restored two-dimensional image is more realistic and free of deformation and other problems.

[0119] Fig.10FIG. 1 is a flowchart of another method for correcting a video desktop overhead shot in an embodiment of the present invention. Fig.10 As shown, compared with the above-mentioned embodiment, another method for correcting a video desktop overhead shot in the embodiment of the present invention further includes:

[0120] Step S5: performing target recognition on a preset area on the desktop, and scaling the target object after removing the background area to obtain the best display effect of the target object.

[0121] In this step, target recognition is performed on the preset area to obtain a more accurate target object area. The background of the target object area is removed to obtain a target object without background. The background can be a solid color, such as white, black, etc. The target object without background is scaled to make it the best proportion of the display area, such as filling the display area, and being a fixed distance away from the border.

[0122] This embodiment provides the target object so that the target object is not affected by background factors during display, thereby being unaffected by background factors and lighting during display, and having a better display effect.

[0123] Fig.11 FIG. 1 is a structural diagram of a video desktop overhead correction system according to an embodiment of the present invention. Fig.11 As shown, a video desktop overhead shooting correction system in an embodiment of the present invention includes:

[0124] A point cloud module is used to obtain a first frame of image taken by an RGBD camera, and then calculate the three-dimensional point cloud information of pixels in the region of interest in the RGB image in combination with the internal parameter matrix of the RGBD camera; wherein the region of interest is obtained from a frame of image previous to the first frame of image;

[0125] A first calculation module is used to filter out point clouds outside the desktop based on the three-dimensional point cloud information, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop;

[0126] The second calculation module is used to calculate the perspective transformation matrix H and perform perspective transformation on the image to obtain the desktop area after overhead correction;

[0127] The region of interest module is used to proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the region of interest of the next frame.

[0128] Specifically, the point cloud module generates a three-dimensional point cloud through an RGBD image. The first calculation module uses the point cloud information to calculate the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop. The second calculation module calculates the perspective transformation matrix H based on the data of the first calculation module. Finally, the transformation module performs perspective transformation on the image to obtain a corrected desktop image.

[0129] This embodiment uses an RGBD camera to obtain three-dimensional point cloud information including the desktop, and uses the area of ​​interest provided by the previous frame image to narrow the range, and then fits the desktop plane, and obtains the perspective transformation matrix of the RGBD camera and the desktop, thereby obtaining a desktop image after overhead correction, which has the advantages of low cost, simple operation, high accuracy and speed.

[0130] The embodiment of the present invention also provides a video desktop overhead correction device, comprising a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of a video desktop overhead correction method by executing the executable instructions.

[0131] As mentioned above, this embodiment uses an RGBD camera to obtain three-dimensional point cloud information including the desktop, and uses the area of ​​interest provided by the previous frame image to narrow the range, and then fits the desktop plane, and obtains the perspective transformation matrix of the RGBD camera and the desktop, thereby obtaining a desktop image after overhead correction, which has the advantages of low cost, simple operation, high precision and speed.

[0132] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0133] Fig.12 Schematic diagram of the structure of a video desktop overhead correction device in an embodiment of the present invention. Fig.12 The electronic device 600 according to this embodiment of the present invention is described. Fig.12 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0134] like Fig.12 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0135] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned video desktop overhead correction method of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0136] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0137] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0138] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0139] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.12 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0140] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and the steps of a method for correcting a video desktop overhead shot are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present invention described in the above-mentioned method for correcting a video desktop overhead shot of this specification.

[0141] As shown above, this embodiment uses an RGBD camera to obtain three-dimensional point cloud information including the desktop, and uses the region of interest provided by the previous frame image to narrow the range, and then fits the desktop plane, and obtains the perspective transformation matrix of the RGBD camera and the desktop, thereby obtaining a desktop image after overhead correction, which has the advantages of low cost, simple operation, high accuracy and speed.

[0142] Fig.13 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.13 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0143] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0145] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0146] This embodiment uses an RGBD camera to obtain three-dimensional point cloud information including the desktop, and uses the area of ​​interest provided by the previous frame image to narrow the range, and then fits the desktop plane, and obtains the perspective transformation matrix of the RGBD camera and the desktop, thereby obtaining a desktop image after overhead correction, which has the advantages of low cost, simple operation, high accuracy and speed.

[0147] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0148] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A video desktop overhead shooting correction method, characterized in that: The steps include: Step S1: obtaining a first frame image taken by an RGBD camera, and then calculating the three-dimensional point cloud information of pixels in a region of interest in the RGB image in combination with an internal parameter matrix of the RGBD camera; wherein the region of interest is obtained from a frame image preceding the first frame image; Step S2: based on the three-dimensional point cloud information, filter out the point clouds outside the desktop, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop; Step S3: Calculate the perspective transformation matrix H, perform perspective transformation on the image, and thus obtain the desktop area after overhead correction; Step S4: proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the area of ​​interest of the next frame.

2. A method for correcting a video desktop overhead shot according to claim 1, characterized in that: Step S2 includes: Step S21: filtering out pixel points whose depth values ​​are too large to obtain a core area; Step S22: identifying a plane area in the core area and performing fitting to obtain a desktop plane; Step S23: Calculate the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop.

3. A method for correcting a video desktop overhead shot according to claim 2, characterized in that: Step S22 includes: Step S221: identifying a plane region in the core region; wherein the plane region includes a plurality of planes; Step S222: obtaining a first plane with the largest area and a second plane connected to the largest number of the planes; Step S223: if the first plane and the second plane are different planes, execute step S224; if the first plane and the second plane are the same plane, mark the first plane as a desktop plane; Step S224: identify the target object on the second plane and calculate the area. If the target object exists and the area is greater than the first threshold, mark the second plane as the desktop plane; otherwise, mark the first plane as the desktop plane.

4. The method for correcting a video desktop overhead shot according to claim 1, characterized in that: Step S3 includes: Step S31: Calculate the rotation matrix R and translation vector T required to convert the oblique viewing angle of the desktop into a top-down viewing angle; Step S32: Calculate the perspective transformation matrix H based on the rotation matrix R and the translation vector T; Step S33: using the perspective transformation matrix H, the image is perspective transformed to obtain the desktop area after overhead correction.

5. A method for correcting a video desktop overhead shot according to claim 4, characterized in that: Step S31 includes: Step S311: Based on the normal vector a of the desktop and the target normal vector b=(0,0,-1), obtain the rotation matrix R: Where I is the unit matrix, v = a × b, s = ||v||sin(α), α is the angle between a and b, c = a·b, Step S312: According to the formula T = -(R·P center_above ) to obtain the translation vector T; where R is the rotation matrix, P center_above =P center +h*a,P center is the coordinate of the center of mass of the desktop, and h is the overlooking height specified by the user.

6. A method for correcting a video desktop overhead shot according to claim 4, characterized in that: In step S32, the perspective transformation matrix H is obtained according to the following formula: Among them, the first item C int is the camera internal parameter, and the r in the second term is the corresponding element in the 3x3 rotation matrix, T x , T y , T z are the corresponding elements in the translation vector T respectively.

7. The method for correcting a video desktop overhead shot according to claim 1, characterized in that: Also includes: Step S5: performing target recognition on a preset area on the desktop, and scaling the target object after removing the background area to obtain the best display effect of the target object.

8. A video desktop overhead correction system, used to implement the video desktop overhead correction method according to any one of claims 1 to 7, characterized in that: include: A point cloud module is used to obtain a first frame of image taken by an RGBD camera, and then calculate the three-dimensional point cloud information of pixels in the region of interest in the RGB image in combination with the internal parameter matrix of the RGBD camera; wherein the region of interest is obtained from a frame of image previous to the first frame of image; A first calculation module is used to filter out point clouds outside the desktop based on the three-dimensional point cloud information, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the RGBD camera and the center of mass of the desktop; The second calculation module is used to calculate the perspective transformation matrix H and perform perspective transformation on the image to obtain the desktop area after overhead correction; The region of interest module is used to proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the region of interest of the next frame.

9. A video desktop overhead correction device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the video desktop overhead shooting correction method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the video desktop overhead shooting correction method described in any one of claims 1 to 7 are implemented.

Images