Depth camera with content correction function

Abstract

A depth camera with a content correction function comprises a light projection module used for projecting a structured light beam or floodlight to a target space; the light receiving module is used for receiving the structured light beam or the floodlight to generate an electric signal; and the processor is used for generating image information according to the electric signal, performing three-dimensional reconstruction on the image information, fitting a desktop plane in a three-dimensional space, calculating a distance between a normal vector of the desktop plane and the center of mass of the desktop to obtain a perspective transformation matrix H, and obtaining a corrected desktop area. The method has the advantages of being low in cost, easy and convenient to operate, high in precision and rapid, can support output of corrected characters and images, and is more suitable for conferences and other scenes compared with a traditional depth camera.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a depth camera with a content correction function. Background Art

[0002] In recent years, with the rise of online video communication, online education, online meetings, online live broadcasts and other online video communication methods have become increasingly popular. However, the camera used for online video usually needs to face the person in front of the screen. Therefore, when people need to write and draw manually during communication, the desktop in the picture will appear slanted or reversed due to the camera's viewing angle, making communication very inconvenient.

[0003] Currently, the main solution to this problem is to use a tablet or other device to connect to the online conference software at the same time, use the tablet as an electronic writing board, and then project the electronic writing board generated in real time on the tablet onto the screen, thereby obtaining a calibrated writing board that is more in line with the human eye.

[0004] However, this solution has two main problems: 1. The solution is expensive. In addition to the conference camera, a special electronic writing board is required. 2. The operation is inconvenient. Users need to manually connect the electronic writing board and the video conferencing software. The complexity depends on the support of the conference software for the electronic writing board. For ordinary video communication users, they hope to bypass these two problems. It would be best if they can directly use the most convenient paper and pen to write and draw to assist communication, just like face-to-face communication.

[0005] In other separate image processing schemes, a camera is used to obtain an image of the object to be converted, such as A4 paper. This scheme is only feasible for specific objects (such as only A4 paper), and has high requirements on the relative relationship between the camera and the object to be converted. The A4 paper needs to fill the camera's field of view to a large extent, and the angle between the camera and the object to be converted needs to be close to 90 degrees.

[0006] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0007] To this end, the present invention uses the image information obtained by the light receiving module to perform three-dimensional reconstruction, and then fits the desktop plane, obtains the perspective transformation matrix of the depth camera and the desktop, and thus obtains the desktop image after overhead correction. It has the advantages of low cost, simple operation, high precision and high speed, and can support the output of corrected text and images. Compared with traditional depth cameras, it is more suitable for scenes such as meetings.

[0008] In a first aspect, the present invention provides a depth camera with a content correction function, characterized in that it includes:

[0009] A light projection module, used to project a structured light beam or flood light into a target space;

[0010] A light receiving module, used for receiving the structured light beam or the flood light to generate an electrical signal;

[0011] A processor is used to generate image information according to the electrical signal, perform three-dimensional reconstruction on the image information, and fit a desktop plane in three-dimensional space, calculate the distance between the normal vector of the desktop plane and the center of mass of the desktop, calculate the perspective transformation matrix H, and obtain the corrected desktop area.

[0012] Optionally, the depth camera with content correction function is characterized in that the processor comprises:

[0013] An image information module, used to generate image information according to the electrical signal;

[0014] A point cloud module is used to obtain the first frame of image generated by the image information module, and then calculate the three-dimensional point cloud information of the pixels in the region of interest in the RGB image in combination with the intrinsic parameter matrix of the depth camera; wherein the region of interest is obtained from the previous frame of image of the first frame of image;

[0015] A first calculation module is used to filter out point clouds outside the desktop based on the three-dimensional point cloud information, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop;

[0016] The second calculation module is used to calculate the perspective transformation matrix H and perform perspective transformation on the image to obtain the desktop area after overhead correction;

[0017] The region of interest module is used to proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the region of interest of the next frame.

[0018] Optionally, the depth camera with content correction function is characterized by further comprising:

[0019] The adjustment module is used to adjust the density or intensity of the projection of the light projection module according to the desktop area after the overhead shooting correction.

[0020] Optionally, the depth camera with content correction function is characterized in that the first calculation module comprises the following steps during processing:

[0021] Step S21: filtering out pixel points whose depth values ​​are too large to obtain a core area;

[0022] Step S22: identifying a plane area in the core area and performing fitting to obtain a desktop plane;

[0023] Step S23: Calculate the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop.

[0024] Optionally, the depth camera with content correction function is characterized in that step S22 comprises:

[0025] Step S221: identifying a plane region in the core region; wherein the plane region includes a plurality of planes;

[0026] Step S222: obtaining a first plane with the largest area and a second plane connected to the largest number of the planes;

[0027] Step S223: if the first plane and the second plane are different planes, execute step S224; if the first plane and the second plane are the same plane, mark the first plane as a desktop plane;

[0028] Step S224: identify the target object on the second plane and calculate the area. If the target object exists and the area is greater than the first threshold, mark the second plane as the desktop plane; otherwise, mark the first plane as the desktop plane.

[0029] Optionally, the depth camera with content correction function is characterized in that the second calculation module comprises the following steps during processing:

[0030] Step S31: Calculate the rotation matrix R and translation vector T required to convert the oblique viewing angle of the desktop into a top-down viewing angle;

[0031] Step S32: Calculate the perspective transformation matrix H based on the rotation matrix R and the translation vector T;

[0032] Step S33: using the perspective transformation matrix H, the image is perspective transformed to obtain the desktop area after overhead correction.

[0033] Optionally, the depth camera with content correction function is characterized in that step S31 comprises:

[0034] Step S311: Based on the normal vector a of the desktop and the target normal vector b=(0,0,-1), obtain the rotation matrix R:

[0035]

[0036] Where I is the unit matrix, v = a × b, s = ||v||sin(α), α is the angle between a and b, c = a·b,

[0037]

[0038] Step S312: According to the formula T = -(R·P center_above ) to obtain the translation vector T; where R is the rotation matrix, P center_above =P center +h*a,P center is the coordinate of the center of mass of the desktop, and h is the overlooking height specified by the user.

[0039] Optionally, the depth camera with content correction function is characterized in that in step S32, the perspective transformation matrix H is obtained according to the following formula:

[0040]

[0041] Among them, the first item C int is the camera internal parameter, and the r in the second term is the corresponding element in the 3x3 rotation matrix, T x , T y , T z are the corresponding elements in the translation vector T respectively.

[0042] In a second aspect, the present invention provides a depth camera with a content correction function, characterized in that it includes:

[0043] A light projection module, used to project a structured light beam or flood light into a target space;

[0044] A light receiving module, used for receiving the structured light beam or the flood light to generate an electrical signal;

[0045] A processor is used to generate image information according to the electrical signal, perform three-dimensional reconstruction of the image information, fit a desktop plane in three-dimensional space, identify the position of the human eye, calculate the vector and distance between the center of mass of the desktop and the human eye, calculate the perspective transformation matrix H' that converts the perspective of the depth camera into the perspective of the human eye, and obtain the desktop area from the perspective of the human eye.

[0046] Optionally, the depth camera with content correction function is characterized in that when multiple depth cameras with content correction function are connected through a network, the display device can display images acquired by the depth cameras of other depth cameras with content correction function.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention obtains information including the desktop by using a depth camera, and then uses the three-dimensional point cloud information to identify and calculate the desktop. Compared with using a two-dimensional plane for calculation, the present invention can better utilize the three-dimensional space to segment the target object, and can use a larger range for transformation, thus avoiding distortion of the edge area of ​​the target object due to transformation, and has higher quality.

[0049] The present invention uses the desktop as the conversion object, does not rely on the recognition of the target object, can be applicable to the conversion of target objects of various shapes and sizes, and has very strong adaptability and versatility.

[0050] The present invention only requires a depth camera to achieve the conversion of the target object. Compared with relying on devices such as tablets, users can display various types of content more independently without being restricted by the size of hardware devices. At the same time, users can also use a variety of tools for display, which has higher freedom and convenience.

[0051] The present invention can intelligently correct the squinting or reverse-viewing desktop into a top-down view that conforms to the human eye's senses with one click, facilitating video users to communicate more effectively, and can also solve the problems of high cost and inconvenient operation caused by external electronic writing boards in the current mainstream solutions.

[0052] The present invention processes image information with the front-end computing power of the depth camera, can directly use the original data for processing, and can share three-dimensional space and other information with other degrees, with higher efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0054] Figure 1 is a schematic structural diagram of a depth camera with a content correction function according to an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a processor in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of a region of interest in an embodiment of the present invention;

[0057] Figure 4This is a schematic diagram of a correction effect in an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the structure of another processor in an embodiment of the present invention;

[0059] Figure 6 A flowchart of steps for obtaining the distance between a normal vector and a center of mass in an embodiment of the present invention;

[0060] Figure 7 A flowchart of steps for obtaining a desktop plane in an embodiment of the present invention;

[0061] Figure 8 This is a schematic diagram of a core area in an embodiment of the present invention;

[0062] Fig. 9 This is another schematic diagram of a core area in an embodiment of the present invention;

[0063] Fig.10 A flowchart of the steps of obtaining a desktop area after overhead correction in an embodiment of the present invention;

[0064] Fig.11 A flowchart of the steps of calculating a rotation matrix and a translation vector in an embodiment of the present invention;

[0065] Fig.12 Schematic diagram of the structure of another depth camera with content correction function in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0067] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] An embodiment of the present invention provides a depth camera with a content correction function, aiming to solve the problems existing in the prior art.

[0069] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0070] The present invention uses the image information obtained by the light receiving module to perform three-dimensional reconstruction, and then fits the desktop plane, obtains the perspective transformation matrix of the depth camera and the desktop, and thus obtains the desktop image after overhead correction. It has the advantages of low cost, simple operation, high precision and high speed, and can support the output of corrected text and images. Compared with traditional depth cameras, it is more suitable for scenes such as meetings.

[0071] Figure 1 FIG. 1 is a schematic diagram of the structure of a depth camera with a content correction function according to an embodiment of the present invention. Figure 1 As shown, a depth camera with a content correction function in an embodiment of the present invention includes:

[0072] The light projection module 1 is used to project a structured light beam or flood light into a target space.

[0073] Specifically, the light projection module 1 can project both structured light beams and floodlights. In more detail, the light projection module 1 can project a single light source type, such as a structured light beam or a floodlight, and can switchably project a structured light beam or a floodlight, that is, sometimes project a structured light beam, sometimes project a floodlight, or project a structured light beam and a floodlight at the same time. Different projection methods require different structures of the light projection module. There are many schemes for the setting of the corresponding structure in the prior art, and the present embodiment can adopt any one of them or not.

[0074] The light receiving module 2 is used to receive the structured light beam or the flood light to generate an electrical signal.

[0075] Specifically, the light receiving module can be a structured light receiver or a TOF receiver. The light receiving module is compatible with the light projection module. If the light projection module is a structured light projector, the light receiving module is a structured light receiver. If the light projection module is a floodlight projector, the light receiving module is a TOF receiver. If the light projection module can emit both structured light and floodlight, the light receiving module is a TOF receiver.

[0076] Processor 3 is used to generate image information according to the electrical signal, reconstruct the image information in three dimensions, and fit the desktop plane in three-dimensional space, calculate the perspective transformation matrix H by calculating the distance between the normal vector of the desktop plane and the center of mass of the desktop, and obtain the corrected desktop area.

[0077] Specifically, the processor 3 generates image information according to the electrical signal. The processor 3 can generate various types of image information such as depth images, depth maps, RGB maps, speckle maps, grayscale maps, etc. Since the signal obtained by the light receiving module contains depth information, the processor 3 can perform three-dimensional reconstruction according to the image information. The reconstructed three-dimensional space is a three-dimensional space under the camera coordinate system. By identifying the desktop, the desktop plane can be fitted, and then the normal vector of the desktop plane can be calculated. The center of mass of the desktop is obtained to obtain the distance between the center of mass of the desktop and the origin of the depth coordinate. According to the angle of the normal vector and the distance from the center of mass, the perspective transformation matrix H of the desktop plane transformation is calculated, so that the desktop plane can be converted into a bird's-eye view.

[0078] In some embodiments, the power of the light projection module is adjusted according to the brightness or clarity of the corrected desktop area. When the brightness or clarity of the corrected desktop area is poor, the power of the light projection module is increased so that the brightness or clarity of the corrected desktop area meets the preset requirements. When the brightness of the corrected desktop area is too high, the power of the light projection module is reduced so that the brightness or clarity of the corrected desktop area meets the preset requirements.

[0079] Figure 2 FIG. 1 is a schematic diagram of the structure of a processor in an embodiment of the present invention. Figure 2 As shown, a processor in an embodiment of the present invention includes:

[0080] The image information module 31 is used to generate image information according to the electrical signal.

[0081] Specifically, the image information is determined by the type of the light receiving module. The image information includes depth information and may also include at least one of an infrared image, an RGB image, a light spot image, etc. The three-dimensional information of the target space can be obtained through the image information.

[0082] The point cloud module 32 is used to obtain the first frame image generated by the image information module, and then calculate the three-dimensional point cloud information of the pixels in the area of ​​interest in the RGB image in combination with the intrinsic parameter matrix of the depth camera.

[0083] Specifically, the region of interest is obtained from the previous frame of the first frame. The region of interest is an area that includes the target object. For example, for an online meeting, the presenter needs to display the content written on the desktop. At this time, the target object is the content written on the desktop, and the region of interest is an area that includes the content. The region of interest is a set of pixels on the RGB image, representing an area on the desktop plane. Combined with the intrinsic parameters of the depth camera, three-dimensional reconstruction can be performed to obtain three-dimensional point cloud information for each pixel. Three-dimensional reconstruction is to reproduce the scene in three-dimensional space using the depth information of each pixel in the camera coordinate system, which can achieve the replication of the target space, so that various types of processing can be performed.

[0084] The first calculation module 33 is used to filter out point clouds outside the desktop based on the three-dimensional point cloud information, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop.

[0085] Specifically, when filtering out point clouds outside the desktop, it can be done based on the characteristics of the desktop, such as using point cloud density, point cloud quantity, point cloud distribution and other features for screening. When using point cloud density for screening, point clouds with point cloud density below the preset threshold are removed, leaving only point clouds with higher density. It is also possible to remove smaller areas based on the number of point clouds and only retain larger areas. It is also possible to retain only faces that can be well fitted to the plane based on the distribution of point clouds. Of course, it is also possible to filter using two or more methods. Then, the plane where the desktop is located is fitted using RANSAC, ICP algorithm or other similar algorithms.

[0086] Based on the fitted plane, the normal vector of the desktop plane can be calculated. The normal vector points to the center of the target space, not away from it. Then the center of mass of the desktop is calculated, and the distance between the center of mass and the depth camera can be obtained.

[0087] The second calculation module 34 is used to calculate the perspective transformation matrix H and perform perspective transformation on the image, so as to obtain the desktop area after overhead correction.

[0088] Specifically, the perspective transformation matrix H is a transformation matrix in three-dimensional space. The perspective transformation matrix H is used to perform perspective transformation on the desktop part of the RGB image to obtain the corrected desktop. When performing the transformation, the perspective transformation matrix H can be used to perform perspective transformation on the entire image, and then the corrected desktop can be obtained based on the correspondence between the desktop pixel points; or the desktop pixel points can be obtained first, and then the perspective transformation matrix H can be used to perform perspective transformation on the desktop pixel points. Figure 4 As shown, this step can transform the numbers in the figure, that is, from Figure 4 The left picture is converted to the right picture, which is suitable for human eye observation and has no distortion or other problems.

[0089] The region of interest module 35 is used to proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the region of interest of the next frame.

[0090] Specifically, the region of interest is enlarged based on the desktop area identified in step S3. Figure 3 As shown, on the RGB image, the above steps obtain the desktop area. The desktop area contains the content to be converted. The area corresponding to the desktop area is enlarged, such as 1.1 times, 1.2 times, 1.3 times, 1.4 times or any other multiple greater than 1, to obtain the region of interest of the next frame.

[0091] In some embodiments, the desktop area is subjected to target recognition to obtain the target object area, and then the target object area is proportionally enlarged as the region of interest of the next frame. Compared with the aforementioned embodiment, this embodiment does not enlarge the desktop area, but enlarges the target object area in the desktop area, which can further reduce the amount of calculation and achieve adaptive adjustment of the target object. When performing target recognition, this embodiment recognizes various graphics, texts and other contents on the desktop, regards various symbols as target objects, and takes the smallest rectangle that can contain each target object as the target object area.

[0092] This embodiment utilizes the region of interest provided by the previous frame to greatly reduce the number of pixels that need to be calculated, and utilizes the identified desktop to provide the region of interest to the next frame. By transferring the region of interest, it is possible to reduce the amount of calculation and increase the recognition conversion speed, and to adjust the region of interest for each frame, thereby achieving adaptive adjustment of the target object and fully displaying the target object.

[0093] Figure 5 FIG. 1 is a schematic diagram of the structure of another processor in an embodiment of the present invention. Figure 5 As shown, compared with the above-mentioned embodiment, another processor in the embodiment of the present invention further includes:

[0094] The adjustment module 36 is used to adjust the density or intensity of the light projection module according to the desktop area after the overhead correction.

[0095] Specifically, when the target object in the desktop area after overhead correction is too dark, the density or intensity of the light projection module can be increased. When the target object in the desktop area after overhead correction is overexposed, the density or intensity of the light projection module can be reduced. When the clarity of the desktop area after overhead correction is not enough, increase the density of the structured light beam, or increase the floodlight intensity. It should be noted that increasing the density of the structured light beam requires the ability to control the number of light-emitting units of the light source during structured light projection. For example, the laser units are arranged in an array and divided into several groups. The laser units between each group are controlled individually, and the number of light-emitting units of the light source can be controlled by controlling the lighted groups and the number of groups.

[0096] In some embodiments, the structured light sources are grouped, and the intensity or density of the light sources corresponding to the group in the desktop area is adjusted to adjust the brightness or clarity of the desktop area. This embodiment can achieve separate adjustment of the desktop area, so that the desktop area and other areas can both obtain optimal illumination, and the entire image and desktop content can be clearly imaged.

[0097] In this embodiment, the density or intensity of the light projection module is adjusted according to the desktop area after the correction of the overhead view, so as to adjust the brightness and clarity of the desktop area, ensure clear imaging of the desktop area, and thus better generate a corrected desktop area image.

[0098] Figure 6 FIG. 1 is a flow chart of steps for obtaining the distance between the normal vector and the center of mass in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of obtaining the distance between the normal vector and the center of mass includes:

[0099] Step S21: filtering out pixels whose depth values ​​are too large to obtain a core area.

[0100] In this step, the depth value is used to filter out distant areas, such as filtering out areas outside 2m, 2.5m, 3m or any other depth value, to obtain all pixels within the close distance as the core area. The core area can be the effective measurement range of the depth camera or the working range set by the user.

[0101] In some embodiments, in addition to filtering using depth values, filtering is also performed based on the area where the pixel points are located in the image. For example, an area in the middle of the image is set as a valid area, and the intersection of this area and the depth filtered area constitutes the core area. This embodiment can minimize the number of pixels that need to be processed, thereby reducing the amount of calculation, lowering the requirements for computing power, and reducing calculation time.

[0102] Step S22: identifying a plane area in the core area and performing fitting to obtain a desktop plane.

[0103] In this step, an average area is identified in the core area, and then a plane fitting is performed to obtain a desktop plane. When fitting a plane, a variety of fitting methods can be used. In this embodiment, the RANSAC algorithm is preferably used because the point cloud distribution of the plane measurement points presents a certain randomness, and the RANSAC algorithm can quickly, efficiently and accurately obtain the plane.

[0104] Step S23: Calculate the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop.

[0105] In this step, the normal vector of the desktop plane can be calculated according to the desktop plane equation. The distance between the center of the depth camera lens and the center of mass of the desktop can also be calculated. In the camera coordinate system, the distance between the depth camera and the center of mass of the desktop can be directly expressed by the center of mass coordinates of the desktop.

[0106] This embodiment filters the pixels by depth values, reduces the amount of data processing, improves data accuracy, and better fits the desktop plane, so that the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop can be calculated.

[0107] Figure 7 FIG. 1 is a flow chart of steps for obtaining a desktop plane in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of obtaining a desktop plane includes:

[0108] Step S221: Identify a plane area within the core area.

[0109] In this step, the plane area includes multiple planes, such as 2, 3, 4, 5 or any other number. Figure 8 and Fig. 9 As shown, there are often various items placed on the desktop, such as books, stands, express boxes, exhibits, etc., and these items have flat surfaces and need to be distinguished from the desktop.

[0110] Step S222: Obtain the first plane with the largest area and the second plane connected to the largest number of the planes.

[0111] In this step, the first plane and the second plane may be different planes, such as Figure 8 As shown. Figure 8 In the table, the tabletop is the plane with the largest area, and a supporting surface is the plane connected to the most planes. When there are many objects on the table and they are scattered, usually the plane with the largest area and the plane connected to the most planes are the same plane.

[0112] Step S223: If the first plane and the second plane are different planes, execute step S224; if the first plane and the second plane are the same plane, mark the first plane as a desktop plane.

[0113] In this step, the first plane and the second plane are compared to see if they are the same plane. The comparison can be performed by the overlap of the plane point clouds or the plane equation. Through the comparison in this step, the final desktop plane can be determined, providing a benchmark for the subsequent calculation steps.

[0114] Step S224: identify the target object on the second plane and calculate the area. If the target object exists and the area is greater than the first threshold, mark the second plane as the desktop plane; otherwise, mark the first plane as the desktop plane.

[0115] In this step, when the first plane is different from the second plane, the second plane is determined. Since desktop display usually requires a larger area for display, the first plane is assumed to be the desktop. Only when the second plane meets the requirements in this step, the second plane is marked as the desktop plane.

[0116] This embodiment identifies different planes, obtains a first plane and a second plane, and then processes them. That is, by splitting, comparing and processing multiple planes, various situations existing on the desktop are processed separately, so that an accurate desktop can be obtained in various display scenarios.

[0117] Fig.10 FIG. 1 is a flowchart of a step of obtaining a desktop area after overhead correction in an embodiment of the present invention. Fig.10 As shown, in an embodiment of the present invention, a step of obtaining a desktop area after overhead correction includes:

[0118] Step S31: Calculate the rotation matrix R and translation vector T required to convert the oblique viewing angle of the desktop into a top-down viewing angle.

[0119] In this step, the rotation matrix R can be calculated based on the angle between the desktop normal vector and the optical axis of the depth camera. The rotation matrix R transforms the desktop into a fixed orientation, that is, the orientation of the desktop normal vector changes. According to the center of mass position of the desktop, the translation vector T can be calculated. The translation vector T moves the center of mass position to a preset position. The rotation matrix R and the translation vector T are both data in three-dimensional space.

[0120] Step S32: Based on the rotation matrix R and the translation vector T, calculate the perspective transformation matrix H.

[0121] In this step, the perspective transformation matrix H is:

[0122]

[0123] Among them, the first item C int is the camera internal parameter, and the r in the second term is the corresponding element in the 3x3 rotation matrix, T x 、T y 、T z are the corresponding elements in the translation vector T. The perspective transformation matrix H does not rely on a fixed reference point and can achieve the transformation of any position on the desktop without any reference object, without the need for target recognition, and has a wide range of applications.

[0124] Step S33: using the perspective transformation matrix H, the image is perspective transformed to obtain the desktop area after overhead correction.

[0125] In this step, the desktop part in the RGB image is perspective transformed using the perspective transformation matrix H to obtain the corrected desktop. When performing the transformation, the perspective transformation matrix H can be used to perform perspective transformation on the entire image, and then the corrected desktop can be obtained based on the correspondence between the desktop pixels; or the desktop pixels can be obtained first, and then the perspective transformation matrix H can be used to perform perspective transformation on the desktop pixels.

[0126] This embodiment first calculates the rotation matrix R and the translation vector T, and then calculates the perspective transformation matrix H, and then corrects the RGB image. Through the transformation in the three-dimensional space, the display content is accurately transformed without distortion and other phenomena, with very high accuracy.

[0127] Fig.11 FIG. 1 is a flow chart of the steps of calculating a rotation matrix and a translation vector in an embodiment of the present invention. Fig.11 As shown, in an embodiment of the present invention, a step of calculating a rotation matrix and a translation vector includes:

[0128] Step S311: Based on the normal vector a of the desktop and the target normal vector b=(0,0,-1), obtain the rotation matrix R:

[0129]

[0130] In this step, I is the unit matrix, v = a × b, s = ||v||sin(α), α is the angle between a and b, c = a·b,

[0131]

[0132] Step S312: According to the formula T = -(R·P center_above ) to find the translation vector T.

[0133] In this step, R is the rotation matrix, P center_above =P center+h*a,P center is the coordinate of the center of mass of the desktop, and h is the overlooking height specified by the user.

[0134] This embodiment realizes the calculation of the rotation matrix and the translation vector in three-dimensional space, and can realize image conversion in three-dimensional space with higher accuracy than image conversion in two-dimensional space. At the same time, the restored two-dimensional image is more realistic and free of deformation and other problems.

[0135] Fig.12 FIG. 1 is a schematic diagram of the structure of another depth camera with content correction function according to an embodiment of the present invention. Fig.12 As shown, compared with the above-mentioned embodiment, another depth camera with content correction function in the embodiment of the present invention further includes:

[0136] The zoom module is used to perform target recognition on a preset area on the desktop and zoom the target object after removing the background area to obtain the best display effect of the target object.

[0137] Specifically, target recognition is performed on the preset area to obtain a more accurate target object area. The background of the target object area is removed to obtain a target object without background. The background can be a solid color, such as white, black, etc. The target object without background is scaled to make it the best proportion of the display area, such as filling the display area, and being a fixed distance away from the border.

[0138] This embodiment provides the target object so that the target object is not affected by background factors during display, thereby being unaffected by background factors and lighting during display, and having a better display effect.

[0139] In some embodiments, Figure 1 As shown, a depth camera with content correction function includes:

[0140] A light projection module, used to project a structured light beam or flood light into a target space;

[0141] A light receiving module, used for receiving the structured light beam or the flood light to generate an electrical signal;

[0142] A processor is used to generate image information according to the electrical signal, perform three-dimensional reconstruction of the image information, fit a desktop plane in three-dimensional space, identify the position of the human eye, calculate the vector and distance between the center of mass of the desktop and the human eye, calculate the perspective transformation matrix H' that converts the perspective of the depth camera into the perspective of the human eye, and obtain the desktop area from the perspective of the human eye.

[0143] Specifically, compared to the aforementioned embodiment that converts the desktop area into a top-down angle, this embodiment converts the desktop area into a human eye angle. When calculating, it is only necessary to replace the normal vector in the aforementioned embodiment with the vector of the desktop center of mass and the human eye, and replace the distance between the desktop center of mass and the camera with the distance between the desktop center of mass and the human eye. When the display device is a VR glasses, AR display, visionpro or other display devices, the presenter's perspective is more in line with people's viewing habits. When the video is connected, it is also easier for the presenter and the viewer to watch in the same dimension, which is more immersive.

[0144] When multiple depth cameras with a content correction function are connected via a network, the display device may display images acquired by the depth cameras of other depth cameras with a content correction function.

[0145] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0146] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth camera with content correction function, characterized in that: include: A light projection module, used to project a structured light beam or flood light into a target space; A light receiving module, used for receiving the structured light beam or the flood light to generate an electrical signal; The processor is used to generate image information according to the electrical signal, reconstruct the image information in three dimensions, and fit a desktop plane in three-dimensional space, calculate the distance between the normal vector of the desktop plane and the center of mass of the desktop, calculate the perspective transformation matrix H, and obtain the corrected desktop area.

2. The depth camera with content correction function according to claim 1, characterized in that: The processor comprises: An image information module, used to generate image information according to the electrical signal; A point cloud module is used to obtain the first frame of image generated by the image information module, and then calculate the three-dimensional point cloud information of the pixels in the region of interest in the RGB image in combination with the intrinsic parameter matrix of the depth camera; wherein the region of interest is obtained from the previous frame of image of the first frame of image; A first calculation module is used to filter out point clouds outside the desktop based on the three-dimensional point cloud information, fit the desktop plane, and then obtain the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop; The second calculation module is used to calculate the perspective transformation matrix H and perform perspective transformation on the image to obtain the desktop area after overhead correction; The region of interest module is used to proportionally enlarge the area corresponding to the desktop area in the RGB image to obtain the region of interest of the next frame.

3. The depth camera with content correction function according to claim 2, characterized in that: Also includes: The adjustment module is used to adjust the density or intensity of the projection of the light projection module according to the desktop area after the overhead shooting correction.

4. The depth camera with content correction function according to claim 2, characterized in that: The first calculation module includes the following steps during processing: Step S21: filtering out pixel points whose depth values ​​are too large to obtain a core area; Step S22: identifying a plane area in the core area and performing fitting to obtain a desktop plane; Step S23: Calculate the normal vector of the desktop plane and the distance between the depth camera and the center of mass of the desktop.

5. The depth camera with content correction function according to claim 4, characterized in that: Step S22 includes: Step S221: identifying a plane region in the core region; wherein the plane region includes a plurality of planes; Step S222: obtaining a first plane with the largest area and a second plane connected to the largest number of the planes; Step S223: if the first plane and the second plane are different planes, execute step S224; if the first plane and the second plane are the same plane, mark the first plane as a desktop plane; Step S224: identify the target object on the second plane and calculate the area. If the target object exists and the area is greater than the first threshold, mark the second plane as the desktop plane; otherwise, mark the first plane as the desktop plane.

6. The depth camera with content correction function according to claim 2, characterized in that: The second calculation module includes the following steps during processing: Step S31: Calculate the rotation matrix R and translation vector T required to convert the oblique viewing angle of the desktop into a top-down viewing angle; Step S32: Calculate the perspective transformation matrix H based on the rotation matrix R and the translation vector T; Step S33: using the perspective transformation matrix H, the image is perspective transformed to obtain the desktop area after overhead correction.

7. The depth camera with content correction function according to claim 5, characterized in that: Step S31 includes: Step S311: Based on the normal vector a of the desktop and the target normal vector b=(0,0,-1), obtain the rotation matrix R: Where I is the unit matrix, v = a × b, s = ||v||sin(α), α is the angle between a and b, c = a·b, Step S312: According to the formula T = -(R·P center_above ) to obtain the translation vector T; where R is the rotation matrix, P center_above =P center +h*a,P center is the coordinate of the center of mass of the desktop, and h is the overlooking height specified by the user.

8. The depth camera with content correction function according to claim 6, characterized in that: In step S32, the perspective transformation matrix H is obtained according to the following formula: Among them, the first item C int is the camera internal parameter, and the r in the second term is the corresponding element in the 3x3 rotation matrix, T x , T y , T z are the corresponding elements in the translation vector T respectively.

9. A depth camera with content correction function, characterized in that: include: A light projection module, used to project a structured light beam or flood light into a target space; A light receiving module, used for receiving the structured light beam or the flood light to generate an electrical signal; A processor is used to generate image information according to the electrical signal, perform three-dimensional reconstruction of the image information, fit a desktop plane in three-dimensional space, identify the position of the human eye, calculate the vector and distance between the center of mass of the desktop and the human eye, calculate the perspective transformation matrix H' that converts the perspective of the depth camera into the perspective of the human eye, and obtain the desktop area from the perspective of the human eye.

10. The depth camera with content correction function according to claim 1 or 9, characterized in that: When multiple depth cameras with a content correction function are connected via a network, the display device may display images acquired by the depth cameras of other depth cameras with a content correction function.

Images