Point cloud data correction method and correction device, electronic device and storage medium

Abstract

The present application discloses a correction method and correction device for point cloud data, an electronic device and a storage medium. The correction method for point cloud data includes: receiving annotation input for initial point cloud data; in response to the annotation input, annotating a target object in the initial point cloud data; determining N target points in the initial point cloud data according to the position information of the target object, where N is a positive integer; determining whether the target point is a visible point; and in the case where the target point is a visible point, correcting the depth information of the target point. The embodiment of the present application annotates target objects with mirror properties such as mirrors and windows in the room corresponding to the point cloud data through annotation input, and corrects the depth information of points near the target objects, thereby eliminating the acquisition error of the depth information of the mirror object caused by the depth acquisition device, and improving the accuracy of the depth information of the point cloud data.

Description

Technical Field

[0001] The present application belongs to the field of VR (Virtual Reality) house viewing technology, and specifically relates to a point cloud data correction method and correction device, electronic equipment and storage medium. Background Art

[0002] In related technologies, in order to improve the efficiency of house viewing, VR house viewing scenarios have emerged. Before VR house viewing, it is necessary to use depth acquisition equipment such as TOF (Time of flight) cameras, structured light cameras, laser cameras, etc. to shoot the room to obtain a three-dimensional point cloud model, and then build a VR model of the room based on the point cloud model.

[0003] However, the depth acquisition equipment cannot collect correct depth information when objects such as mirrors and windows have strong reflection or transmission, resulting in inaccurate depth information of some points in the point cloud data. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a point cloud data correction method and correction device, electronic device and storage medium, which can solve the problem of inaccurate depth information of point cloud data.

[0005] In a first aspect, an embodiment of the present application provides a method for correcting point cloud data, comprising:

[0006] Receiving annotation input of initial point cloud data;

[0007] In response to the annotation input, annotating a target object in the initial point cloud data;

[0008] According to the location information of the target object, N target points are determined in the initial point cloud data, where N is a positive integer;

[0009] Determine whether the target point is a visible point;

[0010] When the target point is a visible point, the depth information of the target point is corrected.

[0011] In a second aspect, an embodiment of the present application provides a point cloud data correction device, comprising:

[0012] A receiving module, used for receiving annotation input of initial point cloud data;

[0013] A labeling module, for labeling a target object in the initial point cloud data in response to a labeling input;

[0014] A determination module is used to determine N target points in the initial point cloud data according to the position information of the target object, where N is a positive integer;

[0015] A judgment module is used to judge whether the target point is a visible point;

[0016] The correction module is used to correct the depth information of the target point when the target point is a visible point.

[0017] In a third aspect, an embodiment of the present application provides a point cloud data correction device, comprising:

[0018] Memory, used to store programs or instructions;

[0019] A processor is used to implement the steps of the method of the first aspect when executing a program or an instruction.

[0020] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0022] In a sixth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method of the first aspect.

[0023] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0024] In an embodiment of the present application, after point cloud data is acquired through a depth acquisition device and an image acquisition device, such as a TOF camera and an RGB camera (Red Green Blue, red, green and blue three-channel color), the target objects with mirror properties, such as mirrors and windows, in the room corresponding to the point cloud data are marked through the annotation input according to the annotation input of the photographer or the user, and the depth information of the points nearby are corrected according to the corresponding position information of these target objects with mirror properties in the point cloud data, thereby eliminating the acquisition error of the depth information of the mirror object caused by the depth acquisition device and improving the accuracy of the depth information of the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart showing a method for correcting point cloud data according to an embodiment of the present application is shown;

[0026] Figure 2A schematic diagram of marking a target object according to an embodiment of the present application is shown;

[0027] Figure 3 A schematic diagram showing whether a target point is a visible point according to an embodiment of the present application is shown;

[0028] Figure 4 A schematic diagram of processing a depth map of a specular object according to an embodiment of the present application is shown;

[0029] Figure 5 A schematic diagram of a marked glass partition according to an embodiment of the present application is shown;

[0030] Figure 6 A schematic diagram showing a depth map of a glass partition according to an embodiment of the present application is shown;

[0031] Figure 7 A structural block diagram of a point cloud data correction device according to an embodiment of the present application is shown;

[0032] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present application is shown;

[0033] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] The point cloud data correction method and correction device, electronic device and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0037] In some embodiments of the present application, a method for correcting point cloud data is provided. Figure 1 A flow chart of a method for correcting point cloud data according to an embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0038] Step 102, receiving annotation input of initial point cloud data;

[0039] Step 104, in response to the annotation input, annotate the target object in the initial point cloud data;

[0040] Step 106, determining N target points in the initial point cloud data according to the position information of the target object, where N is a positive integer;

[0041] Step 108, determining whether the target point is a visible point;

[0042] Step 110: When the target point is a visible point, calibrate the depth information of the target point.

[0043] In an embodiment of the present application, after point cloud data is acquired through a depth acquisition device and an image acquisition device, such as a TOF camera and an RGB camera (Red Green Blue, red, green and blue three-channel color), the point cloud data can be displayed in a visual view, such as displaying a top view of the point cloud data.

[0044] After displaying the point cloud data, the system accepts annotation input from the photographer or user, and based on the annotation input, marks target objects with mirror properties such as mirrors and windows in the room corresponding to the point cloud data, and obtains the position information of the target objects in the point cloud data.

[0045] Since the depth acquisition device has strong reflection or transmission for objects such as mirrors and windows, it is impossible to collect correct depth information. Therefore, based on the existence of the target object, the depth information of some points near the target object may not be correctly obtained.

[0046] In response to this situation, the embodiment of the present application acquires the target point near the target object after marking the position information of the target object in the point cloud data, and determines whether the target point is a visualized point, that is, whether the target point is within the visible range after the room VR model is established through the initial point cloud data of the room.

[0047] If the target point is a visible point, that is, after the room VR model is normally established, the target point should be "seen" by the user, then the depth information of the target point is corrected to eliminate the error in the current point caused by the depth acquisition device's acquisition of the depth information of the mirror object, and obtain the correct point information.

[0048] After depth information correction is performed on all points that may have incorrect depth information sampling due to target objects with mirror properties, all points in the corrected point cloud data have correct depth information, thereby improving the accuracy of the depth information of the point cloud data and the accuracy of the room VR model established using the corrected point cloud data.

[0049] In some embodiments of the present application, in response to the annotation input, annotating the target object in the initial point cloud data includes:

[0050] According to the annotation input, determine the first point position, the second point position and thickness information;

[0051] Taking the first point as the starting point and the second point as the end point, determine the directed line segment;

[0052] Label the target object based on the directed line segments and thickness information.

[0053] In an embodiment of the present application, a photographer or a user may use a labeling module to label a directed line segment in the initial point cloud data.

[0054] Specifically, taking the example of a photographer marking a target object, first, the photographer marks a first point in the top view of the initial point cloud data, and the first point is the starting point of the target object. Then, the photographer marks a second point in the top view of the initial point cloud data, and the second point is the end point of the target object.

[0055] Specifically, Figure 2 A schematic diagram of marking a target object according to an embodiment of the present application is shown. Figure 2 As shown, the photographer marks the first point P0 and the second point P1, and forms a directed line segment L through the first point and the second point, which can effectively mark the position of the target object with mirror properties, such as mirrors, windows, etc. in the initial point cloud data of the room, as well as the mirror orientation with mirror properties.

[0056] Afterwards, the photographer inputs the thickness information, which is used to mark the thickness of the target object. For example, when the target object is a glass partition or double-sided glass, the special reflection or projection properties caused by its thickness will cause additional impact on the depth information of the point. Therefore, in this case, accurately marking the thickness of the target object is conducive to improving the correction accuracy of the depth information of the point.

[0057] Among them, the thickness information can be input by marking an undirected line segment, and the length of the marked undirected line segment can be accurately adjusted. For example, the user can manually input the thickness of the target object after marking the directed line segment, thereby improving the marking accuracy.

[0058] It can be understood that for specular surfaces, such as mirrors or windows, the thickness can be ignored. In this case, the thickness information can be set to 0 or default.

[0059] By accurately marking the target object, the correction effect of the initial point cloud data can be improved, and the accuracy of the corrected point cloud data can be improved.

[0060] In some embodiments of the present application, according to the position information of the target object, N target points are determined in the initial point cloud data, including:

[0061] Determine the center point of the target object;

[0062] Determine the distance value between each point in the initial point cloud data and the central point;

[0063] The points whose distance values ​​are less than the threshold are determined as target points.

[0064] In the embodiment of the present application, for each marked target object with mirror properties, points within a certain range around it are collected as target points to be corrected. Specifically, the center point of the target object is first determined, and the center point can be the center point of the mirror of the target object or the geometric center point of the target object, which is not limited in the present application.

[0065] After obtaining the center point of the target object, obtain the information of all N target points contained in the acquisition range, where the target point acquisition range refers to the points whose distance value from the center point is less than a threshold. The threshold is related to the effective depth acquisition range of the depth acquisition device used when collecting the initial point cloud data. For example, if the effective depth acquisition range of the depth acquisition device is 5 meters, the threshold can be set to 4.5 meters to 5.5 meters, such as 5 meters.

[0066] Specifically, the following formula can be used to determine whether the distance between a point and the center point of the target object is less than a threshold, that is, to determine whether a point is a target point:

[0067]

[0068] Among them, P start is the coordinate of the first point, P end is the coordinate of the second point, X is the coordinate of any point in the initial point cloud data, d min is the threshold value.

[0069] In the above formula, the coordinates of the first point P start , the coordinates of the second point P endThe coordinates X of any point in the initial point cloud data are all coordinates in the world coordinate system, and only represent the x-coordinate and the z-coordinate. If the coordinates of point X satisfy the above discriminant, it means that the distance between point X and the center point of the target object is less than the threshold, and point X is the target point.

[0070] By only including points whose distance to the target object is less than a threshold, that is, points near the target object, as points to be corrected, the amount of calculation during depth correction can be effectively reduced and the correction efficiency can be improved.

[0071] In some embodiments of the present application, determining whether a target point is a visible point includes:

[0072] Determine a corresponding first ray according to the position information of the target point, wherein the first ray passes through the target point and is related to a shooting direction of a camera that shoots depth information of the target point;

[0073] When the first ray intersects the target object, the target point is determined to be a visible point;

[0074] When the first ray does not intersect the target object, the target point is determined to be a non-visible point.

[0075] In the embodiment of the present application, after N target points that meet the conditions are determined according to the position of the target object, it is further determined whether these target points are visible points, that is, it is determined whether the target points should be "seen" by the user.

[0076] Specifically, Figure 3 A schematic diagram of determining whether a target point is a visible point according to an embodiment of the present application is shown. Figure 3 As shown in Figure 1, first, mark the position x of the camera that shoots the current target point. Then, according to the internal parameters of the camera that shoots the target point, determine the camera orientation and camera shooting range, that is, the camera orientation is Figure 3 Vector in Among them, P start That is, the first point, P end is the second point, the directed line segment P start P end That is, the marked target object.

[0077] According to the shooting direction of the camera, the first ray xP is determined. The starting point of the first ray xP is camera x, the direction of the first ray satisfies the shooting direction of camera x, and the first ray passes through the target point P. If the first ray xP is aligned with the directed line segment P start P end Intersect, such as Figure 3 As shown, the target point P is determined to be a visible point, that is, the depth information of the target point P needs to be corrected.

[0078] If the first ray xP is aligned with the directed line segment P start P end If they do not intersect, it is determined that the target point P is a non-visible point, and there is no need to correct the depth information of the target point P.

[0079] The specific judgment conditions are as follows:

[0080]

[0081] Where c1 is calculated by the following formula:

[0082]

[0083] Calculate c2 using the following formula:

[0084]

[0085] Among them, X is the coordinate of the target point, P start That is, the coordinates of the first point, P end is the coordinate of the second point, D is the orientation vector of the camera that captures the depth map of the target point, the direction vector D is determined by the rotation parameter of the camera device, c1 and c2 are the cross products of the direction vector D and the two vectors of the center point and the first and second points of the standard directed line segment, respectively. Specifically, Figure 3 For example, c1 is the direction vector D and the target point x and P start The cross product of the vectors between, c2 is the direction vector D and the target point x and P end The cross product of the vectors between .

[0086] In some embodiments of the present application, correcting the depth information of the target point includes:

[0087] Determine the intersection point of the first ray and the target object;

[0088] Determine the intersection point of the first ray and the target object;

[0089] According to the intersection point and the corresponding depth map information, the depth information of the target point is corrected.

[0090] In the embodiment of the present application, if the current target point is a visible point, the depth information of the current target point needs to be corrected. Specifically, according to the coordinates of the camera x and the coordinates of the target point P, the ray xP is calculated, and according to the target object, that is, the directed line segment P start P end Position information, calculate the ray xP and directed line segment P start P end The intersection point.

[0091] First, traverse the depth map pixels and reconstruct the point cloud based on the camera pose. Figure 3 As shown, for a point P in the point cloud, calculate whether it is located in the target object, that is, within the mirror annotation range (determine whether xP is within xP start With xP end between).

[0092] Specifically, determine whether the following conditions are met:

[0093]

[0094]

[0095] c3=P start P×P start P end >0;

[0096] Among them, c1, c2 and c3 are the target quantities, P is the coordinate of any point in the point cloud, P start That is, the coordinates of the first point, P end is the coordinate of the second point, and D is the direction vector of the camera that captures the depth map of the target point.

[0097] Among them, due to the line segment P start P end It is annotated based on the top view of the initial point cloud data to be corrected, so the line segment P start P end It can be regarded as a plane with height attributes. Therefore, the ray xP and the line segment P start P end The intersection point is also equivalent to the intersection point of the ray xP and the plane where the target object is located.

[0098] The calculation method is as follows:

[0099]

[0100] P new =x+ratio·(P–x)

[0101] Among them, r and p are intermediate quantities, P start That is, the coordinates of the first point, P end is the coordinate of the second point, X is the coordinate of the target point, P new That is, the ray xP and the line segment P start P end The intersection point of the ray xP and the plane where the target object is located.

[0102] Get the depth map taken by camera x, and determine the intersection point P based on the depth map and the location information of the intersection point new The depth information of Pnew The depth information of is used as the depth information of the target point P, and the depth map is filled to obtain the depth correction result of the target point.

[0103] For invalid depths in the depth map, such as a depth value of 0, the point with an infinite depth value can be set to the maximum pixel value and then participate in the above calculation.

[0104] Figure 4 A schematic diagram of processing a depth map of a mirror object according to an embodiment of the present application is shown. Figure 4 As shown, the photographer who takes the depth map marks the position of the mirror object based on the point cloud top view formed by the shooting.

[0105] After the annotation is completed, the annotation data and point cloud data are sent to the server. The server obtains the mirror position based on the annotation data and collects valid points within the range. For each valid point, it is determined whether it is visible or not. For invisible points, no correction is required. For visible points, the depth map is obtained, the depth map is filled, and the point cloud correction is completed.

[0106] In some embodiments of the present application, determining the intersection point of the first ray and the target object includes:

[0107] When the thickness information is 0, the intersection point is determined according to the intersection point of the first ray and the directed line segment;

[0108] When the thickness information is not 0, the surface vector of the target object is determined according to the directed line segment and the thickness information, and the intersection point is determined according to the intersection point of the first ray and the surface vector.

[0109] In an embodiment of the present application, for single-sided reflective objects such as mirrors and windows, their thickness will not affect the calculation. Therefore, in this case, the thickness information can be marked as 0 during annotation. At this time, the intersection of the first ray and the target object is also the intersection of the first ray and the directed line segment.

[0110] For objects with double-sided reflections such as glass partitions, the thickness information of the glass partitions is further marked after the standard directed line segment. Figure 5 A schematic diagram of a marked glass partition according to an embodiment of the present application is shown, such as Figure 5 As shown, P1 is the first point, P2 is the second point, and d is the thickness information. Based on the directed line segment P1P2 and the thickness information d, the normal of the target object is calculated to obtain the surface vector of the target object. and According to the direction of the first ray, the surface vector participating in the reflection is determined, and according to the intersection point between the first ray and the surface vector participating in the reflection, the corresponding intersection point position is determined.

[0111] The surface vector, also known as the normal, is calculated as follows:

[0112] v = P2–P1;

[0113] n1=(-v[1],v[0]) T ;

[0114] n2=(v[1],-v[0]) T ;

[0115] Among them, P1 is the coordinate of the first point, P2 is the coordinate of the second point, and n1 and n2 are the normals of the directed line segment P1P2.

[0116] The surface vector of the target object and for:

[0117]

[0118]

[0119] Figure 6 A schematic diagram of processing a depth map of a glass partition according to an embodiment of the present application is shown. Figure 4 As shown, the photographer who takes the depth map marks the position of the mirror object based on the point cloud top view formed by the shooting.

[0120] After the labeling is completed, the labeling data and point cloud data are sent to the server. The server obtains the position of the double-sided mirror based on the labeling data and processes it into Figure 4 The same mirror data. Then collect the valid points within the range, and judge whether each valid point is visible or not. For invisible points, no correction is required. For visible points, obtain the depth map, fill the depth map, and complete the point cloud correction.

[0121] In some embodiments of the present application, a point cloud data correction device is provided. Figure 7 FIG. 4 shows a structural block diagram of a point cloud data correction device according to an embodiment of the present application. Figure 7 As shown, the calibration device 700 includes:

[0122] A receiving module 702 is used to receive annotation input of initial point cloud data;

[0123] The annotation module 704 is used to annotate the target object in the initial point cloud data in response to the annotation input;

[0124] A determination module 706 is used to determine N target points in the initial point cloud data according to the position information of the target object, where N is a positive integer;

[0125] A determination module 708 is used to determine whether the target point is a visible point;

[0126] The correction module 710 is used to correct the depth information of the target point when the target point is a visible point.

[0127] In an embodiment of the present application, after point cloud data is acquired through depth acquisition equipment and image acquisition equipment, such as a TOF camera and an RGB camera, target objects with mirror properties, such as mirrors and windows, in the room corresponding to the point cloud data are annotated through the annotation input according to the annotation input of the photographer or user, and the depth information of the points nearby are corrected according to the corresponding position information of these target objects with mirror properties in the point cloud data, thereby eliminating the acquisition error of the depth information of the mirror object caused by the depth acquisition equipment and improving the accuracy of the depth information of the point cloud data.

[0128] In some embodiments of the present application, the determination module is further configured to:

[0129] According to the annotation input, determine the first point position, the second point position and thickness information;

[0130] Taking the first point as the starting point and the second point as the end point, determine the directed line segment;

[0131] The annotation module is also used to annotate the target object according to the directed line segments and thickness information.

[0132] The embodiment of the present application can improve the correction effect of the initial point cloud data and improve the accuracy of the corrected point cloud data by accurately marking the target object.

[0133] In some embodiments of the present application, the determination module is further configured to:

[0134] Determine the center point of the target object;

[0135] Determine the distance value between each point in the initial point cloud data and the central point;

[0136] The points whose distance values ​​are less than the threshold are determined as target points.

[0137] The embodiment of the present application includes only points whose distance to the target object is less than a threshold, that is, points near the target object, as points to be corrected, which can effectively reduce the amount of calculation during depth correction and improve correction efficiency.

[0138] In some embodiments of the present application, the determination module is further configured to:

[0139] Determine a corresponding first ray according to the position information of the target point, wherein the first ray passes through the target point and is related to a shooting direction of a camera that shoots depth information of the target point;

[0140] When the first ray intersects the target object, the target point is determined to be a visible point;

[0141] When the first ray does not intersect the target object, the target point is determined to be a non-visible point.

[0142] In the embodiment of the present application, after N target points that meet the conditions are determined according to the position of the target object, it is further determined whether these target points are visible points, that is, it is determined whether the target points should be "seen" by the user.

[0143] According to the shooting direction of the camera, the first ray xP is determined. The starting point of the first ray xP is camera x, the direction of the first ray satisfies the shooting direction of camera x, and the first ray passes through the target point P. If the first ray xP is aligned with the directed line segment P start P end Intersect, such as Figure 3 As shown, the target point P is determined to be a visible point, that is, the depth information of the target point P needs to be corrected.

[0144] If the first ray xP is aligned with the directed line segment P start P end If they do not intersect, it is determined that the target point P is a non-visible point, and there is no need to correct the depth information of the target point P.

[0145] In some embodiments of the present application, the determination module is further configured to:

[0146] Determine the intersection point of the first ray and the target object;

[0147] Determine the intersection point of the first ray and the target object;

[0148] The correction module is also used to correct the depth information of the target point according to the intersection point and the corresponding depth map information.

[0149] After the labeling is completed, the embodiment of the present application sends the labeling data and the point cloud data to the server. The server obtains the mirror position based on the labeling data, and collects valid points within the range. For each valid point, it is determined whether it is visible or not. For invisible points, no correction is required. For visible points, a depth map is obtained, the depth map is filled, and the correction of the point cloud is completed.

[0150] In some embodiments of the present application, the determination module is further configured to:

[0151] When the thickness information is 0, the intersection point is determined according to the intersection point of the first ray and the directed line segment;

[0152] When the thickness information is not 0, the surface vector of the target object is determined according to the directed line segment and the thickness information, and the intersection point is determined according to the intersection point of the first ray and the surface vector.

[0153] After the labeling is completed, the embodiment of the present application sends the labeling data and point cloud data to the server side, and the server side obtains the position of the double-sided mirror according to the labeling data and processes it into mirror surface data. After that, the valid points within the acquisition range are determined, and for each valid point, whether it is visible or not is determined. For invisible points, no correction is required. For visible points, a depth map is obtained, and the depth map is filled to complete the correction of the point cloud.

[0154] The point cloud data correction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0155] The point cloud data correction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0156] The point cloud data correction device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0157] Optionally, an embodiment of the present application further provides an electronic device, Figure 8 A structural block diagram of an electronic device according to an embodiment of the present application is shown. Figure 8As shown, the electronic device 800 includes a processor 802, a memory 804, and a program or instruction stored in the memory 804 and executable on the processor 802. When the program or instruction is executed by the processor 802, each process of the above-mentioned method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0158] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0159] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0160] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, a processor 910 and other components.

[0161] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0162] The processor 910 is used to receive annotation input of initial point cloud data;

[0163] In response to the annotation input, annotating a target object in the initial point cloud data;

[0164] According to the location information of the target object, N target points are determined in the initial point cloud data, where N is a positive integer;

[0165] Determine whether the target point is a visible point;

[0166] When the target point is a visible point, the depth information of the target point is corrected.

[0167] In an embodiment of the present application, after point cloud data is acquired through a depth acquisition device and an image acquisition device, such as a TOF camera and an RGB camera (Red Green Blue, red, green and blue three-channel color), the target objects with mirror properties, such as mirrors and windows, in the room corresponding to the point cloud data are marked through the annotation input according to the annotation input of the photographer or the user, and the depth information of the points nearby are corrected according to the corresponding position information of these target objects with mirror properties in the point cloud data, thereby eliminating the acquisition error of the depth information of the mirror object caused by the depth acquisition device and improving the accuracy of the depth information of the point cloud data.

[0168] Optionally, the processor 910 is further configured to determine the first point position, the second point position and thickness information according to the annotation input;

[0169] Taking the first point as the starting point and the second point as the end point, determine the directed line segment;

[0170] Label the target object based on the directed line segments and thickness information.

[0171] The embodiment of the present application can improve the correction effect of the initial point cloud data and improve the accuracy of the corrected point cloud data by accurately marking the target object.

[0172] Optionally, the processor 910 is further configured to determine a center point of the target object;

[0173] Determine the distance value between each point in the initial point cloud data and the central point;

[0174] The points whose distance values ​​are less than the threshold are determined as target points.

[0175] The embodiment of the present application includes only points whose distance to the target object is less than a threshold, that is, points near the target object, as points to be corrected, which can effectively reduce the amount of calculation during depth correction and improve correction efficiency.

[0176] Optionally, the processor 910 is further configured to determine a corresponding first ray according to the position information of the target point, wherein the first ray passes through the target point, and a direction of the first ray is related to a shooting direction of a camera that shoots depth information of the target point;

[0177] When the first ray intersects the target object, the target point is determined to be a visible point;

[0178] When the first ray does not intersect the target object, the target point is determined to be a non-visible point.

[0179] In the embodiment of the present application, after N target points that meet the conditions are determined according to the position of the target object, it is further determined whether these target points are visible points, that is, it is determined whether the target points should be "seen" by the user.

[0180] According to the shooting direction of the camera, the first ray xP is determined. The starting point of the first ray xP is camera x, the direction of the first ray satisfies the shooting direction of camera x, and the first ray passes through the target point P. If the first ray xP is aligned with the directed line segment P start P end Intersect, such as Figure 3 As shown, the target point P is determined to be a visible point, that is, the depth information of the target point P needs to be corrected.

[0181] If the first ray xP is aligned with the directed line segment P start P end If they do not intersect, it is determined that the target point P is a non-visible point, and there is no need to correct the depth information of the target point P.

[0182] Optionally, the processor 910 is further configured to determine a point of intersection between the first ray and the target object;

[0183] According to the intersection point and the corresponding depth map information, the depth information of the target point is corrected.

[0184] After the labeling is completed, the embodiment of the present application sends the labeling data and the point cloud data to the server. The server obtains the mirror position based on the labeling data, and collects valid points within the range. For each valid point, it is determined whether it is visible or not. For invisible points, no correction is required. For visible points, a depth map is obtained, the depth map is filled, and the correction of the point cloud is completed.

[0185] Optionally, the processor 910 is further configured to determine the intersection point according to the intersection point of the first ray and the directed line segment when the thickness information is 0;

[0186] When the thickness information is not 0, the surface vector of the target object is determined according to the directed line segment and the thickness information, and the intersection point is determined according to the intersection point of the first ray and the surface vector.

[0187] After the labeling is completed, the embodiment of the present application sends the labeling data and point cloud data to the server side, and the server side obtains the position of the double-sided mirror according to the labeling data and processes it into mirror surface data. After that, the valid points within the acquisition range are determined, and for each valid point, whether it is visible or not is determined. For invisible points, no correction is required. For visible points, a depth map is obtained, and the depth map is filled to complete the correction of the point cloud.

[0188] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0189] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0190] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0191] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0192] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0194] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0195] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0196] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0197] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0198] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for correcting point cloud data, characterized in that: include: Receiving annotation input of initial point cloud data; In response to the annotation input, annotating a target object in the initial point cloud data; According to the position information of the target object, N target points are determined in the initial point cloud data, where N is a positive integer; Determining whether the target point is a visible point; When the target point is a visible point, correcting the depth information of the target point; The step of labeling a target object in the initial point cloud data in response to the labeling input includes: Determine the first point position, the second point position and thickness information according to the annotation input; Determine a directed line segment with the first point as a starting point and the second point as an end point; The target object is marked according to the directed line segment and the thickness information.

2. The calibration method according to claim 1, characterized in that: Determining N target points in the initial point cloud data according to the position information of the target object includes: Determine the center point of the target object; Determine the distance value between each point in the initial point cloud data and the central point; The point whose distance value is less than the threshold is determined as the target point.

3. The calibration method according to claim 1, characterized in that: The determining whether the target point is a visible point includes: Determine a corresponding first ray according to the position information of the target point, wherein the first ray passes through the target point, and a direction of the first ray is related to a shooting direction of a camera that shoots depth information of the target point; When the first ray intersects the target object, determining the target point as a visible point; When the first ray does not intersect the target object, the target point is determined to be a non-visible point.

4. The calibration method according to claim 3, characterized in that: The correcting the depth information of the target point includes: Determine the intersection point of the first ray and the target object; The depth information of the target point is corrected according to the intersection point and the corresponding depth map information.

5. The calibration method according to claim 4, characterized in that: The determining the intersection point of the first ray and the target object includes: When the thickness information is 0, determining the intersection point according to the intersection point of the first ray and the directed line segment; When the thickness information is not 0, determining the surface vector of the target object according to the directed line segment and the thickness information, and determining the position of the intersection point according to the intersection point of the first ray and the surface vector; Wherein, the normal of the target object is calculated according to the directed line segment and the thickness information, so as to obtain the surface vector of the target object.

6. A point cloud data correction device, characterized in that: include: A receiving module, used for receiving annotation input of initial point cloud data; a labeling module, configured to label a target object in the initial point cloud data in response to the labeling input; A determination module, configured to determine N target points in the initial point cloud data according to the position information of the target object, where N is a positive integer; A judgment module, used to judge whether the target point is a visible point; A correction module, used for correcting the depth information of the target point when the target point is a visible point; The determining module is further used for: Determine the first point position, the second point position and thickness information according to the annotation input; Determine a directed line segment with the first point as a starting point and the second point as an end point; The target object is marked according to the directed line segment and the thickness information.

7. A point cloud data correction device, characterized in that: include: Memory, used to store programs or instructions; A processor, configured to implement the steps of the correction method according to any one of claims 1 to 5 when executing the program or instruction.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the correction method according to any one of claims 1 to 5 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the correction method according to any one of claims 1 to 5 are implemented.

Images