Shooting point location prediction method and device, equipment and medium
The method predicts optimal shooting points in panoramic image capture by extracting plane and door information, addressing inconsistent shooting practices and improving efficiency and quality in panoramic image construction.
Patent Information
- Application Number
- CN202411053273.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
AI Technical Summary
The selection of shooting points for panoramic images is often random and inconsistent among users, leading to slow processing speeds, longer shooting times, and poor registration results due to dense or isolated points, affecting the efficiency and quality of panoramic image construction.
A method and device for predicting shooting points using panoramic images with depth information to extract plane and door information, allowing for the prediction of optimal next shooting locations based on valid plane and door point information, ensuring continuity and efficiency in panoramic image capture.
The method guides users to shoot panoramic images efficiently, reducing redundant content and improving the quality of panoramic image construction by selecting optimal shooting points, thus enhancing the shooting experience and reducing user workload.
Smart Images

Figure CN119071638A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for predicting shooting points. Background Art
[0002] With the continuous development of virtual reality technology, panoramas are widely used in house rental, hotel homestay, home decoration, various exhibitions and other fields. Compared with ordinary single-perspective images, panoramas can provide broader field of view information, allowing people to restore the scene to a certain extent, and users can get a better sense of immersion when watching. Therefore, panorama previews are becoming more and more popular and sought after by the public.
[0003] In the fields of 3D reconstruction and VR (Virtual Reality), more and more panoramic data are used to construct subsequent tasks. However, in the process of selecting shooting points, different people have different ideas. They may shoot particularly dense points, resulting in slow processing speed and longer shooting time. They may also shoot some isolated points, which will affect the subsequent automatic registration algorithm and make the registration result very poor. There will also be various other uncontrollable situations that affect the effect and efficiency of panoramic image construction and increase the workload of image shooting users. Therefore, there is a need for a solution that can guide users to correctly shoot panoramic images. Summary of the invention
[0004] Various aspects of the present application provide shooting point prediction methods, devices, equipment and media to solve the technical problem of poor panoramic image shooting effects.
[0005] In a first aspect, an exemplary embodiment of the present application provides a shooting point prediction method, which is applied to a server side, and the method includes:
[0006] Acquire a panoramic image including depth information captured at a first shooting point; the panoramic image includes a plane image and a door image;
[0007] Extracting scene information based on the panoramic image to obtain door point information and plane information;
[0008] extracting at least one valid plane from the panoramic image using the plane information;
[0009] The effective plane and / or the door point information is used to predict a next shooting point that meets the screening condition from the panoramic image.
[0010] In a second aspect, an exemplary embodiment of the present application provides a shooting point prediction device, the device comprising:
[0011] An acquisition module, used for acquiring a panoramic image including depth information obtained by shooting at a first shooting point; the panoramic image includes a plane image and a door image;
[0012] An extraction module, used to extract scene information based on the panoramic image to obtain door point information and plane information;
[0013] The extraction module is further used to extract at least one valid plane from the panoramic image using the plane information;
[0014] A prediction module is used to predict a next shooting point that meets the screening conditions from the panoramic image by using the effective plane and / or the door point information.
[0015] In a third aspect, an exemplary embodiment of the present application further provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program for executing the steps in the above-mentioned electronic contract processing method.
[0016] In a fourth aspect, an exemplary embodiment of the present application provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the steps in the above-mentioned electronic contract processing method.
[0017] In this embodiment, when shooting in a house environment, since there are many rooms in the house, it is necessary to shoot comprehensively from multiple different positions to generate a complete panoramic image. When shooting starts, the camera will first be placed at the first shooting point, and the camera will be used to collect a part of the panoramic image containing depth information. The panoramic image contains a plane image and a door image. Then, scene information is extracted from the panoramic image to obtain door point information and plane information. After obtaining the plane information, it is necessary to further identify which planes are included in the panoramic image, and extract at least one valid plane from it. Using the valid plane and / or door point information, a point is selected from the panoramic image that has been shot as the next shooting point. The user only needs to move the camera to the predicted next shooting point to shoot the second panoramic image, which can ensure the continuity of the shot image, and will not shoot too much repeated image content. It can meet the needs of multi-point continuous shooting of generating panoramic images, and better guide the shooting user to shoot continuously and efficiently to complete the panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0019] Figure 1 A schematic diagram of a process flow of a shooting point prediction method provided in an embodiment of the present application;
[0020] Figure 2 A schematic diagram of establishing a coordinate system and candidate points for illustrating an embodiment of the present application;
[0021] Figure 3 A schematic diagram of a shooting point prediction solution illustrated in an embodiment of the present application;
[0022] Figure 4 A schematic diagram of the structure of a shooting point prediction device provided in an embodiment of the present application;
[0023] Figure 5 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0025] With the continuous development of virtual reality technology, panoramas are widely used in house rental, hotel homestay, home decoration, various exhibitions and other fields. Compared with ordinary single-perspective images, panoramas can provide broader field of view information, allowing people to restore the scene to a certain extent. Therefore, panorama previews are becoming more and more popular and sought after by the public.
[0026] In the fields of 3D reconstruction and VR (Virtual Reality), more and more panoramic data are used to construct subsequent tasks. However, in the process of selecting shooting points, different people have different ideas and the shooting points are more random. Therefore, there may be points that are particularly densely shot, resulting in slow processing speed and longer shooting time. Some isolated points may also be shot, which will affect the subsequent automatic registration algorithm, making the registration result very poor, as well as various other uncontrollable situations. Therefore, in order to ensure the quality of shooting and reduce the burden on shooting users as much as possible, a method to guide users to shoot correctly is needed.
[0027] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0028] Figure 1This is a flow chart of the method for predicting shooting points provided in the embodiment of the present application. Figure 1 As shown, the shooting point prediction method provided in the embodiment of the present application can be run on a server side, which can be a local server or a cloud server. The method specifically includes the following steps:
[0029] Step 101: Acquire a panoramic image including depth information captured at a first shooting point; the panoramic image includes a plane image and a door image.
[0030] Step 102: extract scene information based on the panoramic image to obtain door point information and plane information.
[0031] Step 103: Utilize the plane information to extract at least one valid plane from the panoramic image.
[0032] Step 104: using the effective plane and / or the door point information, predicting a next shooting point that meets the screening condition from the panoramic image.
[0033] When shooting a video of a house, in order to obtain a complete video of the house, it is necessary to shoot at multiple points in the house, and then perform image synthesis to obtain a panoramic image that can present the complete image information of the house. Therefore, the selection of shooting points is crucial during the shooting process.
[0034] In the present application, after the user obtains the first panoramic image through shooting, the user will analyze the first panoramic image obtained through shooting to identify the plane image and the door image in the first panoramic image.
[0035] The plane image mentioned here can be a wall image, a ground image, a roof image and other architectural images in a house, or it can be a furniture image (for example, a bed image), a home appliance image (for example, an air conditioner cabinet image, a refrigerator image), or it can be various decoration images in the house (for example, a wall painting image).
[0036] The door images mentioned here refer to various doors in the captured house, including the entrance door and the doors of each room (such as bedroom door, kitchen door, study door, bathroom door, etc.). When extracting and identifying door images, it is necessary to distinguish between the entrance door and the room door, because the outside of the entrance door (that is, the outside of the house) does not need to be captured. In other words, when predicting the shooting points, the points outside the entrance door should be excluded or blocked.
[0037] As an optional solution, a deep learning model can be used to identify the door image contained in the first panoramic image. For example, in order to enable the deep model to better recognize the door image, it is first necessary to collect a large number of panoramic images of the house as a data set. These images can come from a public data set or be taken with professional equipment. The collected images are labeled to mark the location and category of all doors in the image (such as single-opening doors, double-opening doors, sliding doors, fire doors, anti-theft doors, entrance doors, etc.). This is usually done using a bounding box or a pixel-level segmentation mask. Generally speaking, the entrance door is a fire door or an anti-theft door. Therefore, it is necessary to distinguish between marked entrance doors and non-entry doors.
[0038] If the panoramic image is too large, you can crop it into smaller image blocks to reduce the amount of computation for the model. Increase the diversity of the data set and improve the generalization ability of the model by rotating, scaling, flipping, color transformation, etc. For the detection task of the gate image, you can choose a target detection model or a semantic segmentation model based on deep learning. For example, choose Faster R-CNN: a classic two-stage target detection model that first generates candidate regions and then performs classification and position regression. Or choose YOLO (such as YOLOv4, YOLOv5): a single-stage target detection model that directly predicts the category and location of the target at a faster speed. For example, you can choose Mask R-CNN as a semantic segmentation model: Based on Faster R-CNN, a branch is added to predict the segmentation mask, which can perform target detection and semantic segmentation at the same time. You can also choose U-Net: a model commonly used in medical image segmentation, which can also be used for pixel-level segmentation of gate images.
[0039] According to the selected model, configure the corresponding network architecture and hyperparameters (such as learning rate, batch size, number of iterations, etc.). Load the preprocessed dataset into the training environment. Input the image into the model and get the prediction result through the forward propagation of the model. Calculate the loss (such as cross entropy loss, smooth L1 loss, etc.) based on the labeled data and the prediction result. Update the weight of the model through optimization algorithms such as gradient descent to minimize the loss. Repeat the above process until the performance of the model on the validation set reaches a satisfactory level or reaches the preset number of iterations. Deploy the trained model to the actual application, and automatically identify and mark the location and category of the door by processing the panoramic image of the house in real time.
[0040] When extracting scene information through panoramic images, you first need to define a world coordinate system and a camera coordinate system. The world coordinate system is used to represent the spatial position of the point cloud, while the camera coordinate system is used to describe the position and direction of the camera. For each pixel in the depth map, the coordinates of the pixel in three-dimensional space are calculated based on its position in the image and the corresponding depth value. This process involves the transformation from image coordinates to camera coordinates and then to world coordinates. Assuming that the position of a pixel in the image coordinate system is (u, v) and the corresponding depth value is d, the coordinates Xc, Yc, and Zc in the camera coordinate system can be calculated according to the following formula (taking orthogonal projection as an example):
[0041] X c =(uu 0 )·d / f x
[0042] Yc=(v-v0)·d / fyY c =(vv 0 )·d / f y
[0043] Zc=dZ c =d
[0044] where u0,v0u 0 ,v 0 are the coordinates of the center point of the image, fx,fyf x ,f y is the focal length.
[0045] Remove noise points, such as through statistical filtering, distance filtering, etc. Reduce the number of points to increase processing speed, such as uniform downsampling, voxel grid downsampling, etc.
[0046] When detecting the ground plane, RANSAC (Random Sampling Consensus Algorithm) or other plane fitting methods may also be used to detect and fit the ground plane.
[0047] When identifying planes other than the ground plane, ground points can be removed by comparing the distance of each point to the ground plane, leaving only non-ground points.
[0048] When extracting planes that are higher than the camera height, determine a threshold that corresponds to the camera height. Traverse the remaining non-ground plane points and keep those with heights higher than the threshold. Finally, the extracted point cloud data can be visualized to facilitate inspection and analysis of the results.
[0049] When extracting plane information, it is necessary to further identify and mark the door point information. When identifying the door, the intelligent model can be used for image recognition. When marking the door point information, the correspondence between the door in the panoramic image and the point cloud image can be established based on the previous recognition results of the door in the panoramic image, the door point cloud image can be determined in the point cloud image, and at least 2 points can be marked on the plane where the door is located. In the above manner, the door point information and plane information in the panoramic image are obtained.
[0050] After the door information and plane information are extracted through the above scheme, when predicting the next shooting point, the effective plane information can be used to predict the next shooting point from the candidate points, or the door point information can be used to predict the next shooting point from the candidate points. In practical applications, if the next shooting point is predicted using both the effective plane information and the door point information, it is necessary to combine the results of the two methods for screening. Alternatively, if no suitable predicted point is obtained using the above two methods, a point can be selected as the next shooting point in an open position in the first panoramic image (without furniture, home appliances, etc. and indoors, with no obstacles between the first shooting point and the point). This allows the newly selected point to be within the same field of view as the first shooting point. Of course, when selecting, a point with a moderate distance from the first shooting point (for example, between 0.5 meters and 3 meters) may be selected.
[0051] Through the above method, the prediction result of the next point can be provided to the user within the current field of view. That is, when the user takes a panoramic image indoors according to the predicted point, the shooting effect can be effectively improved. It should be noted that in the present application scheme, by identifying the planes (planes other than the ground and the roof) and doors in the first panoramic image, shooting points can be selected near these planes or doors to avoid the next shooting point being too close or too far from the first shooting point. At the same time, it can also ensure that the distance between the next shooting point and the plane or door is appropriate, and will not affect the shooting field of view and the shooting effect (for example, the field of view will not be filled with walls because the distance to the wall is too close).
[0052] In one or more embodiments of the present application, the use of the plane information to determine the effective plane from the panoramic image includes: using the depth information contained in the panoramic image to obtain a point cloud image; determining a first normal vector of the point cloud image; using the first normal vector to segment the point cloud image to obtain at least one effective plane; filtering out point cloud information in the point cloud image that is higher than the camera height at the first shooting point; using the first normal vector and the point cloud projection to determine the ground, and filtering out the ground; after filtering out the planes higher than the camera and filtering out the ground, determining the maximum vertical plane from the at least one effective plane having a vertical axis perpendicular to the normal.
[0053] Using the conversion relationship between panoramic images and point clouds, the depth map pixel coordinates are converted into point cloud data in three-dimensional space. The specific steps are as follows:
[0054] For each pixel in the depth image depth_pixel(x,y), where x and y are the pixel coordinates, depth_co ls and depth_rows are the width and height of the depth map respectively.
[0055] Calculate longitude l on and latitude l at:
[0056] fl oat l on=(0.5f-depth_p ixe lx / depth_co ls)*2*M_PI;
[0057] fl oat l at=(0.5f-depth_p ixe ly / depth_rows)*M_PI;
[0058] Calculate the x, y, and z coordinates in the spherical coordinate system (here we assume that the origin is at the center of the camera and the camera is facing the negative direction of the Z axis):
[0059] fl oat cos_l at=std::cos(l at);
[0060] fl oat x=cos_l at*std::sin(l on);
[0061] float y = std::sin(l at);
[0062] float z=-cos_l at*std::cos(l on); / / pay attention to the adjustment of Z axis direction
[0063] The actual 3D coordinates are calculated based on the actual depth value d (obtained from the depth map) and the scale factor scale:
[0064] float = point3d_depth_value * scale; / / point3d_depth_value is the depth value of the point in the depth map
[0065] fl oat scale_x=x*d;
[0066] fl oat scale_y=y*d;
[0067] fl oat sca led_z = z*d;
[0068] For each point or local area in the point cloud, use PCA or other methods to calculate its normal vector.
[0069] It may also be necessary to rotate or transform the point cloud to ensure that the vertical direction of the point cloud data (such as the Z axis) is consistent with the gravity direction of the real world. The specific method may depend on the camera's attitude information and calibration parameters.
[0070] The RANSAC algorithm is used to segment and fit the point cloud to identify multiple planes in the scene. During the segmentation and fitting process, planes with angle differences less than 10 degrees are merged into one plane. The normal vector information is used to quickly iterate and extract the plane.
[0071] Extract direction information (i.e., the normal vector of the plane) from the plane obtained by RANSAC segmentation. Filter planes higher than the camera height: Use the height information of the plane to filter out planes higher than the camera installation height. Determine whether it is the ground based on the angle between the normal vector of the plane and the normal vector of the ground. Alternatively, project the point cloud onto a specific straight line (such as the Z axis), determine the approximate height of the ground by analyzing the distribution density of the points, and further confirm the ground in combination with the normal information. After filtering out the ground and the roof planes higher than the camera, select the plane with the largest angle between the normal vector and the vertical axis (such as the Y axis, assuming that the X axis is horizontal and the Z axis is vertical) as a candidate.
[0072] Among the identified multiple candidate planes, the largest vertical plane is determined according to the area or height information. The above scheme can be used to extract effective planes (especially the largest vertical plane) from the depth information of the panoramic image.
[0073] In one or more embodiments of the present application, the effective plane is used to select a next shooting point that meets the screening conditions from the panoramic image, including: establishing multiple first candidate points in a two-dimensional coordinate system established based on the maximum vertical plane; wherein the horizontal axis in the two-dimensional coordinate system is parallel to the maximum vertical plane, and the vertical axis is perpendicular to the maximum vertical plane. According to the positional relationship between the first candidate point and the effective plane, unqualified candidate points in the first candidate points are filtered out, and the reverse effective plane is filtered out. Multiple candidate points that are not filtered out are marked as second candidate points, and the first target candidate point is determined as the next shooting point according to the distance between the second candidate point and the effective plane that is not filtered out.
[0074] After finding the maximum vertical plane from the first panoramic image in the above manner, a coordinate system can be established based on the maximum numerical plane, which is generally a two-dimensional coordinate system. This coordinate system is established to simplify the subsequent point analysis and selection process. The direction parallel to the maximum vertical plane is selected as the horizontal axis. This means that moving on this axis will not significantly change the distance from the point to the maximum vertical plane. The direction perpendicular to the maximum vertical plane is selected as the vertical axis. Movement on this axis will directly affect the distance from the point to the plane and is a key dimension in subsequent analysis.
[0075] like Figure 2 Schematic diagram of establishing a coordinate system and candidate points for example in the embodiment of the present application. As an optional scheme, in a two-dimensional coordinate system, including an X-axis and a Y-axis, multiple first candidate points can be marked based on certain rules or strategies. These points may represent potential shooting positions for subsequent analysis and screening. For example, several candidate points can be set on the horizontal and vertical coordinate axes, respectively, wherein 2 points are set in the positive and negative directions of the horizontal axis (i.e., the X-axis), that is, there are a total of 4 points on the horizontal axis as candidate points. In the same way, 4 points are also set on the vertical axis (i.e., the Y-axis) as candidate points. For example, a candidate point position is set at a position of plus or minus 1 meter and plus or minus 2 meters from the origin of the two-dimensional coordinate system. Of course, the specific position can be set according to actual needs. If the room space is large, a position of 1.5 meters or 3 meters can be set. However, it should be noted that if the distance is too far, the image display may not be clear. Therefore, it is generally necessary to set the candidate point position within 5 meters.
[0076] In addition, candidate points can also be generated based on specific algorithms (such as grid division, random point scattering, etc.) or according to actual application requirements. The number of candidate points should be sufficient to cover possible shooting positions, while also considering the limitations of computational efficiency and actual application scenarios.
[0077] After the first candidate points are generated, it is necessary to filter out unqualified candidate points based on their positional relationship with the valid plane.
[0078] Position relationship with valid planes: Analyze the distance and angle of each candidate point to all valid planes that have not been filtered. If the candidate point is on the back side of a plane (i.e., the normal vector points in the opposite direction of the line connecting the candidate point to the plane), or is too close / too far from some key planes, it may be considered unqualified.
[0079] Filtering reverse valid planes: While filtering candidate points, it is also necessary to check and filter out those valid planes whose normal vectors are opposite to the main observation direction (such as camera viewing angle). These planes may not produce valuable information in subsequent analysis and shooting.
[0080] After the above filtering steps, the remaining candidate points are marked as second candidate points. These points represent potential and reasonable shooting positions. It is necessary to select the best one from the second candidate points as the first target candidate point, that is, the next shooting point.
[0081] Calculate the distance from each second candidate point to all valid planes that have not been filtered, and consider the weighted sum of these distances or some optimization criteria (such as minimizing the maximum distance, maximizing the minimum distance, etc.). According to the results of the distance analysis, select the candidate point that best meets the requirements as the first target candidate point. This selection should be based on the needs and limitations of the actual application, such as shooting efficiency, field of view coverage, shooting difficulty, etc. The next shooting point can be effectively determined in the two-dimensional coordinate system based on the maximum vertical plane and valid plane information.
[0082] In one or more embodiments of the present application, filtering out unqualified candidate points in the first candidate points and filtering out the opposite valid plane according to the positional relationship between the first candidate point and the valid plane includes: constructing a candidate straight line where the first shooting point and any one of the first candidate points are located; if there is an intersection between the candidate straight line and any one of the valid planes, filtering out the candidate point, and using the candidate point that has not been filtered out as the second candidate point; when the angle between the candidate straight line and the effective normal of any one of the valid planes is greater than an angle threshold, filtering out the valid plane in the opposite direction of the candidate straight line.
[0083] As mentioned above, the multiple first candidate points are points on the X-axis and the Y-axis, with 4 points on each axis, 2 of which are in the positive direction and 2 in the negative direction, for a total of 8 first candidate points.
[0084] For each candidate point, a straight line is constructed starting from the reference point (that is, the first shooting point) along the X-axis or Y-axis direction (depending on the axis where the candidate point is located), which is called the candidate line.
[0085] Since the candidate points are located on the X-axis and Y-axis, their geometric relationship with the valid plane needs to be specially considered.
[0086] For example, the intersection point needs to be checked. Although the candidate points are on the axis, their candidate lines may intersect with the valid plane. If the candidate line has an intersection with any valid plane, and the intersection is between the candidate point and the reference point (that is, along the line direction, the intersection does not exceed the range of the candidate point), then the candidate point may be blocked or interfered by the plane, and should be filtered out.
[0087] Since the candidate points are located on the axis, they may intersect with the valid plane only in certain specific cases (such as when the plane is parallel or almost parallel to the axis). In most cases, the points on the axis may not directly cause the line to intersect the plane, but their candidate line directions (X-axis or Y-axis) may form unfavorable angles with the effective normals of some planes. For example, in the shooting of a house scene, a candidate point in the established coordinate system is on one side of the wall, and the reference point is on the other side of the wall. Obviously, this candidate point is inappropriate because the wall blocks the two shooting points and a panoramic image cannot be established.
[0088] Angle judgment: For each candidate point's candidate line, it is necessary to check the angle between it and the valid normal of all valid planes. If the angle is greater than a certain set angle threshold (usually this threshold is determined based on the shooting requirements and application scenarios), the visibility of the corresponding plane may be affected when shooting from the candidate point. When judging the angle, you can establish a normal on the front of the valid plane, and then judge the angle between the candidate line and the normal. If the angle is less than 90 degrees, it means that the normal and the candidate point are on the same side of the valid plane, that is, the reference point (first shooting point) and the candidate point are on the same side of the valid plane. In this case, the candidate point can be retained as the second candidate point. If the angle is greater than 90 degrees, it means that the normal and the candidate point are on different sides of the valid plane, that is, the reference point (first shooting point) and the candidate point are on both sides of the valid plane. In this case, the candidate point cannot be retained as the second candidate point.
[0089] Since the candidate points are located on the axis and the main focus is on the intersection of the line and the plane, angle judgment may not be the only criterion for directly filtering candidate points. Instead, angle judgment can be used as auxiliary information to evaluate the quality of candidate points in subsequent analysis.
[0090] Although the candidate points are on the axis, the direct opposition between their candidate line directions and the effective normal direction of the valid plane may not be applicable (because the line is along the axis). However, if the normal direction of a valid plane forms a very unfavorable angle (close to or greater than 180 degrees) with the expected shooting direction of the candidate line (i.e., along the positive or negative direction of the axis), it can still be considered to be given a lower evaluation or weight in subsequent analysis.
[0091] After the above analysis and filtering steps, those candidate points that do not intersect with the effective plane (or the intersection does not affect the shooting) and are relatively favorable in the shooting direction are retained as the second candidate points. These points represent more potential shooting positions because they have a more favorable geometric relationship with the effective plane and are located on the axis, which is convenient for subsequent shooting operations. In actual applications, it may be necessary to further adjust and optimize this process according to specific shooting requirements, application scenarios, and other factors in the scene.
[0092] In one or more embodiments of the present application, using the door point information to predict a next shooting point that meets the screening condition from the panoramic image includes:
[0093] Two door points on the same door image plane in the panoramic image are used as the door point information, and a door straight line formed by the two door points;
[0094] Constructing a candidate straight line where the candidate point and the door point are located;
[0095] The corresponding candidate point whose angle between the candidate straight line and the gate straight line is not greater than the angle threshold is used as the third candidate point;
[0096] According to the comparison result of the candidate distance between the third candidate point and the door point, the second target candidate point is determined as the next shooting point. (If there are multiple third candidate points in the same direction, the third candidate points greater than the distance threshold are eliminated according to the distance between the door line and the first shooting point).
[0097] In the panoramic image, the image area representing the door is first identified and two door points are determined. These two points are usually selected as the diagonal points of the door or two points with significant features to define the position and direction of the door in the image. The two door points are connected to form a door straight line, which indicates the main direction of the door.
[0098] As mentioned above, there are 8 first candidate points on the X-axis and Y-axis. There are 4 points on the X-axis, 2 of which are in the positive direction (such as +X direction) and 2 are in the negative direction (such as -X direction); there are also 4 points on the Y-axis, 2 in the positive direction (such as +Y direction) and 2 in the negative direction (such as -Y direction). These candidate points are used for subsequent shooting point selection.
[0099] Next, determine the angle between the candidate point and the gate line. For each first candidate point, construct a candidate line starting from the point and along the axis (X-axis or Y-axis) where it is located. Then, calculate the angle between these candidate lines and the gate line. Set an angle threshold to filter candidate lines that are close to the gate line in direction. Candidate lines with angles not greater than this threshold are considered to be consistent with the gate line in direction.
[0100] There are many ways to determine whether the door and the candidate point are suitable. You can directly calculate the angle between the plane where the door is located (that is, the door line) and the candidate line. You can also establish the angle between the normal of the door plane (that is, the perpendicular line of the door line) and the candidate line. The principle of filtering the door plane or door line here is similar to the principle of filtering the valid plane, but it should be noted that since the door can be opened and closed, even if the reference point and the candidate point are on different sides of the door, this candidate point should not be filtered out.
[0101] Those candidate points whose angles with the door line are not greater than the angle threshold and their corresponding candidate lines are retained, and these candidate points are marked as third candidate points. At this time, the third candidate points may include multiple points on the X-axis and the Y-axis. If there are multiple third candidate points in the same direction (such as +X, -X, +Y, -Y), and the selection range needs to be further narrowed, a distance threshold can be set. This distance threshold can be the distance between the third candidate point and the first shooting point, or it can be the projection distance between the third candidate point and the door line on a reference point. According to this distance threshold, those candidate points that are too far or too close are eliminated. In some cases, it may be more inclined to select candidate points in a specific direction (such as the positive direction) as shooting points. This can be achieved by giving higher weights to candidate points in a specific direction when comparing candidate distances.
[0102] After the above screening, one or a few optimal third candidate points are finally selected as the second target candidate points, i.e., the next shooting point. This selection can be based on the comparison results of the candidate distances, direction preferences, and possible other factors (such as obstacles, light conditions, etc.). It should be noted that the parameters such as the angle threshold, distance threshold, and specific screening strategies in the above process may need to be adjusted and optimized according to the actual scenario and needs.
[0103] In one or more embodiments of the present application, the second target candidate point and the third target candidate point are screened, and it is determined whether the second target candidate point and the third target candidate point are in the same direction; if so, the third target candidate point is selected as the fourth target candidate point; if not, the first target candidate point and the second target candidate point in different directions are both selected as the fourth target candidate point.
[0104] In automated shooting or navigation tasks, it is critical to consider the direction when selecting shooting points. The consistency of direction helps maintain the coherence of the shooting sequence and reduce unnecessary turns or movements. Therefore, judging whether the second target candidate point and the third target candidate point are in the same direction is crucial to determine the final shooting point.
[0105] If the second target candidate point is in the same direction as the third target candidate point, this usually means that they point to potential shooting targets along similar paths or directions. In this case, it may be reasonable to select the third target candidate point as the fourth target candidate point (i.e., the final shooting point position) because the door can be opened and closed, which can provide a better shooting angle, a closer distance or other advantages.
[0106] If they are not in the same direction, more factors may need to be considered to determine the final shooting point. In this case, it is a conservative but comprehensive choice to use both the second target candidate point and the third target candidate point as the fourth target candidate point. Doing so ensures that there are multiple options to consider in subsequent operations, especially in situations where quick response or decision-making is required.
[0107] In practical applications, in addition to direction and distance, other factors may need to be considered to determine the final shooting point, such as light conditions, obstacle distribution, visibility of the shooting target, battery life (for mobile devices), time constraints, etc.
[0108] In addition, if the system has sufficient intelligence and computing power, it can also use more complex algorithms (such as path planning, optimization algorithms, etc.) to comprehensively consider multiple factors and automatically select the best shooting point.
[0109] In some cases, user input and feedback are also important factors in determining the shooting point. For example, the user may specify the shooting direction, angle, or position according to their needs or preferences. The system can also provide real-time visual feedback (such as preview images, augmented reality views, etc.) to help users better understand the effects of different shooting points and make more informed choices.
[0110] In one or more embodiments of the present application, the corresponding candidate point whose angle between the candidate straight line and the gate straight line is not greater than the angle threshold is taken as the third candidate point, including: when it is judged that the candidate straight line and the gate straight line are perpendicular according to the angle and the angle threshold, marking the point in front of the gate and the point behind the gate based on the perpendicular bisector of the gate straight line as the third candidate point; when it is judged that the candidate straight line and the gate straight line are parallel according to the angle and the angle threshold, marking the point in front of the gate based on the perpendicular bisector of the gate straight line as the third candidate point.
[0111] The setting of the angle threshold is crucial for accurately judging the relationship between the candidate line and the gate line. This threshold should be set according to the needs and accuracy requirements of the actual application scenario. A smaller threshold can improve the accuracy of the judgment, but may increase the risk of misjudgment; a larger threshold can reduce misjudgment, but may reduce the accuracy of the judgment.
[0112] When the angle between the candidate line and the door line is within the set angle threshold range (for example, if it is greater than 70 degrees, it is considered vertical, and this threshold can be set to 70 degrees), and it is judged to be a vertical relationship, the point in front of the door and the point behind the door are marked as the third candidate point based on the perpendicular median of the door line. This is because the perpendicular relationship means that the candidate line intersects the door line at one point (or is very close to the intersection), and the perpendicular median represents the extension of this intersection (or the area near the intersection) in the vertical direction. The point in front of the door and the point behind the door are respectively located on the perpendicular median, and are respectively located in front of and behind the door, providing different viewing angles and distance options for shooting.
[0113] When the angle between the candidate line and the door line is within the set angle threshold range (for example, if it is less than 20 degrees, it is considered parallel, and this threshold can be set to 20 degrees), and it is judged to be a parallel relationship, the point in front of the door is marked as the third candidate point based on the perpendicular bisector of the door line. The logic here may be slightly different because the parallel relationship does not directly point to one side of the door (such as the front or the back). However, in some cases, we may want to select shooting points along the direction parallel to the door in order to capture a wider field of view or a specific scene layout. In this case, the selection of the point in front of the door may be based on other factors (such as visibility, lighting conditions, obstacle avoidance, etc.), and may be located at the intersection of the perpendicular bisector and a specific direction (such as a direction perpendicular to the candidate line but pointing to the shooting target).
[0114] The specific definitions of the points in front of the door and behind the door may vary depending on the application scenario. In general, the points in front of the door can be considered to be points within a certain distance and angle range directly in front of the door, while the points behind the door are located in the opposite direction of the door. However, in actual operation, the precise locations of these points may need to be determined according to specific shooting requirements and scene layout.
[0115] When selecting the third candidate point, in addition to the angle and angle threshold, other factors need to be considered, such as obstacle avoidance, light conditions, visibility of the target, performance limitations of the shooting equipment, etc. These factors may affect the quality and feasibility of the final shooting effect.
[0116] The above strategy should have certain flexibility and scalability so that it can adapt to different application scenarios and needs. For example, the angle threshold, the definition and selection criteria of the points before and after the door can be adjusted according to the actual situation. In addition, other advanced algorithms and technologies (such as machine learning, computer vision, etc.) can be integrated into this process to improve the accuracy and efficiency of judgment.
[0117] In one or more embodiments of the present application, if there is a gate image that has not been eliminated in the depth image, a supplementary line segment between the center point of the gate straight line and the first shooting point is constructed; a supplementary line parallel to the gate straight line is constructed based on the center point of the supplementary line segment, and a point perpendicular to the gate straight line is marked on the supplementary straight line as the fourth candidate point.
[0118] In practical applications, doors in depth images can usually be identified by their specific shape, size, position, and depth difference from the surrounding environment. However, due to factors such as noise, occlusion, and reflection, some door images may not be accurately identified or eliminated in the initial processing.
[0119] For each gate image that has not been eliminated, we first need to determine the straight line representing the gate (usually the edge line or the center line of the gate) and calculate the center point of this straight line. This center point is crucial for the subsequent construction of line segments and straight lines.
[0120] Construct a supplementary line segment from the known first shooting point (which may be the position of the camera or a reference point) to the center point of the door line. This line segment helps locate the relative position between the door and the shooting point in three-dimensional space.
[0121] Based on the center point of the supplementary line segment, a supplementary line is constructed that is parallel to the door line. This line is selected to find possible operation points or path points in the direction perpendicular to the door.
[0122] On the supplementary line, select a point perpendicular to the door line as the fourth candidate point. This candidate point may be used for subsequent navigation, positioning, monitoring, or other tasks that require precise location information. Selecting a point perpendicular to the door line helps ensure that the point is on one side of the door to avoid collision or interference with the door itself.
[0123] In the depth image, noise and interference may lead to misidentification or missed recognition of the door image. Therefore, appropriate filtering and denoising measures need to be taken during the processing. For application scenarios with high real-time requirements, the algorithm needs to be optimized to improve processing speed and accuracy.
[0124] It should be noted that in actual applications, if there are multiple gates in the image, it is necessary to ensure that each gate can be processed correctly and avoid mutual influence between different gates. If the automatic processing is inaccurate, it can be marked by staff.
[0125] In one or more embodiments of the present application, if no target candidate point is found, an open area in the depth image is identified; and any point in the open area of the depth image is designated as the next shooting point.
[0126] Of course, when no candidate point that meets certain conditions (such as being perpendicular to the door line and suitable) is found in the depth image, identifying and selecting an open area as the next shooting point is a reasonable alternative solution. The details are as follows:
[0127] In depth images, open areas usually refer to areas without obvious obstacles, structures, or significant changes in depth. These areas may be corridors, halls, living rooms, etc., which provide greater freedom for camera movement.
[0128] You can set a depth threshold and treat areas within the depth range as empty areas. This helps exclude parts of the image with drastic depth changes, such as walls, furniture, etc.
[0129] By performing connectivity analysis on the depth image, we can identify continuous areas with small depth changes as open areas. These areas appear as large sets of continuous pixels with similar depth values on the image.
[0130] In practical applications, obstacle detection algorithms (such as edge detection, contour detection, etc.) can also be used to identify obstacles in the image, and areas not covered by obstacles are regarded as open areas.
[0131] In the identified open area, a point can be randomly selected as the next shooting point. This method is simple and easy, but it may not be the optimal solution, because the randomly selected point may not be the most conducive point for subsequent tasks. For this reason, an optional solution is also provided. In order to obtain a better next shooting point, some optimization criteria can be considered, such as maximizing the field of view, minimizing the number of obstructions, and approaching the target area. According to these criteria, the most suitable point is selected in the open area as the next shooting point.
[0132] For ease of understanding, the technical solution of this application will be described below through a complete embodiment. Figure 3 Schematic diagram of a shooting point prediction scheme illustrated in an embodiment of the present application. Figure 3 As you can see, the specific generation process is as follows:
[0133] The first panoramic image is captured at the previously preset point, that is, the first shooting point. Then, the depth image can be processed in two ways. One is to obtain the door point information, and then use the door point information to predict the next shooting point. The other is to obtain the plane information, and then use the plane information to predict the next shooting point.
[0134] After obtaining the first panoramic image, convert it into a point cloud image. Mark two points on the plane where the identified door image is located. The two points can be randomly marked on the plane, or marked on the border or the door vertex.
[0135] After obtaining the first panoramic image, it is converted into a point cloud image. After identifying the effective plane, the depth information of the first panoramic image is obtained through the model. The depth information is further processed to obtain the plane information.
[0136] In practical applications, the next shooting point can be predicted using the door point information and the plane information. One of the target candidate points obtained by these two methods can be selected as the next shooting point. If only one of the methods obtains a prediction result, the corresponding prediction method can be selected. For example, the next shooting point is predicted only by the door point information, or the next shooting point is predicted only by the plane information.
[0137] like Figure 4 A schematic diagram of the structure of a shooting point prediction device provided in an embodiment of the present application. Figure 4 As can be seen in the figure, the device includes:
[0138] An acquisition module 41 is used to acquire a panoramic image including depth information obtained by shooting at a first shooting point; the panoramic image includes a plane image and a door image;
[0139] An extraction module 42, configured to extract scene information based on the panoramic image to obtain door point information and plane information;
[0140] The extraction module 42 is further used to extract at least one valid plane from the panoramic image using the plane information;
[0141] The prediction module 43 is used to predict a next shooting point that meets the screening conditions from the panoramic image by using the effective plane and / or the door point information.
[0142] An extraction module 42, configured to obtain a point cloud image using the depth information contained in the panoramic image;
[0143] Determining a first normal vector of the point cloud image;
[0144] Segmenting the point cloud image using the first normal vector to obtain at least one valid plane;
[0145] Filtering out point cloud information in the point cloud image that is higher than the height of the camera at which the first shooting point is located;
[0146] Determine the ground using the first normal vector and point cloud projection, and filter out the ground;
[0147] After filtering out planes higher than the camera and filtering out the ground, a maximum vertical plane is determined from the at least one valid plane having a vertical axis perpendicular to the normal.
[0148] An extraction module 42 is used to establish a plurality of first candidate points in a two-dimensional coordinate system established based on the maximum vertical plane; wherein the horizontal axis of the two-dimensional coordinate system is parallel to the maximum vertical plane, and the vertical axis is perpendicular to the maximum vertical plane;
[0149] According to the positional relationship between the first candidate points and the valid plane, unqualified candidate points in the first candidate points are filtered out, and the reverse valid plane is filtered out;
[0150] The plurality of candidate points that are not filtered out are marked as second candidate points, and the first target candidate point is determined as the next shooting point according to the distance between the second candidate point and the valid plane that is not filtered out.
[0151] An extraction module 42 is used to construct a candidate straight line where the first shooting point and any one of the first candidate points are located;
[0152] If there is an intersection point between the candidate straight line and any of the valid planes, the candidate point is filtered out, and the candidate point that is not filtered out is used as the second candidate point;
[0153] When the angle between the candidate straight line and the effective normal of any effective plane is greater than an angle threshold, the effective plane in the opposite direction to the candidate straight line is filtered out.
[0154] A prediction module 43, configured to use two door points on the same door image plane in the panoramic image as the door point information, and a door straight line formed by the two door points;
[0155] Constructing a candidate straight line where the candidate point and the door point are located;
[0156] The corresponding candidate point whose angle between the candidate straight line and the gate straight line is not greater than the angle threshold is used as the third candidate point;
[0157] According to the comparison result of the candidate distance between the third candidate point and the door point, the second target candidate point is determined as the next shooting point. (If there are multiple third candidate points in the same direction, the third candidate points greater than the distance threshold are eliminated according to the distance between the door line and the first shooting point;)
[0158] A prediction module 43, configured to screen the first target candidate point and the second target candidate point, and determine whether the first target candidate point and the second target candidate point are in the same direction;
[0159] If yes, then selecting the second target candidate point as the fourth target candidate point;
[0160] If not, the first target candidate point and the second target candidate point in different directions are both used as the fourth target candidate point.
[0161] A prediction module 43, configured to mark a point in front of the door and a point behind the door as the third candidate point based on a perpendicular bisector of the door line when it is determined that the candidate line is perpendicular to the door line according to the angle and the angle threshold;
[0162] When it is determined that the candidate straight line is parallel to the gate straight line according to the angle and the angle threshold, the point in front of the gate is marked as the third candidate point based on the perpendicular bisector of the gate straight line.
[0163] A prediction module 43, configured to construct a supplementary line segment between the center point of the door straight line and the first shooting point if there is still a door image that has not been eliminated in the depth image;
[0164] A supplementary straight line parallel to the door straight line is constructed based on the center point of the supplementary line segment, and a point perpendicular to the door straight line is marked on the supplementary straight line as the fourth candidate point.
[0165] A prediction module 43, for identifying an empty area in the depth image if no target candidate point is found;
[0166] In the open area of the depth image, any point is designated as the next shooting point.
[0167] Figure 5 The structure diagram of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, it includes: a memory 50a and a processor 50b; wherein the memory 50a is used to store a computer program; the processor 50b is coupled to the memory 50a, and is used to execute the computer program to perform the following steps:
[0168] Acquire a panoramic image including depth information captured at a first shooting point; the panoramic image includes a plane image and a door image;
[0169] Extracting scene information based on the panoramic image to obtain door point information and plane information;
[0170] extracting at least one valid plane from the panoramic image using the plane information;
[0171] The effective plane and / or the door point information is used to predict a next shooting point that meets the screening condition from the panoramic image.
[0172] The processor 50b is used to obtain a point cloud image using the depth information contained in the panoramic image;
[0173] Determining a first normal vector of the point cloud image;
[0174] Segmenting the point cloud image using the first normal vector to obtain at least one valid plane;
[0175] Filtering out point cloud information in the point cloud image that is higher than the height of the camera at which the first shooting point is located;
[0176] Determine the ground using the first normal vector and point cloud projection, and filter out the ground;
[0177] After filtering out planes higher than the camera and filtering out the ground, a maximum vertical plane is determined from the at least one valid plane having a vertical axis perpendicular to the normal.
[0178] The processor 50b is used to establish a plurality of first candidate points in a two-dimensional coordinate system established based on the maximum vertical plane; wherein the horizontal axis of the two-dimensional coordinate system is parallel to the maximum vertical plane, and the vertical axis is perpendicular to the maximum vertical plane;
[0179] According to the positional relationship between the first candidate points and the valid plane, unqualified candidate points in the first candidate points are filtered out, and the reverse valid plane is filtered out;
[0180] The plurality of candidate points that are not filtered out are marked as second candidate points, and the first target candidate point is determined as the next shooting point according to the distance between the second candidate point and the valid plane that is not filtered out.
[0181] The processor 50b is used to construct a candidate straight line where the first shooting point and any one of the first candidate points are located;
[0182] If there is an intersection point between the candidate straight line and any of the valid planes, the candidate point is filtered out, and the candidate point that is not filtered out is used as the second candidate point;
[0183] When the angle between the candidate straight line and the effective normal of any effective plane is greater than an angle threshold, the effective plane in the opposite direction to the candidate straight line is filtered out.
[0184] The processor 50b is used for taking two door points on the same door image plane in the panoramic image as the door point information, and a door straight line formed by the two door points;
[0185] Constructing a candidate straight line where the candidate point and the door point are located;
[0186] The corresponding candidate point whose angle between the candidate straight line and the gate straight line is not greater than the angle threshold is used as the third candidate point;
[0187] According to the comparison result of the candidate distance between the third candidate point and the door point, the second target candidate point is determined as the next shooting point. (If there are multiple third candidate points in the same direction, the third candidate points greater than the distance threshold are eliminated according to the distance between the door line and the first shooting point;)
[0188] The processor 50b is used to screen the first target candidate point and the second target candidate point, and determine whether the first target candidate point and the second target candidate point are in the same direction;
[0189] If yes, then selecting the second target candidate point as the fourth target candidate point;
[0190] If not, the first target candidate point and the second target candidate point in different directions are both used as the fourth target candidate point.
[0191] The processor 50b is used for marking the point in front of the door and the point behind the door as the third candidate point based on the perpendicular bisector of the door line when it is determined that the candidate line is perpendicular to the door line according to the angle and the angle threshold;
[0192] When it is determined that the candidate straight line is parallel to the gate straight line according to the angle and the angle threshold, the point in front of the gate is marked as the third candidate point based on the perpendicular bisector of the gate straight line.
[0193] The processor 50b is configured to construct a supplementary line segment between the center point of the door straight line and the first shooting point if there is still a door image that has not been eliminated in the depth image;
[0194] A supplementary straight line parallel to the door straight line is constructed based on the center point of the supplementary line segment, and a point perpendicular to the door straight line is marked on the supplementary straight line as the fourth candidate point.
[0195] The processor 50b is used to identify an empty area in the depth image if no target candidate point is found;
[0196] In the open area of the depth image, any point is designated as the next shooting point.
[0197] It should be noted that the specific implementation of the above modules or units provided in this embodiment can refer to the above Figure 1 The relevant description of the illustrated method embodiment will not be repeated here.
[0198] An exemplary embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes one or more processors to execute the steps in the above-mentioned shooting point prediction method. The detailed implementation process of each step can be found in the relevant description of the above-mentioned embodiments, and will not be repeated here.
[0199] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0200] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0203] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0204] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0205] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transit media), such as modulated data signals and carrier waves.
[0206] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0207] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A shooting point prediction method, characterized in that: The method comprises: Acquire a panoramic image including depth information captured at a first shooting point; the panoramic image includes a plane image and a door image; Extracting scene information based on the panoramic image to obtain door point information and plane information; extracting at least one valid plane from the panoramic image using the plane information; The effective plane and / or the door point information is used to predict a next shooting point that meets the screening condition from the panoramic image.
2. The method according to claim 1, characterized in that The using the plane information to determine a valid plane from the panoramic image includes: Obtaining a point cloud image using the depth information contained in the panoramic image; Determining a first normal vector of the point cloud image; Segmenting the point cloud image using the first normal vector to obtain at least one valid plane; Filtering out point cloud information in the point cloud image that is higher than the height of the camera at which the first shooting point is located; Determine the ground using the first normal vector and point cloud projection, and filter out the ground; After filtering out planes higher than the camera and filtering out the ground, a maximum vertical plane is determined from the at least one valid plane having a vertical axis perpendicular to the normal.
3. The method according to claim 2, characterized in that Selecting a next shooting point that meets a screening condition from the panoramic image using the effective plane includes: Establishing a plurality of first candidate points in a two-dimensional coordinate system established based on the maximum vertical plane; wherein the horizontal axis of the two-dimensional coordinate system is parallel to the maximum vertical plane, and the vertical axis is perpendicular to the maximum vertical plane; According to the positional relationship between the first candidate points and the valid plane, unqualified candidate points in the first candidate points are filtered out, and the reverse valid plane is filtered out; The plurality of candidate points that are not filtered out are marked as second candidate points, and the first target candidate point is determined as the next shooting point according to the distance between the second candidate point and the valid plane that is not filtered out.
4. The method according to claim 3, characterized in that The filtering out unqualified candidate points in the first candidate points and filtering out the reverse valid plane according to the positional relationship between the first candidate points and the valid plane includes: Constructing a candidate straight line between the first shooting point and any one of the first candidate points; If there is an intersection point between the candidate straight line and any of the valid planes, the candidate point is filtered out, and the candidate point that is not filtered out is used as the second candidate point; When the angle between the candidate straight line and the effective normal of any effective plane is greater than an angle threshold, the effective plane in the opposite direction to the candidate straight line is filtered out.
5. The method according to claim 1 or 4, characterized in that: Using the door point information, predicting a next shooting point that meets the screening condition from the panoramic image includes: Two door points on the same door image plane in the panoramic image are used as the door point information, and a door straight line formed by the two door points; Constructing a candidate straight line where the first candidate point and the door point are located; The corresponding candidate point whose angle between the candidate straight line and the gate straight line is not greater than the angle threshold is used as the third candidate point; According to the comparison result of the candidate distance between the third candidate point and the door point, the second target candidate point is determined as the next shooting point.
6. The method according to claim 5, characterized in that Screening the first target candidate point and the second target candidate point, and determining whether the first target candidate point and the second target candidate point are in the same direction; If yes, then selecting the second target candidate point as the fourth target candidate point; If not, the first target candidate point and the second target candidate point in different directions are both used as the fourth target candidate point.
7. The method according to claim 5, characterized in that The step of taking a corresponding candidate point whose included angle between the candidate straight line and the gate straight line is not greater than an included angle threshold as a third candidate point includes: When it is determined according to the angle and the angle threshold that the candidate straight line is in a perpendicular relationship with the gate straight line, a point in front of the gate and a point behind the gate are marked as the third candidate point based on the perpendicular bisector of the gate straight line; When it is determined that the candidate straight line is parallel to the gate straight line according to the angle and the angle threshold, the point in front of the gate is marked as the third candidate point based on the perpendicular bisector of the gate straight line.
8. The method according to claim 6, characterized in that If there is still a door image that has not been eliminated in the depth image, constructing a supplementary line segment between the center point of the door straight line and the first shooting point; A supplementary straight line parallel to the door straight line is constructed based on the center point of the supplementary line segment, and a point perpendicular to the door straight line is marked on the supplementary straight line as the fourth candidate point.
9. The method according to claim 1, characterized in that: If no target candidate point is found, identifying an empty area in the depth image; In the open area of the depth image, any point is designated as the next shooting point.
10. A shooting point prediction device, characterized in that: The device comprises: An acquisition module, used for acquiring a panoramic image including depth information obtained by shooting at a first shooting point; the panoramic image includes a plane image and a door image; An extraction module, used to extract scene information based on the panoramic image to obtain door point information and plane information; The extraction module is further used to extract at least one valid plane from the panoramic image using the plane information; A prediction module is used to predict a next shooting point that meets the screening conditions from the panoramic image by using the effective plane and / or the door point information.
11. An electronic device, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; and the processor is coupled to the memory and is used to execute the computer program to execute the method according to any one of claims 1 to 9.
12. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 9.
Citation Information
Cited By
Method for intelligently identifying river and lake water area change and early warning river and lake abnormal problems by unmanned aerial vehicle
CN121095818A
