Image processing method and device, equipment and medium

Abstract

The embodiment of the invention provides an image processing method and device, equipment and a medium. In the embodiment of the invention, a camera is controlled to shoot an indoor environment to obtain a panoramic image; establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system; establishing a linkage relationship between the indoor environment and the camera in the camera coordinate system by using a relative position relationship between the indoor environment and the camera in the world coordinate system; and in response to a shooting angle adjustment request for the panoramic image or the point cloud top view, synchronously adjusting the panoramic image and the point cloud top view according to the linkage relationship. A linkage relationship between an indoor environment and a camera in a camera coordinate system is established, and in an image shooting process, no matter whether a shooting person adjusts any one of an angle of a panoramic image or an angle of a point cloud top view, the other angle is adjusted, so that the angles of the two images can be kept in a real-time synchronous state, and the real-time synchronous state of the angles of the panoramic image and the point cloud top view is realized. And the user can be better guided to find the next appropriate shooting point position for image shooting.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to image processing methods, devices, equipment and media. Background Art

[0002] With the development of virtual reality (VR) technology, it has played an important role in more and more fields. For example, real estate promotion is carried out online. In order to help users better understand the housing situation, a complete house video will be recorded for users.

[0003] In the prior art, fixed-point shooting is performed based on a panoramic camera. After shooting is completed, the panoramic image is generated into a point cloud through a deep model algorithm, and then the point cloud is spliced ​​to obtain a point cloud top view. The point cloud is a collection of feature points in a house, which can display the basic information of the house. In order to facilitate users to intuitively feel the outline of the room, a top view is generally used for display. However, when splicing the point cloud top view, the user needs to observe and identify which direction in the point cloud top view corresponds to which content in the panoramic image, which is prone to alignment problems or alignment errors. Summary of the invention

[0004] Various aspects of the present application provide image processing methods, devices, equipment and media to solve the technical problems of complex and inefficient image processing.

[0005] In a first aspect, an exemplary embodiment of the present application provides an image processing method, which is applied to a server side, and the method includes:

[0006] Control the camera to shoot the indoor environment to obtain a panoramic image;

[0007] Establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system;

[0008] Using the relative position relationship between the indoor environment and the camera in the world coordinate system, a linkage relationship between the indoor environment and the camera in the camera coordinate system is established;

[0009] In response to a request for adjusting a shooting angle of the panoramic image or the point cloud overhead view, the panoramic image and the point cloud overhead view are synchronously adjusted according to the linkage relationship.

[0010] In a second aspect, an exemplary embodiment of the present application provides an image processing method, the method comprising:

[0011] The panoramic image and the point cloud overhead view captured by the camera are displayed in real time on the user terminal through an image display window; the image display window includes at least a first display area and a second display area; wherein the first display area displays a partial viewing angle image in the panoramic image captured by the camera in real time, and the second display area displays the point cloud overhead view captured by the camera in real time;

[0012] In response to a rotation operation on the first display area or the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

[0013] In a third aspect, an exemplary embodiment of the present application provides an image processing device, the device comprising:

[0014] A control module, used to control the camera to capture the indoor environment and obtain a panoramic image;

[0015] An establishing module, used for establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system;

[0016] The establishing module is used to establish a linkage relationship between the indoor environment and the camera in the camera coordinate system by using the relative position relationship between the indoor environment and the camera in the world coordinate system;

[0017] An adjustment module is used to respond to a request for adjusting the shooting angle of the panoramic image or the point cloud overhead view, and synchronously adjust the panoramic image and the point cloud overhead view according to the linkage relationship.

[0018] In a fourth aspect, an exemplary embodiment of the present application further provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program to execute the steps in the above-mentioned image processing method.

[0019] In a fifth aspect, an exemplary embodiment of the present application provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the steps in the above-mentioned image processing method.

[0020] In this embodiment, when shooting the indoor environment, a panoramic camera is often used to collect images of the indoor environment to obtain a panoramic image. When the panoramic image is displayed to the user, it is usually an image corresponding to the indoor environment that the user can currently see, that is, an image of a partial perspective in the panoramic image. In addition, in practical applications, in order to draw a complete house image, it is also necessary to display a point cloud top view of the current indoor environment. In order to make the point cloud top view consistent with the panoramic image, it is necessary to establish a linkage relationship between the indoor environment and the camera, that is, to make the viewing direction displayed by the panoramic image of the indoor environment consistent with the direction of the point cloud top view in real time. Specifically, it is necessary to first establish a camera coordinate system based on the location of the camera, and then draw the point cloud top view corresponding to the panoramic image based on the camera coordinate system. Further, the relative position relationship between the indoor environment and the camera in the world coordinate system is calculated, and then the linkage relationship between the indoor environment and the camera in the camera coordinate system is established. The linkage relationship here refers to keeping the perspective of the panoramic image and the perspective of the point cloud top view synchronized. During the image capture process, no matter whether the photographer adjusts the angle of the panoramic image or the angle of the point cloud overhead view, the other angle will also be adjusted accordingly, so that the angles of the two images can remain synchronized in real time, which is convenient for users to understand their current location more accurately when shooting the whole house, and better guide users to find the next suitable shooting point for image capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0023] Figure 2 Another image processing method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of an image captured by a user terminal device as an example of an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0029] When shooting an indoor environment, a panoramic camera is often used to collect images of the indoor environment to obtain a panoramic image. When a panoramic image is displayed to a user, it usually displays the image corresponding to the indoor environment that the user can currently see, that is, displays an image of a partial perspective in the panoramic image. In addition, in practical applications, in order to draw a complete house image, it is also necessary to display a point cloud top view of the current indoor environment. In the prior art, when establishing a point cloud top view, a fixed-point shooting is usually performed based on a camera (generally a panoramic camera), and the image obtained is also a panoramic image (a panoramic image is a 360° image, but when it is displayed in the display area of ​​the user-end device, only one of the perspectives can be displayed. If you want to see images from other perspectives or directions, the user needs to manually rotate and adjust the perspective). After the panoramic image is obtained by shooting, the panoramic image is generated into a point cloud map through a deep model algorithm, and then the point cloud map is spliced. The point cloud map is a collection of feature points of the indoor environment of the house, which can display the basic information of the indoor environment of the house. In order to facilitate users to intuitively feel the outline of the room, a point cloud map is generally displayed from a top view perspective, that is, a point cloud top view. When displaying a panoramic image, it is difficult for the user to accurately know the perspective correspondence and position correspondence with the point cloud top view through vision. In the case that the shooting point set by the user is not good, in the process of generating a point cloud top view using a panoramic image, the point cloud top view generation will fail or the effect will be poor due to incomplete or missing feature points. Therefore, a solution is needed to help users understand the state of the captured image so that users can choose the best shooting point for image capture.

[0030] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0031] Figure 1 The following is a flow chart of the image processing method provided in the embodiment of the present application. Figure 1 As shown, the image processing method provided in the embodiment of the present application can be run on a user terminal, which is equipped with a camera for taking panoramic images (of course, it can also be equipped with a camera that can collect depth information) and a controller that controls the camera operation and background data processing. The method specifically includes the following steps:

[0032] Step 101: Control the camera to capture the indoor environment to obtain a panoramic image.

[0033] Step 102: Establish a camera coordinate system, and draw a point cloud top view corresponding to the panoramic image based on the camera coordinate system.

[0034] Step 103: using the relative position relationship between the indoor environment and the camera in the world coordinate system, establish a linkage relationship between the indoor environment and the camera in the camera coordinate system.

[0035] Step 104: In response to a request to adjust the shooting angle of the panoramic image or the point cloud overhead view, synchronously adjust the panoramic image and the point cloud overhead view according to the linkage relationship.

[0036] The user can control the camera's image acquisition process through the user end or the controller on the user end, and the subsequent coordinate conversion and coordinate system establishment are all completed by the controller. When the camera is shooting at a certain shooting point, it can capture a panoramic image, that is, a 360° image around the camera as the center. However, when the display device on the user end is displayed, only one image of the perspective can be displayed at a time. If the user wants to view the image content of other perspectives through the display device, the panoramic image needs to be manually rotated. For example, the user wants to see which position in the surrounding environment can be used as the next shooting point. At this time, the user has the need to switch the image content of other perspectives. Assuming that the display device on the user end supports touch operation, the user can manually swipe on the display device, and then drag the panoramic image to rotate to achieve perspective switching. If the current perspective direction is due north, you can turn the perspective 90 degrees to the left and face due west.

[0037] When the camera captures the panoramic image, the controller establishes a camera coordinate system based on the camera's position. The camera coordinate system is only used to align the rotation angle (that is, the viewing angle) of the panoramic image and the point cloud top view, and no translation alignment is performed. When establishing the point cloud top view, the top view is established based on the camera coordinate system.

[0038] For example, when defining the camera coordinate system, the camera coordinate system is established with the center of the camera lens as the origin, the optical axis of the lens as the Z axis, the horizontal right as the X axis, and the vertical downward as the Y axis. In order to establish a point cloud map, it is necessary to obtain the depth information corresponding to the panoramic image. If the camera supports depth perception (such as using an RGB-D camera), the depth value of each pixel in the panoramic image is directly obtained; if the camera does not support depth perception, it is necessary to obtain the depth data of the indoor environment through other methods (such as structured light, lidar scanning, etc.). Then, according to the panoramic image and the corresponding depth information, each pixel in the image is converted into a point in three-dimensional space to generate point cloud data. The point cloud data is projected onto the XY plane (that is, the horizontal plane of the camera coordinate system) to generate a point cloud top view. Optionally, different colors or height encodings can be set to enhance the visual effect.

[0039] The relative position relationship between the indoor environment and the camera in the world coordinate system is used to establish a linkage relationship between the indoor environment and the camera in the camera coordinate system. The linkage relationship mentioned here can be understood as a relationship in which the viewing angles of the panoramic image and the point cloud top view remain synchronized, rather than just coordinate synchronization. That is, when the viewing angle image currently displayed by the panoramic image rotates, the corresponding viewing angle of the point cloud top view also rotates synchronously.

[0040] Before establishing the camera coordinate system, it is necessary to use the world coordinate system to calculate the relative position relationship between the indoor image and the camera. Select a fixed reference point (such as a corner of the room) as the origin of the world coordinate system to establish the world coordinate system. When the camera takes an image, you can know the world coordinates of the camera and the panoramic image taken by the camera in the world coordinate system. Then, the relative position relationship between the panoramic image and the camera can be calculated using the corresponding world coordinates. Then, when the camera coordinates are reset to zero in the camera coordinate system, it is easy to get the relative coordinates of the panoramic image in the camera coordinate system. The specific coordinate conversion relationship will be explained in the following embodiments, so it will not be repeated here.

[0041] In actual applications, users can issue adjustment requests for the shooting angle (that is, the display viewing angle of the panoramic image on the user end) through the user end according to their needs. There are many ways to issue this adjustment request, generally speaking, it is related to the functions supported by the user end. If the user end supports touch screen, the user can issue an adjustment request by swiping on the touch screen. If the user end does not support touch screen, the adjustment request can be issued through controls or mouse, keyboard, etc. In addition, when the user issues a touch request, it can be an adjustment request for the panoramic image or for the point cloud overhead view. Since a linkage relationship is established between the panoramic image and the point cloud overhead view, as long as one of the images rotates the shooting angle (that is, the viewing angle of the panoramic image), the other also rotates accordingly, so that the panoramic image and the point cloud overhead view remain in real-time synchronization. By establishing a linkage method, users can understand the angle direction of the panoramic image viewing angle in the point cloud overhead view in real time, so that users can have an overall understanding of the current space, which is more conducive to guiding users to choose the appropriate next shooting point.

[0042] In one or more embodiments of the present application, the relative position relationship between the indoor environment and the camera in the world coordinate system is utilized to establish a linkage relationship between the indoor environment and the camera in the camera coordinate system, including: determining a first coordinate of the camera in the world coordinate system, and a second coordinate of the indoor environment in the world coordinate system; calculating the relative position relationship using the first coordinate and the second coordinate; and establishing a linkage relationship between the indoor environment and the camera in the camera coordinate system based on the relative position relationship and a defined initial direction.

[0043] First, determine the coordinates of the camera and the indoor environment in the world coordinate system. That is, it is necessary to determine the precise position of the camera in the world coordinate system, which is usually obtained through the camera's position sensor (such as GPS, IMU, etc.) or external calibration equipment (such as laser rangefinder, total station, etc.). The position of the camera can be expressed as a three-dimensional vector C = (C x ,C y ,C z ).

[0044] The coordinates of the indoor environment (which can be understood as the coordinates of various objects in the indoor environment, and one of the objects can be selected to participate in the coordinate transformation) may refer to the coordinates of a key point or feature point in the environment, or a wider area. For simplicity, several representative points (such as corners, door handles, etc.) can be selected as the coordinates of the indoor environment, and their positions in the world coordinate system can be recorded respectively. These points can be represented as a series of three-dimensional vectors Pi = (Pix, Piy, Piz), where i represents different points.

[0045] For each point Pi in the indoor environment, its position vector Vi = Pi-C relative to the camera position C can be calculated. These vectors describe the direction and distance from the camera to each point in the indoor environment. In addition to translation, it is also necessary to determine the rotation relationship between the camera coordinate system and the world coordinate system. This is usually determined by the camera's attitude (i.e., orientation and tilt), which can be represented by the rotation matrix R. The rotation matrix R can be obtained through the camera's attitude sensor (such as a gyroscope, accelerometer, etc.) or an external calibration process.

[0046] Establish linkage relationship in camera coordinate system

[0047] Apply rotation and translation: To transform a point in the indoor environment from the world coordinate system to the camera coordinate system, you need to apply a rotation matrix R and a translation vector C to each point Pi (or more accurately, its negative value, because you are transforming from the world coordinate system to the camera coordinate system). The coordinates of the transformed point Pi′ in the camera coordinate system can be calculated as follows:

[0048] P i′ =R·(P i -C)

[0049] Note that we are actually doing an inverse transformation here, because we want to convert from the world coordinate system to the camera coordinate system.

[0050] Once we have the coordinates of all points in the indoor environment in the camera coordinate system, we can locate them in the camera image based on these coordinates, or perform further analysis and processing (such as 3D reconstruction, augmented reality, etc.).

[0051] In one or more embodiments of the present application, in response to a request to adjust the shooting angle of the panoramic image or the point cloud overhead view, synchronously adjusting the panoramic image and the point cloud overhead view according to the linkage relationship includes: determining a first rotation angle in response to a request to adjust the image viewing angle in the panoramic image; synchronizing the first rotation angle to the point cloud overhead view; and rotating the panoramic image and the point cloud overhead view by the first rotation angle based on the current angle. Alternatively, determining a second rotation angle in response to a request to adjust the image viewing angle of the point cloud overhead view; synchronizing the second rotation angle to the panoramic image; and rotating the panoramic image and the point cloud overhead view by the second rotation angle based on the current angle.

[0052] In the display interface, the point cloud top view and panoramic image are displayed separately through different display areas. In order to facilitate the user to manipulate the image, intuitive rotation controls (such as rotation buttons, sliders, or drag areas) can be provided to allow users to directly operate to adjust the viewing angle. Of course, no rotation controls may be provided, and users can rotate directly on the image, thereby effectively changing the display viewing angle. The controller's background system monitors the interaction events of these controls, and once the user's rotation operation or click on the rotation button is detected, it is immediately recognized as a request to adjust the shooting angle.

[0053] As mentioned above, there are two ways to adjust the viewing angle. One is to adjust the viewing angle through the panoramic image, and the other is to adjust the viewing angle through the point cloud top view. The details are as follows:

[0054] Panoramic image viewing angle adjustment: If the user directly operates the panoramic image, the system calculates the first rotation angle according to the user's operation (such as drag distance, slider position change, etc.). This angle can be a rotation angle around the vertical axis (Y axis, assumed to be in the upward direction).

[0055] Point cloud top view perspective adjustment: If the user directly operates the point cloud top view, the second rotation angle is also calculated based on the user operation. Since the point cloud top view is usually fixed to a top view perspective, the rotation here may be more reflected in fine-tuning around the horizontal axis (X axis or Z axis, depending on the specific implementation), but in this scenario, the focus is on the rotation of the vertical axis to maintain consistency.

[0056] Regardless of whether the panoramic image or the point cloud top view is adjusted, the system synchronizes the calculated rotation angle (first rotation angle or second rotation angle) to the other view. This ensures the consistency of the rotation angle between the two.

[0057] The controller obtains the current rotation angle of the panoramic image and the point cloud top view, and then adds the synchronized rotation angle (first rotation angle or second rotation angle) to achieve synchronous rotation of the two. After the user adjusts the rendering angle, the panoramic image and the point cloud top view are re-rendered according to the new rotation angle to ensure that the user sees the adjusted perspective.

[0058] In actual applications, during the rotation process, a smooth animation effect is provided so that users can intuitively feel the change in perspective. The current rotation angle is displayed on the interface or other forms of feedback are provided to help users understand the current perspective status. It should be noted that during the rotation process, check whether the rotation angle exceeds a reasonable range (such as within 360 degrees) to avoid invalid or abnormal perspectives. If the rotation cannot be completed due to technical reasons (such as rendering failure, misaligned perspectives, etc.), the system should give a clear error prompt and try to restore to the last valid state (for example, the perspectives are adjusted to the initial state).

[0059] In one or more embodiments of the present application, establishing a camera coordinate system and drawing a point cloud overhead view corresponding to the panoramic image based on the camera coordinate system includes: establishing the camera coordinate system with the first shooting position of the camera as the origin of the camera coordinate system; performing depth estimation on the panoramic image to obtain an estimated depth image; and generating a point cloud overhead view based on the estimated depth image and two-dimensional coordinate information of the panoramic image in the camera coordinate system.

[0060] In the present application, the acquired panoramic image can be understood as a horizontal viewing angle of usually 360 degrees, and a vertical viewing angle of 180 degrees or higher. This enables the camera to acquire comprehensive data when performing image acquisition at one position. Although panoramic images can be acquired, the acquisition distance is limited, that is, in space, panoramic images need to be acquired at multiple different points to meet the subsequent generation requirements of complete indoor images. In practical applications, professional image processing software and point cloud generation tools are used, such as MATLAB, Python (combined with OpenCV, PCL and other libraries) or professional 3D reconstruction software.

[0061] In practical applications, necessary preprocessing is performed on panoramic images, such as denoising and contrast enhancement, to improve the accuracy of subsequent processing. Key feature points are extracted from panoramic images, which usually have high texture information or edge information, which is helpful for subsequent registration and reconstruction.

[0062] When a panoramic image is acquired, since some panoramic images do not contain depth information themselves, it is necessary to estimate the depth in some way. This can be achieved through machine learning models (such as deep learning networks), which can infer depth maps from images. Of course, users can choose cameras that support depth information acquisition as needed, that is, the acquired panoramic images contain depth information. Using the estimated depth map and the two-dimensional coordinate information of the panoramic image, the pixels in the two-dimensional image are mapped to the three-dimensional space through inverse projection or other three-dimensional reconstruction techniques to form a point cloud top view. The point cloud top view can also be further optimized. For example, noise points in the point cloud can be removed, which may be caused by depth estimation errors or image noise. The point cloud is streamlined to reduce the amount of data while maintaining the main features of the point cloud.

[0063] In order to make the point cloud top view display better and facilitate users to intuitively identify the correspondence between the panoramic image and the point cloud top view, the color information in the panoramic image can be mapped to the point cloud so that the point cloud has color attributes and is more intuitive.

[0064] In one or more embodiments of the present application, it also includes using a pre-trained point prediction model to identify the panoramic image, selecting at least one alternative point from the panoramic image; determining the rotation angle of the alternative point in the panoramic image; and marking the predicted position of the alternative point in the point cloud overhead view based on the rotation angle.

[0065] Compared with the point cloud top view with a single background, the panoramic image contains more effective feature points, that is, it is easier to identify the content contained in the panoramic image. Therefore, when collecting panoramic images, ensure that high-quality panoramic images are obtained. These images should cover the full range of the target scene without significant occlusion or distortion.

[0066] As mentioned above, prepare the corresponding point cloud data and perform necessary preprocessing, such as denoising, filtering, coordinate conversion, etc., in order to align it with the panoramic image. Load the pre-trained point prediction model, which should have been trained for similar scenes and can accurately identify specific types of points (that is, the next one or more optional panoramic image shooting points).

[0067] In practical applications, the panoramic image is input into a pre-trained point prediction model. The point prediction model can identify and analyze the image and identify at least one candidate point that meets the preset conditions. These points can be the most likely target location predicted by the model, a predicted optimal shooting point, or multiple shooting points that can be selected by the user. The coordinates of each candidate point (in the camera coordinate system) are extracted from the model output.

[0068] It should be noted that when using the point prediction model to identify the shooting point, first determine whether the current distance is appropriate. If it is appropriate, the space or environment will be further judged, and the point with a wide surrounding field of view and obvious features will usually be selected as the next shooting point. If there are many obstacles and the field of view is not good (for example, behind a door), it will not be identified as the next shooting point. If the distance is too far (that is, the distance between the next point and the current shooting point), it will not be selected as the next shooting point, because if the distance between the two shooting points is too far, the image connection will not be tight, and it will be difficult to find the same feature points when synthesizing the image. Based on the above requirements, the predicted shooting points found by the point prediction model are more conducive to subsequent image processing.

[0069] After at least one shooting point is determined using the panoramic image, this point needs to be presented in the point cloud top view so that the user can intuitively understand the location of the next shooting point. Specifically, the panoramic image and the point cloud top view are spatially aligned using the known correspondence between the panoramic image and the point cloud top view (such as through GPS, IMU or pre-set calibration parameters). For each candidate point, the rotation angle of the point relative to the point cloud top view is calculated based on its coordinates in the panoramic image and the shooting direction (or camera posture) of the panoramic image. When the rotation angle or viewing angle is calculated using the coordinates, it is calculated in the camera coordinate system. The calculated rotation angle and the coordinates of the candidate point in the camera coordinate system are mapped to the point cloud top view. This may require coordinate transformation to convert the two-dimensional image coordinates into points in three-dimensional space and project them onto the point cloud top view plane. In the point cloud top view, the predicted position of each candidate point is marked with different colors, markers or highlights according to the mapping results. That is, when marking, the user can directly distinguish which are the current points and which are the predicted points in the point cloud overhead view.

[0070] In practical applications, the marked point cloud top view is visualized so that the position of each candidate point in three-dimensional space can be intuitively seen. The shooting user verifies the accuracy of the predicted position through field visits or by using other sensor data (such as lidar scanning). Since the point cloud top view is aligned with the panoramic image content, the user can easily find the corresponding next shooting point from the panoramic image after marking the next predicted point in the point cloud top view. This allows the user to determine whether the next shooting point is appropriate through the panoramic image, or to select the next shooting point from multiple points through the panoramic image.

[0071] In one or more embodiments of the present application, the using a pre-trained point prediction model to identify the panoramic image and selecting at least one candidate point from the panoramic image includes:

[0072] Get the coordinate information of the current shooting point and the first point cloud data of the current point;

[0073] Determine at least one candidate point according to the distance threshold;

[0074] Calculating the point cloud overlap between the first point cloud data and the second point cloud data according to the second point cloud data corresponding to each of the candidate points;

[0075] According to the point cloud overlap, a point is selected from at least one candidate point as the next shooting point.

[0076] Furthermore, the number of overlapping point clouds can be counted, and the ratio of this number to the total number of point clouds in the second point cloud can be calculated. This ratio is the first overlap degree, which reflects the overlap degree of visual information between two points.

[0077] By the above method, the overlap degree of the point cloud between the first shooting point and the first candidate point can be calculated, so as to evaluate the candidate point by using the calculated overlap degree.

[0078] In one or more embodiments of the present application, the use of the first distance value and the first overlap to calculate the evaluation score corresponding to each of the candidate points in the candidate point list corresponding to the first point group includes: obtaining the third coordinate of the next shooting point, and obtaining the fourth coordinate of the current shooting point; using the third coordinate and the fourth coordinate to calculate the first distance value; inputting the first distance value and the first overlap into a first calculation formula: score = 1 / (dist+1e -6 )+overlap; wherein score is the first evaluation score, distb represents the first distance value, and overlap represents the first repetition degree.

[0079] It should be noted that the next shooting point mentioned here is a next shooting point (that is, the parent node point) selected sequentially from the parent list. Before inputting the first distance value and the first overlap into the first calculation formula, it is necessary to first determine whether the first distance value and the first overlap meet the preset conditions. Specifically, determine whether the first distance value is greater than the distance threshold, and determine whether the first overlap is greater than the overlap threshold; if the judgment result is that the first distance value is greater than the distance threshold, and / or the first overlap is less than the overlap threshold, then the target shooting point is assigned a maximum value.

[0080] For example, the distance threshold can be set to 6 meters and the overlap threshold can be set to 0.25. If the judgment result is that the first distance value is greater than the distance threshold, it means that the distance is too far and cannot meet the panoramic image registration synthesis requirements. If the judgment result is that the first overlap is not greater than the overlap threshold, it means that the image content cannot meet the registration synthesis requirements due to the low overlap. In this case, in order to facilitate the subsequent adjustment of the candidate points, the ID value of the next shooting point can be temporarily assigned a maximum value, and it will not be used as a candidate point for evaluation this time. For candidate points that do not meet the threshold conditions, a very large evaluation score (such as positive infinity) is assigned to ensure that they will not be selected in the subsequent selection process. In addition, there are other methods, such as directly removing these candidate points from the list to avoid unnecessary calculations. Here, it is marked as a maximum value so that the next shooting point can be re-evaluated when needed later. If it is removed from the list, it is a temporary removal method, such as transferring to a substitute list so that the candidate point can be re-evaluated when needed later.

[0081] If the judgment result is that the first distance value is less than the distance threshold, and the first overlap is greater than the overlap threshold, it means that the distance of the next shooting point is appropriate, and the first overlap of the image of the next shooting point and the image of the current shooting point is greater than the overlap threshold, it means that the next shooting point meets the basic requirements and can participate in the score evaluation. Threshold judgment can help exclude candidate points that are too far away or have too low overlap, thereby improving the quality and efficiency of the panorama.

[0082] After the above threshold judgments for distance and overlap, the evaluation score of the next shooting point can be further calculated.

[0083] When calculating the evaluation score corresponding to each candidate point in the candidate point list, two factors, distance and overlap, need to be considered comprehensively.

[0084] When calculating the distance, the calculation is performed in the same coordinate system. The first distance value is calculated using the third coordinate of the next shooting point and the fourth coordinate of the current shooting point through the Euclidean distance formula. This distance value reflects the physical distance between the two points. In indoor scenes, the influence of walls, indoor doors, etc. on the calculated distance should be considered. Specifically, if the next shooting point and the current shooting point are located on both sides of the wall, when calculating the physical distance, the straight-line distance between the two points cannot be directly calculated, but the influence of the wall should be considered. If the next shooting point is located on both sides of the door of the current shooting point, since the door is a moving object that can be opened and closed, when calculating the distance, it can be calculated according to the straight-line distance, and the influence of the door does not need to be considered.

[0085] Evaluation score calculation: Input the first distance value and the first overlap into the first calculation formula to calculate the evaluation score. However, please note that the formula score=1 / (dist+1e -6 )+overlap may need to be adjusted, because the overlap is usually a value between 0 and 1, and direct addition may cause the evaluation score to exceed the expected range. A more reasonable approach may be to multiply the overlap by a weight factor, or to normalize the distance term and overlap term separately before calculation.

[0086] As an alternative, the above formula can be optimized:

[0087] Assume that overlap is already a value between 0 and 1, dist is the distance value, and dist_max is the set maximum distance threshold.

[0088] dist_normalized = 1 / (dist+1e -6 ), and normalize the distance.

[0089] overlap_weight=0.5, where it is assumed that the weight of overlap is 0.5.

[0090] score=dist_normalized*(1-overlap_weight)+overlap*overlap_weight.

[0091] It should be noted that 1-overlap_weight here is the weight of the distance term, and its sum with the weight of the overlap is 1, ensuring the rationality of the evaluation score.

[0092] The candidate point list contains multiple candidate points. Existing candidate points are generally points where shooting has been completed. When the user shoots at a new shooting point (that is, the current shooting point mentioned above), it is necessary to evaluate whether the current shooting point is suitable and reasonable. The suitable and reasonable here refers to the convenience of subsequent image synthesis. In the actual evaluation, the distance between points and the overlap of point clouds are used for evaluation, and the rationality of the points can be comprehensively evaluated from the perspective of image content and image effect.

[0093] Based on the same idea, this application also provides an image processing method. Figure 2 Another image processing method provided in an embodiment of the present application. The method can be applied to a user terminal equipped with a camera, and the method includes:

[0094] Step 201: Display the panoramic image and point cloud overhead view taken by the camera in real time through the image display window on the user terminal; the image display window includes at least a first display area and a second display area; wherein the first display area displays a partial viewing angle image of the panoramic image acquired by the camera in real time, and the second display area displays the point cloud overhead view acquired by the camera in real time.

[0095] Step 202: In response to a rotation operation on the first display area or the second display area, synchronously perform a rotation operation on display contents in the first display area and the second display area.

[0096] For ease of understanding, the following will be described by way of specific embodiments. Figure 3 The following is a schematic diagram of an image captured by a user terminal device as an example of an embodiment of the present application. Figure 3 As can be seen in the figure, there are two display areas on the user side, namely the first display area (displaying the panoramic image) and the second display area (displaying the point cloud top view). The panoramic image display area is used to display images within a viewing angle range of the panoramic image (that is, the partial viewing angle image mentioned above), and cannot display 360° panoramic images. If the user wants to see content from other angles, the viewing angle can be rotated by sliding the panoramic image. Figure 3 As can be seen, the image of the indoor entrance door is displayed in the panoramic image. Correspondingly, the direction of the point cloud top view is also pointing to the direction of the entrance door. At this time, the panoramic image and the point cloud top view are aligned, realizing the linkage relationship between the two.

[0097] After the panoramic image is captured, the panoramic image is further identified using a pre-trained point prediction model, and at least one candidate point is selected from the panoramic image. In response to the rotation operation on the first display area or the second display area, the display content in the first display area and the second display area is synchronously rotated, including:

[0098] If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the first display area, synchronously performing a rotation operation on display contents in the first display area and the second display area;

[0099] or,

[0100] If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

[0101] from Figure 3It can be seen that a panoramic image is displayed in the first display area, and the corresponding point cloud overhead view is displayed in the second display area. After establishing the linkage relationship between the panoramic image and the point cloud overhead view, the linkage control of the two images can be realized. In order to facilitate the user to rotate the perspective, a rotation perspective control can be configured, and the user can control the rotation perspective of two images (panoramic image and point cloud overhead view) at the same time through one control. Of course, it is also possible to configure a rotation control for different images separately, and the user can choose which control to control according to needs. You can also set a reset control, which is mainly used to align the initial position of the panoramic image and the point cloud overhead view.

[0102] Based on the same idea, the embodiment of the present application also provides an image processing device. Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 4 It can be seen that the device includes:

[0103] The control module 41 is used to control the camera to capture the indoor environment to obtain a panoramic image.

[0104] The establishing module 42 is used to establish a camera coordinate system and draw a point cloud top view corresponding to the panoramic image based on the camera coordinate system.

[0105] The establishing module 42 is used to establish a linkage relationship between the indoor environment and the camera in the camera coordinate system by using the relative position relationship between the indoor environment and the camera in the world coordinate system.

[0106] The adjustment module 43 is used to respond to a request for adjusting the shooting angle of the panoramic image or the point cloud overhead view, and synchronously adjust the panoramic image and the point cloud overhead view according to the linkage relationship.

[0107] Establishing module 42, used for determining the first coordinate of the camera in the world coordinate system, and the second coordinate of the indoor environment in the world coordinate system;

[0108] Calculating the relative position relationship using the first coordinate and the second coordinate;

[0109] According to the relative position relationship and the defined initial direction, a linkage relationship between the indoor environment and the camera is established in the camera coordinate system.

[0110] an adjustment module 43, configured to determine a first rotation angle in response to a request for adjusting the image viewing angle in the panoramic image;

[0111] Synchronizing the first rotation angle to the point cloud top view;

[0112] Rotate the panoramic image and the point cloud top view simultaneously by the first rotation angle based on the current angle;

[0113] or,

[0114] In response to a request for adjusting the image viewing angle of the point cloud top view, determining a second rotation angle;

[0115] synchronizing the second rotation angle to the panoramic image;

[0116] The panoramic image and the point cloud top view are simultaneously rotated by the second rotation angle based on the current angle.

[0117] An establishing module 42, used for establishing the camera coordinate system with the first shooting position of the camera as the origin of the camera coordinate system;

[0118] Performing depth estimation on the panoramic image to obtain an estimated depth image;

[0119] A point cloud top view is generated based on the estimated depth image and the two-dimensional coordinate information of the panoramic image in the camera coordinate system.

[0120] Establishing a module 42, for identifying the panoramic image using a pre-trained point prediction model, and selecting at least one candidate point from the panoramic image;

[0121] Determining a rotation angle of the candidate point in the panoramic image;

[0122] According to the rotation angle, a predicted position is marked at the candidate point in the point cloud top view.

[0123] Establishing module 42, used to obtain the coordinate information of the current shooting point and the first point cloud data of the current point;

[0124] Determine at least one candidate point according to the distance threshold;

[0125] Calculating the point cloud overlap between the first point cloud data and the second point cloud data according to the second point cloud data corresponding to each of the candidate points;

[0126] According to the point cloud overlap, a point is selected from at least one candidate point as the next shooting point.

[0127] Based on the same idea, the present application also provides another image processing device. Figure 5 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application. Figure 5 It can be seen that the device includes:

[0128] The display module 51 is used to display the panoramic image and the point cloud top view taken by the camera in real time through the image display window at the user end; the image display window includes at least a first display area and a second display area; wherein the first display area displays a partial viewing angle image in the panoramic image collected by the camera in real time, and the second display area displays the point cloud top view collected by the camera in real time;

[0129] The rotation module 52 is used for synchronously performing a rotation operation on the display contents in the first display area and the second display area in response to a rotation operation on the first display area or the second display area.

[0130] Optionally, it also includes: using a pre-trained point prediction model to identify the panoramic image and select at least one candidate point from the panoramic image. A rotation module 52 is used to synchronously perform a rotation operation on the display content of the first display area and the second display area in response to a user's rotation operation in the first display area if at least one of the candidate points is not in the partial viewing angle image; or, if at least one of the candidate points is not in the partial viewing angle image, synchronously perform a rotation operation on the display content of the first display area and the second display area in response to a user's rotation operation in the second display area.

[0131] For ease of understanding, the technical solution of this application will be described below through a complete embodiment. Figure 5 Schematic diagram of an image processing scheme for illustrating an embodiment of the present application. Figure 5 As you can see, the specific generation process is as follows:

[0132] Control the camera to shoot the indoor environment to obtain a panoramic image;

[0133] Establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system;

[0134] Using the relative position relationship between the indoor environment and the camera in the world coordinate system, a linkage relationship between the indoor environment and the camera in the camera coordinate system is established;

[0135] In response to a request for adjusting a shooting angle of the panoramic image or the point cloud overhead view, the panoramic image and the point cloud overhead view are synchronously adjusted according to the linkage relationship.

[0136] Optionally, the establishing a linkage relationship between the indoor environment and the camera in the camera coordinate system by using the relative position relationship between the indoor environment and the camera in the world coordinate system includes:

[0137] Determine a first coordinate of the camera in a world coordinate system, and a second coordinate of the indoor environment in the world coordinate system;

[0138] Calculating the relative position relationship using the first coordinate and the second coordinate;

[0139] According to the relative position relationship and the defined initial direction, a linkage relationship between the indoor environment and the camera is established in the camera coordinate system.

[0140] Optionally, in response to a request for adjusting a shooting angle of the panoramic image or the point cloud overhead view, synchronously adjusting the panoramic image and the point cloud overhead view according to the linkage relationship comprises:

[0141] In response to a request to adjust an image viewing angle in the panoramic image, determining a first rotation angle;

[0142] Synchronizing the first rotation angle to the point cloud top view;

[0143] Rotate the panoramic image and the point cloud top view simultaneously by the first rotation angle based on the current angle;

[0144] or,

[0145] In response to a request for adjusting the image viewing angle of the point cloud top view, determining a second rotation angle;

[0146] synchronizing the second rotation angle to the panoramic image;

[0147] The panoramic image and the point cloud top view are simultaneously rotated by the second rotation angle based on the current angle.

[0148] Optionally, establishing a camera coordinate system and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system includes:

[0149] Establishing the camera coordinate system with the first shooting position of the camera as the origin of the camera coordinate system;

[0150] Performing depth estimation on the panoramic image to obtain an estimated depth image;

[0151] A point cloud top view is generated based on the estimated depth image and the two-dimensional coordinate information of the panoramic image in the camera coordinate system.

[0152] Optionally, it also includes,

[0153] Using a pre-trained point prediction model to identify the panoramic image, and selecting at least one candidate point from the panoramic image;

[0154] Determining a rotation angle of the candidate point in the panoramic image;

[0155] According to the rotation angle, a predicted position is marked at the candidate point in the point cloud top view.

[0156] Optionally, the using a pre-trained point prediction model to identify the panoramic image and selecting at least one candidate point from the panoramic image includes:

[0157] Get the coordinate information of the current shooting point and the first point cloud data of the current point;

[0158] Determine at least one candidate point according to the distance threshold;

[0159] Calculating the point cloud overlap between the first point cloud data and the second point cloud data according to the second point cloud data corresponding to each of the candidate points;

[0160] According to the point cloud overlap, a point is selected from at least one candidate point as the next shooting point.

[0161] Optionally, the panoramic image and the point cloud overhead view taken by the camera are displayed in real time through an image display window at the user end; the image display window includes at least a first display area and a second display area; wherein the first display area displays a partial viewing angle image in the panoramic image collected by the camera in real time, and the second display area displays the point cloud overhead view collected by the camera in real time;

[0162] In response to a rotation operation on the first display area or the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

[0163] Optionally, the panoramic image is identified using a pre-trained point prediction model, and at least one candidate point is selected from the panoramic image;

[0164] In response to the rotation operation on the first display area or the second display area, synchronously performing the rotation operation on the display contents in the first display area and the second display area, comprises:

[0165] If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the first display area, synchronously performing a rotation operation on display contents in the first display area and the second display area;

[0166] or,

[0167] If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

[0168] Figure 6 The structure diagram of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 6 As shown, it includes: a memory 60a and a processor 60b; wherein the memory 60a is used to store a computer program; the processor 60b is coupled to the memory 60a, and is used to execute the computer program to perform the following steps:

[0169] Control the camera to shoot the indoor environment to obtain a panoramic image;

[0170] Establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system;

[0171] Using the relative position relationship between the indoor environment and the camera in the world coordinate system, a linkage relationship between the indoor environment and the camera in the camera coordinate system is established;

[0172] In response to a request for adjusting a shooting angle of the panoramic image or the point cloud overhead view, the panoramic image and the point cloud overhead view are synchronously adjusted according to the linkage relationship.

[0173] The processor 60b is used to determine a first coordinate of the camera in a world coordinate system and a second coordinate of the indoor environment in the world coordinate system;

[0174] Calculating the relative position relationship using the first coordinate and the second coordinate;

[0175] According to the relative position relationship and the defined initial direction, a linkage relationship between the indoor environment and the camera is established in the camera coordinate system.

[0176] The processor 60b is configured to determine a first rotation angle in response to a request for adjusting the image viewing angle in the panoramic image;

[0177] Synchronizing the first rotation angle to the point cloud top view;

[0178] Rotate the panoramic image and the point cloud top view simultaneously by the first rotation angle based on the current angle;

[0179] or,

[0180] In response to a request for adjusting the image viewing angle of the point cloud top view, determining a second rotation angle;

[0181] synchronizing the second rotation angle to the panoramic image;

[0182] The panoramic image and the point cloud top view are simultaneously rotated by the second rotation angle based on the current angle.

[0183] The processor 60b is used for establishing the camera coordinate system with the first shooting position of the camera as the origin of the camera coordinate system;

[0184] Performing depth estimation on the panoramic image to obtain an estimated depth image;

[0185] A point cloud top view is generated based on the estimated depth image and the two-dimensional coordinate information of the panoramic image in the camera coordinate system.

[0186] The processor 60b is used to identify the panoramic image using a pre-trained point prediction model, and select at least one candidate point from the panoramic image;

[0187] Determining a rotation angle of the candidate point in the panoramic image;

[0188] According to the rotation angle, a predicted position is marked at the candidate point in the point cloud top view.

[0189] The processor 60b is used to obtain the coordinate information of the current shooting point and the first point cloud data of the current point;

[0190] Determine at least one candidate point according to the distance threshold;

[0191] Calculating the point cloud overlap between the first point cloud data and the second point cloud data according to the second point cloud data corresponding to each of the candidate points;

[0192] According to the point cloud overlap, a point is selected from at least one candidate point as the next shooting point.

[0193] The processor 60b is used to display the panoramic image and the point cloud overhead view taken by the camera in real time through the image display window at the user end; the image display window includes at least a first display area and a second display area; wherein the first display area displays a partial viewing angle image in the panoramic image collected by the camera in real time, and the second display area displays the point cloud overhead view collected by the camera in real time;

[0194] In response to a rotation operation on the first display area or the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

[0195] The processor 60b is used to identify the panoramic image using a pre-trained point prediction model, and select at least one candidate point from the panoramic image;

[0196] In response to the rotation operation on the first display area or the second display area, synchronously performing the rotation operation on the display contents in the first display area and the second display area, comprises:

[0197] If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the first display area, synchronously performing a rotation operation on display contents in the first display area and the second display area;

[0198] or,

[0199] If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

[0200] It should be noted that the specific implementation of the above modules or units provided in this embodiment can refer to the above Figure 1 The relevant description of the illustrated method embodiment will not be repeated here.

[0201] An exemplary embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes one or more processors to execute the steps in the above-mentioned electronic contract processing method. The detailed implementation process of each step can be found in the relevant description of the above-mentioned embodiments, which will not be repeated here.

[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0203] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0207] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0208] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0209] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0210] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Control the camera to shoot the indoor environment to obtain a panoramic image; Establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system; Using the relative position relationship between the indoor environment and the camera in the world coordinate system, a linkage relationship between the indoor environment and the camera in the camera coordinate system is established; In response to a request for adjusting a shooting angle of the panoramic image or the point cloud overhead view, the panoramic image and the point cloud overhead view are synchronously adjusted according to the linkage relationship.

2. The method according to claim 1, characterized in that: The step of establishing a linkage relationship between the indoor environment and the camera in the camera coordinate system by using the relative position relationship between the indoor environment and the camera in the world coordinate system includes: Determine a first coordinate of the camera in a world coordinate system, and a second coordinate of the indoor environment in the world coordinate system; Calculating the relative position relationship using the first coordinate and the second coordinate; According to the relative position relationship and the defined initial direction, a linkage relationship between the indoor environment and the camera is established in the camera coordinate system.

3. The method according to claim 1, characterized in that In response to a request for adjusting the shooting angle of the panoramic image or the point cloud overhead view, synchronously adjusting the panoramic image and the point cloud overhead view according to the linkage relationship includes: In response to a request to adjust an image viewing angle in the panoramic image, determining a first rotation angle; Synchronizing the first rotation angle to the point cloud top view; Rotate the panoramic image and the point cloud top view simultaneously by the first rotation angle based on the current angle; or, In response to a request for adjusting the image viewing angle of the point cloud top view, determining a second rotation angle; synchronizing the second rotation angle to the panoramic image; The panoramic image and the point cloud top view are simultaneously rotated by the second rotation angle based on the current angle.

4. The method according to claim 1, characterized in that: The step of establishing a camera coordinate system and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system includes: Establishing the camera coordinate system with the first shooting position of the camera as the origin of the camera coordinate system; Performing depth estimation on the panoramic image to obtain an estimated depth image; A point cloud top view is generated based on the estimated depth image and the two-dimensional coordinate information of the panoramic image in the camera coordinate system.

5. The method according to claim 1, characterized in that: Also includes, Using a pre-trained point prediction model to identify the panoramic image, and selecting at least one candidate point from the panoramic image; Determining a rotation angle of the candidate point in the panoramic image; According to the rotation angle, a predicted position is marked at the candidate point in the point cloud top view.

6. The method according to claim 5, characterized in that The step of identifying the panoramic image by using a pre-trained point prediction model and selecting at least one candidate point from the panoramic image includes: Get the coordinate information of the current shooting point and the first point cloud data of the current point; Determine at least one candidate point according to the distance threshold; Calculating the point cloud overlap between the first point cloud data and the second point cloud data according to the second point cloud data corresponding to each of the candidate points; According to the point cloud overlap, a point is selected from at least one candidate point as the next shooting point.

7. An image processing method, characterized in that: Applied to a user terminal equipped with a camera, the method comprises: The panoramic image and the point cloud overhead view captured by the camera are displayed in real time on the user terminal through an image display window; the image display window includes at least a first display area and a second display area; wherein the first display area displays a partial viewing angle image in the panoramic image captured by the camera in real time, and the second display area displays the point cloud overhead view captured by the camera in real time; In response to a rotation operation on the first display area or the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

8. The method according to claim 7, characterized in that Also includes: Using a pre-trained point prediction model to identify the panoramic image, and selecting at least one candidate point from the panoramic image; In response to the rotation operation on the first display area or the second display area, synchronously performing the rotation operation on the display contents in the first display area and the second display area, comprises: If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the first display area, synchronously performing a rotation operation on display contents in the first display area and the second display area; or, If at least one of the candidate points is not in the partial viewing angle image, in response to a user's rotation operation in the second display area, a rotation operation is synchronously performed on display contents in the first display area and the second display area.

9. An image processing device, characterized in that: The device comprises: A control module, used to control the camera to capture the indoor environment and obtain a panoramic image; An establishing module, used for establishing a camera coordinate system, and drawing a point cloud top view corresponding to the panoramic image based on the camera coordinate system; The establishing module is used to establish a linkage relationship between the indoor environment and the camera in the camera coordinate system by using the relative position relationship between the indoor environment and the camera in the world coordinate system; An adjustment module is used to respond to a request for adjusting the shooting angle of the panoramic image or the point cloud overhead view, and synchronously adjust the panoramic image and the point cloud overhead view according to the linkage relationship.

10. An electronic device, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; and the processor is coupled to the memory and is used to execute the computer program to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 8.

Images