Point cloud data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a point cloud data processing method and device, equipment and a storage medium. The method comprises the following steps: in response to a shooting instruction input by one or more input components, obtaining three-dimensional point cloud data of a real scene under a real-time shooting view angle, so as to display a corresponding real scene point cloud image by a display generation component; and according to the shot visual angle of the real scene, presenting corresponding visual angle supplementary shooting information in the real scene point cloud image to obtain three-dimensional point cloud data of the real scene under the full visual angle, and constructing a corresponding real scene model. According to the embodiment of the invention, full-view-angle shooting of the real scene can be realized, so that the real scene model supports browsing and viewing at any view angle, the immersive browsing experience of a user on the real scene model is ensured, the spatial visual effect displayed after shooting of the real scene is improved, and the three-dimensional shooting performance of the real scene is enhanced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular to a point cloud data processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] At present, the application scenarios of extended reality (XR) technology are more and more extensive, which specifically includes virtual reality (VR), augmented reality (AR), and mixed reality (MR), etc.

[0003] Generally, an XR device provides a corresponding shooting function for a user to shoot and save a corresponding object in a real scene for later browsing and viewing. The XR device supports shooting binocular images in a real scene and displaying the binocular images through a display screen corresponding to each eye to achieve a three-dimensional visual effect of the real scene after shooting and display.

[0004] However, the viewing angle provided by the image shot by the XR device in the above manner is fixed when the image is displayed, resulting in that the three-dimensional shooting function on the XR device only supports a spatial display effect in a single viewing angle, and there is a certain spatial display limitation. SUMMARY

[0005] Embodiments of the present application provide a point cloud data processing method and device, an electronic device, and a storage medium, which realize full-view shooting of a real scene, improve the spatial visual effect of the real scene after shooting and display, and enhance the three-dimensional shooting performance of the real scene.

[0006] In a first aspect, embodiments of the present application provide a point cloud data processing method applied to an electronic device in communication with a display generation component and one or more input components. The method comprises:

[0007] In response to a shooting instruction input via the one or more input components, three-dimensional point cloud data of a real scene in a real-time shooting viewing angle is acquired to display a corresponding real scene point cloud image via the display generation component;

[0008] According to a shot viewing angle of the real scene, corresponding viewing angle retake information is presented in the real scene point cloud image to acquire three-dimensional point cloud data of the real scene in a full-view angle, and a corresponding real scene model is constructed.

[0009] In a second aspect, embodiments of the present application provide a point cloud data processing device configured in an electronic device in communication with a display generation component and one or more input components. The device comprises:

[0010] a point cloud obtaining module, configured to, in response to a shooting instruction input via the one or more input components, obtain three-dimensional point cloud data of a real scene under a real-time shooting perspective, to display a corresponding real scene point cloud image via the display generation component;

[0011] a retake prompting module, configured to, according to a shot perspective of the real scene, present corresponding perspective retake information in the real scene point cloud image, to obtain three-dimensional point cloud data of the real scene under a full perspective, and construct a corresponding real scene model.

[0012] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0013] a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory, to execute the point cloud data processing method provided in the first aspect.

[0014] In a fourth aspect, a computer readable storage medium is provided, configured to store a computer program, and the computer program causes a computer to execute the point cloud data processing method provided in the first aspect.

[0015] In a fifth aspect, a computer program product is provided, comprising computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the point cloud data processing method provided in the first aspect.

[0016] The technical solution provided in the embodiments of the present application can obtain three-dimensional point cloud data of a real scene under a real-time shooting perspective when a shooting instruction input via one or more input components is detected, and display a corresponding real scene point cloud image via a display generation component. Moreover, during real-time shooting of the real scene, corresponding perspective retake information can be presented in the displayed real scene point cloud image according to a shot perspective of the real scene, to obtain three-dimensional point cloud data of the real scene under a full perspective, and construct a corresponding real scene model, thereby realizing full perspective shooting of the real scene, enabling the real scene model to support browsing and viewing under any perspective, ensuring immersive browsing experience of the real scene model for a user, improving spatial visual effects of the real scene after shooting, and enhancing three-dimensional shooting performance of the real scene. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0018] Figure 1 A flowchart of a point cloud data processing method provided by an embodiment of the present application is shown in FIG. 1.

[0019] Figure 2 A flowchart of a specific presentation process of view angle retake information in a real scene point cloud image provided by an embodiment of the present application is shown in FIG. 2.

[0020] Figure 3 A flowchart of a specific shooting process of a real scene for shooting a corresponding spatial image provided by an embodiment of the present application is shown in FIG. 3.

[0021] Figure 4 A flowchart of a specific shooting process of a real scene for shooting a corresponding spatial video provided by an embodiment of the present application is shown in FIG. 4.

[0022] Figure 5 A principle block diagram of a point cloud data processing device provided by an embodiment of the present application is shown in FIG. 5.

[0023] Figure 6 A schematic block diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0025] It should be noted that the terms “first”, “second”, and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0026] In the embodiments of this application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration, and should not be employed to denote that a preferred or advantageous embodiment is being presented. In the embodiments of this application, any embodiment or design described as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs.

[0027] Before introducing the specific technical solutions of this application, the application scenarios of this application are first described as follows:

[0028] In order to ensure the three-dimensional shooting performance of the real scene, the display generation component in communication with any electronic device can be used to display the real scene image in real time during the real-time shooting of the real scene. The electronic device can be any XR device, specifically including VR devices, AR devices, MR devices, etc., which are not limited in this application.

[0029] Among them, the display generation component can be any display screen in communication with the electronic device. For example, the display generation component can be a display screen configured on the VR device, AR device or MR device, which is not limited in this application.

[0030] Moreover, in order to realize the normal shooting interaction of the real scene, one or more input components in communication with the electronic device can be used to initiate corresponding shooting instructions for the real scene, thereby supporting the user to perform efficient and accurate three-dimensional shooting on the real scene using the electronic device.

[0031] Among them, the one or more input components can be any control device and information acquisition module in communication with the electronic device. For example, the one or more input components can be a handle configured on the VR device, AR device or MR device, or an acquisition module for detecting hand operation and eye movement information, or a voice acquisition device for collecting user voice information, etc., which are not limited in this application.

[0032] To solve the problem that the three-dimensional shooting function of the XR device currently only supports the spatial display effect in a single perspective, the application concept is that, when a shooting instruction input via one or more input components is detected, three-dimensional point cloud data of a real scene in a real-time shooting perspective can be acquired, and a corresponding real scene point cloud image can be displayed via a display generation component. Furthermore, during real-time shooting of the real scene, corresponding perspective retake information can be presented in the displayed real scene point cloud image according to the already shot perspective of the real scene, so as to acquire three-dimensional point cloud data of the real scene in a full perspective, construct a corresponding real scene model, and thus realize full-perspective shooting of the real scene, improve the spatial visual effect of the real scene after shooting, and enhance the three-dimensional shooting performance of the real scene.

[0033] Figure 1 A flowchart of a point cloud data processing method provided by the application is provided, which can be applied to an XR device, but is not limited thereto. The method can be executed by a point cloud data processing apparatus provided by the application, wherein the point cloud data processing apparatus can be implemented in any software and / or hardware manner. Exemplarily, the point cloud data processing apparatus can be configured in an electronic device such as an AR / VR / MR device that can support three-dimensional shooting of a real scene, and the application does not make any limitation on the specific type of the electronic device.

[0034] Specifically, as shown in the method can include the following steps: Figure 1

[0035] S110, in response to a shooting instruction input via one or more input components, acquiring three-dimensional point cloud data of a real scene in a real-time shooting perspective, and displaying a corresponding real scene point cloud image via a display generation component.

[0036] The real scene can be a surrounding real environment in which a user is located. The XR device has a corresponding shooting function to support the user to shoot the real scene in which the user is located, so as to record the changes of the real scene. Specifically, the XR device can have two-dimensional shooting function and three-dimensional shooting function, and the implementation of the two-dimensional shooting function is similar to the two-dimensional shooting function of a mobile phone, a camera or other electronic device. Therefore, the application mainly describes the specific implementation process of the three-dimensional shooting function of the XR device.

[0037] In the application, the user can trigger the shooting function on the XR device through one or more input components to generate a corresponding shooting instruction.

[0038] The shooting instruction input via one or more input components can be determined in at least one of the following ways:

[0039] ​1) Triggering a corresponding shooting function via a user handle to generate a corresponding shooting instruction.

[0040] The user can manipulate a specific key or joystick on any handle in communication connection with the above-mentioned XR device to trigger a corresponding shooting function, so as to generate corresponding shooting request information and send it to the XR device. When the XR device receives the shooting request information input via the user handle, it indicates that the user currently wants to shoot the real scene, thereby generating a corresponding shooting instruction.

[0041] 2) Generating a corresponding shooting instruction via a shooting gesture initiated by the user's hand detected by the hand detection component.

[0042] The XR device can be equipped with a corresponding hand detection component to detect various gesture information made by the user's hand in real time. For example, the hand image of the user can be collected in real time through the camera on the XR device. Then, the hand detection component analyzes the features of the hand image to identify various gesture information initiated by the user's hand. In order to ensure the fast shooting of the XR device, a corresponding shooting gesture can be preset for the shooting function of the XR device. When the hand detection component on the XR device detects a shooting gesture initiated by the user's hand, it indicates that the user currently wants to shoot the real scene, thereby generating a corresponding shooting instruction.

[0043] 3) Generating a corresponding shooting instruction via the preset eye movement information initiated by the user's eye detected by the eye detection component and the shooting gesture initiated by the hand detected by the hand detection component.

[0044] The XR device can be equipped with a corresponding eye detection component and hand detection component to respectively detect various eye movement information initiated by the user's eye and various gesture information made by the user's hand. For example, the eye image of the user and the hand image of the user can be collected in real time through the camera on the electronic device. Then, the eye detection component analyzes the features of the eye image to identify various eye movement information initiated by the user's eye. Moreover, the hand detection component analyzes the features of the hand image to identify various gesture information initiated by the user's hand. When the eye detection component on the XR device detects preset eye movement information initiated by the user's eye, and the hand detection component on the XR device simultaneously detects a shooting gesture initiated by the user's hand, it indicates that the user currently wants to shoot the real scene, thereby generating a corresponding shooting instruction.

[0045] 4) Generating a corresponding shooting instruction via voice shooting information collected by any voice collection component.

[0046] Any voice collection component installed on the XR device can collect audio data uttered by the user in real time and perform corresponding voice recognition processing on the audio data to identify corresponding voice information. When the XR device identifies voice shooting information collected via any voice collection component, it indicates that the user currently wants to shoot the real scene, thereby generating a corresponding shooting instruction.

[0047] Therefore, the present application can determine whether there is a shooting requirement for the real scene by detecting various operation instructions input via one or more input components. When a shooting instruction input via one or more input components is detected, it indicates that the user currently has a shooting requirement for the real scene. Then, the XR device can use the three-dimensional scanning device (such as an RGB-D camera) configured therein to shoot the real scene in which the user is currently located in real time, thereby scanning and obtaining three-dimensional point cloud data of the real scene under the real-time shooting perspective.

[0048] The three-dimensional point cloud data can include, but is not limited to, position information (x, y, z), rotation information (x, y, z, w), color information (r, g, b), depth information (d), and camera pose information of each spatial point of the real scene under the real-time shooting perspective.

[0049] Then, during the real-time shooting of the real scene, the present application can perform corresponding integration processing on the three-dimensional point cloud data of the real scene under the real-time shooting perspective to construct a corresponding real scene point cloud image, and display each real scene point cloud image shot in real time via a display generation component.

[0050] S120, according to the already shot perspective of the real scene, presenting corresponding perspective retake information in the real scene point cloud image to obtain three-dimensional point cloud data of the real scene under the full perspective, and constructing a corresponding real scene model.

[0051] Considering that each real scene point cloud image shot in real time usually only supports spatial display effect under a single real-time shooting perspective when displayed, there is still a certain spatial display limitation. Then, in order to realize full-perspective shooting of the real scene, the present application can determine the already shot perspective of the real scene by analyzing the actual shooting perspective corresponding to each three-dimensional point cloud data obtained by the real scene before the shooting time of the real scene point cloud image when displaying each real scene point cloud image in the real-time shooting process via the display generation component.

[0052] Then, for each real scene point cloud image displayed by the display generation component in the real-time shooting process of the real scene, the application can determine the corresponding unshot perspective by analyzing the shot perspective of the real scene at the shooting time of the real scene point cloud image, thereby generating the perspective retake information of the real scene point cloud image at the shooting time, and presenting it in the real scene point cloud image to prompt the user to shoot each unshot perspective of the real scene, and to obtain three-dimensional point cloud data of the real scene under full perspective, thereby constructing a corresponding real scene model.

[0053] Further, the real scene model constructed in the real-time shooting process of the real scene can be stored as a corresponding three-dimensional shooting object. Moreover, the thumbnail of the three-dimensional shooting object after storage can be a reduced version of the real scene model, to support accurate display of the three-dimensional shooting object.

[0054] The technical scheme provided by the embodiments of the application can obtain three-dimensional point cloud data of the real scene under the real-time shooting perspective, and display a corresponding real scene point cloud image via the display generation component when detecting a shooting instruction input via one or more input components. Moreover, during the real-time shooting process of the real scene, the corresponding perspective retake information can be presented in the displayed real scene point cloud image according to the shot perspective of the real scene, to obtain three-dimensional point cloud data of the real scene under full perspective, and to construct a corresponding real scene model, thereby realizing full-perspective shooting of the real scene, allowing the real scene model to support browsing and viewing under any perspective, ensuring immersive browsing experience of the user for the real scene model, improving the spatial visual effect of the real scene after shooting, and enhancing the three-dimensional shooting performance of the real scene.

[0055] As an optional implementation in the application, considering that there can be multiple real objects in the real scene. Then, for the three-dimensional point cloud data of the real scene under the real-time shooting perspective, it can specifically include three-dimensional point cloud data of each real object in the real scene under the real-time shooting perspective, so that the constructed real scene model needs to be composed of real object models of each real object in the real scene.

[0056] Then, in order to ensure accurate shooting of the real scene under full perspective, the application can construct a complete real scene model by prompting the user about the specific retake situation of each real object in the real scene. Therefore, as shown in Figure 2 The specific process of presenting the corresponding perspective retake information in the real scene point cloud image can be explained in detail.

[0057] Figure 2A method flowchart for a specific presentation process of view angle retake information in a real scene point cloud image provided by an embodiment of the present application. The method can specifically include the following steps:

[0058] In S210, a shot view angle of each real object is determined according to a point cloud state of each real object in the real scene.

[0059] When each real scene point cloud image in a real-time shooting process of a real scene is displayed via a display generation component, the shot view angle of each real object can be determined by analyzing actual shot view angles corresponding to each three-dimensional point cloud data acquired by each real object in the real scene before a shooting time of the real scene point cloud image.

[0060] In S220, corresponding view angle retake information is presented at an associated position of each real object in the real scene point cloud image according to the shot view angle of each real object, so as to construct a real object model of each real object under a full view angle and compose a corresponding real scene model.

[0061] For each real scene point cloud image of a real scene displayed via a display generation component in a real-time shooting process, the shot view angle of each real object at a shooting time of the real scene point cloud image can be analyzed to determine a shot view angle of each real object at this time, so as to generate view angle retake information of each real object.

[0062] Then, the associated position of each real object is determined by analyzing the position of each real object in the real scene point cloud image, so that the associated position of each real object can be explicitly indicated to the real object. Then, in order to ensure full view angle shooting of the real scene, the view angle retake information of each real object can be presented at the associated position of each real object in the real scene point cloud image, so as to intuitively prompt a user to shoot each unshot view angle of each real object, to acquire three-dimensional point cloud data of each real object under a full view angle, to construct each real object model, and to compose a corresponding real scene model.

[0063] That is, each real object model of each real object can be one-to-one corresponding distributed in the real scene model according to the actual distribution of each real object in the real scene, to support full view angle browsing of each real object model by the user.

[0064] According to one or more embodiments provided in the present application, considering that the shooting function of the XR device for the real scene can be divided into picture shooting and video shooting, then the XR device can shoot the corresponding spatial image and spatial video for the real scene. Next, the present application can explain the specific shooting process of the spatial image and spatial video respectively under the following two situations.

[0065] Situation one, the shooting instruction input through one or more input components is an image shooting instruction

[0066] Figure 3 The method flow chart for the specific shooting process of the spatial image for the real scene provided by the embodiments of the present application is shown in the figure. Figure 3 As shown in the figure, the method can specifically include the following steps:

[0067] S310, in response to the image shooting instruction input through one or more input components, acquiring the three-dimensional point cloud data of the real scene under the real-time shooting angle, to display the corresponding real scene point cloud image through the display generation component.

[0068] S320, according to the already shot angle of the real scene, presenting the corresponding angle retake information in the real scene point cloud image to acquire the three-dimensional point cloud data of the real scene under the full angle, and constructing the corresponding real scene model.

[0069] S330, displaying the real scene model through the display generation component, and storing it as the corresponding spatial image.

[0070] After constructing the real scene model, the present application can take the real scene model as the spatial image for shooting the real scene. Then, by directly displaying the real scene model through the display generation component, the user can preview the spatial image for this time shooting of the real scene, so as to judge whether there is shooting error.

[0071] When the real scene model does not exist shooting error, the real scene model can be directly stored as the corresponding spatial image, and the thumbnail of the spatial image when stored can be the reduced version of the real scene model. When the real scene model exists shooting error, the real scene model for this time shooting can be abandoned, and the above steps can be re-executed, so as to make the user shoot new spatial image for the real scene again, and ensure the accurate shooting of the real scene.

[0072] When the user wants to view any of the photographed space images, the user can trigger the display instruction of the space image by one or more input components to generate the thumbnail of the space image. Then, in response to the display instruction input by the one or more input components for any space image, the application can load the three-dimensional point cloud data of the space image under the full view angle to construct the corresponding real scene model, so as to realize the accurate display of the space image.

[0073] It can be understood that for any space image, the initial shooting view angle of the space image can be taken as the default display view angle to accurately display the space image, so that the space image can display the best space visual effect.

[0074] Case two, the shooting instruction input through the one or more input components is a video shooting instruction

[0075] Figure 4 The method flowchart provided by the embodiment of the application for the specific shooting process of the space video corresponding to the real scene shooting. As shown in Figure 4 The method can specifically include the following steps:

[0076] S410, in response to the video shooting instruction input through the one or more input components, acquiring the three-dimensional point cloud data of the real scene under the real-time shooting view angle to display the corresponding real scene point cloud image through the display generation component.

[0077] S420, presenting the corresponding view angle supplement shooting information in the real scene point cloud image according to the photographed view angle of the real scene to acquire the three-dimensional point cloud data of the real scene under the full view angle and construct the corresponding real scene model.

[0078] S430, determining the point cloud index relationship of each rigid real object in the real scene according to the three-dimensional point cloud data of the real scene under the full view angle.

[0079] It can be understood that in the real-time shooting process of the real scene, the application can support the user to perform the full view angle shooting on the real scene at the beginning of the shooting, and then perform the single view angle shooting. It can also not consider the shooting view angle, but perform the free view angle shooting during the shooting, so as to constantly update and improve the photographed view angle of the real scene, so that the view angle supplement shooting information presented in each real scene point cloud image of the real-time shooting is constantly updated until the full view angle shooting of the real scene is completed before the shooting is stopped. The application does not limit the specific video shooting mode of the real scene.

[0080] From the above, no matter which video shooting mode is adopted, the three-dimensional point cloud data of the real scene under the full view angle can be obtained before the video shooting of the real scene is completed. Then, by analyzing the three-dimensional point cloud data of the real scene under the full view angle, the three-dimensional point cloud data of each real object in the real scene under the full view angle can be separated.

[0081] For the rigid real object in the real scene, the shape and size of the rigid real object usually do not change. Then, during the real-time shooting of the real scene, as long as the three-dimensional point cloud data of a rigid real object under the full view angle is obtained at a certain shooting moment, the view angle retake information of the rigid real object will not be prompted in the subsequent shooting process.

[0082] In this application, by analyzing the three-dimensional point cloud data of each rigid real object under the full view angle, the point cloud index relationship of the rigid real object can be determined, which can describe the specific distribution of the object points on the rigid real object and will not change.

[0083] For the non-rigid real object in the real scene, the shape and size of the non-rigid real object usually change during the real-time shooting of the real scene. Then, during the real-time shooting of the real scene, this application can obtain the three-dimensional point cloud data of each non-rigid real object under the full view angle according to the view angle retake information of each non-rigid real object at each shooting moment, so as to accurately construct the real object model of the non-rigid real object at the shooting moment and ensure the accurate shooting of the spatial video.

[0084] S440, in response to the video shooting end instruction input via one or more input components, the three-dimensional point cloud data obtained at each shooting moment of the real scene is completed according to the point cloud index relationship of each rigid real object in the real scene, the real scene model at each shooting moment is constructed, and stored as the corresponding spatial video.

[0085] During the real-time shooting of the real scene, this application can detect various operation instructions input via one or more input components in real time to determine whether there is a demand to end the video shooting of the real scene. When the video shooting end instruction input via one or more input components is detected, this application can control the XR device to directly exit the video shooting of the real scene and determine the three-dimensional point cloud data obtained at each shooting moment of the real scene.

[0086] It can be understood that, considering that the video shooting mode of the real scene is different, the three-dimensional point cloud data of some non-rigid real object in the real scene at a certain shooting time may not be the three-dimensional point cloud data under full view. Therefore, in order to ensure accurate shooting of the space video of the real scene, the present application can construct a panoramic model of each rigid real object in the real scene according to the panoramic point cloud distribution represented by the point cloud index relationship of each rigid real object in the real scene. Then, based on the panoramic model of each rigid real object, the missing part of the point cloud of each rigid real object in the three-dimensional point cloud data obtained by the real scene at each shooting time can be completed under full view, so as to obtain the full-view three-dimensional point cloud data of the real scene at each shooting time, and to construct the real scene model at each shooting time. Further, the real scene model at each shooting time in the real-time shooting process of the real scene is integrated in sequence, so that the space video for shooting the real scene is obtained and stored in the XR device. The thumbnail of the stored space video can be a reduced version of the real scene model at any time in the space video.

[0087] Then, when the user wants to view any space video that has been shot, the user can trigger the thumbnail of the space video to generate a display instruction of the space video through one or more input components. Then, in response to the display instruction input to any space video through one or more input components, the present application can load the full-view three-dimensional point cloud data of each time corresponding to the space video in real time to construct and display the real scene model at each time in real time, so as to realize accurate display of the space video.

[0088] It should be noted that, considering that the space image and the space video are both constructed by the three-dimensional point cloud data of the real scene under full view. Therefore, the space image and the space video can support display under fixed view, following view and free view.

[0089] Among them, the fixed view can be a certain best shooting view set when shooting the space image and the space video. The following view can be to display following the real-time shooting view of the space image and the space video. The free view can be that the user can perform relative rotation of free angle and relative displacement of free position in the space image and the space video to display the space image and the space video from various views.

[0090] In addition, for the display of the spatial image and the spatial video shot by the real scene, it is the display of the real scene model constructed in the real-time shooting process of the real scene. Therefore, in order to ensure the diversified display of the real scene model, the present application can support the user to adjust any real object model in the displayed real scene model. Therefore, in response to the adjustment operation input to any real object model in the real scene model via one or more input components, the three-dimensional point cloud data of the real object under the full view angle is updated to adjust the real object model.

[0091] That is, the present application can detect various operation instructions input via one or more input components in real time to determine whether there is an adjustment requirement for any real object model in the real scene model. When detecting the adjustment operation input to any real object model in the real scene model via one or more input components, the present application can control the real object model to enter the adjustment mode, which can support the user to adjust any part of the real object model. Therefore, the present application analyzes the specific adjustment operation acting on the real object model to determine the adjustment part and the adjustment degree of the real object model, so as to modify the three-dimensional point cloud data of the real object under the full view angle. And according to the modified three-dimensional point cloud data of the real object under the full view angle, a new real object model is constructed, so as to realize the accurate adjustment of the real object model.

[0092] It can be understood that through the above adjustment operation, the user can optimize the defect part in any real object model or remove the redundant part in any real object model.

[0093] In addition, in order to ensure the independent operation of any real object model in the real scene model, the present application can support the segmentation and independent storage of each real object model in the displayed real scene model. Therefore, in response to the marking operation input to any real object model in the real scene model via one or more input components, the real object model is segmented from the real scene model according to the three-dimensional point cloud data of the real object under the full view angle, and the real object model is independently stored.

[0094] That is, the present application can detect various operation instructions input via one or more input components in real time to determine whether there is a segmentation requirement for any real object model in the real scene model. When detecting a marking operation input via one or more input components on any real object model in the real scene model, the present application can filter out the three-dimensional point cloud data of the real object in the full view of the real scene from the three-dimensional point cloud data of the real scene in the full view according to the specific marking information of the real object model, thereby independently constructing the real object model, so that the real object model is segmented from the real scene model. Then, each segmented real object model can be independently stored for subsequent individual operation on the segmented real object model. For example, the present application can use an item recognition technology, such as a Normal Aligned Radial Feature (NARF) algorithm, to automatically recognize and segment the three-dimensional point cloud data of any real object in the full view from the three-dimensional point cloud data of the real scene in the full view, to independently construct the corresponding real object model.

[0095] Wherein, the marking operation input via one or more input components on any real object model can be continuously marking a certain real object model from the real scene model by a handle, a gesture or an eye gaze point.

[0096] In some implementations, for the real scene model or any real object model segmented from the real scene model, in order to improve the scalability of the real scene after being photographed in three-dimensional space, the present application can also support sharing the real scene model or any segmented real object model.

[0097] In the first aspect, when detecting a first sharing instruction input via one or more input components on the real scene model or any real object model in the real scene model, the present application can share the real scene model or the real object model to a designated device.

[0098] Wherein, the first sharing instruction input via one or more input components on the real scene model or any real object model in the real scene model can include a shared designated device, which can include but is not limited to an XR device used by any other user, a three-dimensional printer, etc.

[0099] Therefore, by sharing the real scene model or any real object model in the real scene model to a designated device, the present application realizes the whole or partial sharing of the real scene after being photographed in three-dimensional space, and supports the designated device to perform various operations on the shared real scene model or the real object model, such as displaying on the designated device, three-dimensional printing, etc.

[0100] In a second aspect, the real scene model or any real object model in the real scene model can be fused with a target virtual model when a second sharing instruction input via one or more input components is detected.

[0101] The second sharing instruction input via one or more input components to the real scene model or any real object model in the real scene model can include a shared target virtual scene, and the target virtual model can include both a virtual scene model and a virtual object model.

[0102] Therefore, by sharing the real scene model or any real object model in the real scene model to the target virtual scene, the real scene model or the real object model can be fused with the target virtual scene again to realize secondary creation of virtual and real combination and enhance diversified creation of the real scene model or any real object model in the real scene model.

[0103] Figure 5 A principle block diagram of a point cloud data processing device provided by an embodiment of the present application is shown in FIG. 5. The point cloud data processing device 500 can be configured at an electronic device in communication with a display generation component and one or more input components. As shown in FIG. 5, the device 500 can include: Figure 5

[0104] A point cloud acquisition module 510 is configured to acquire three-dimensional point cloud data of a real scene under a real-time shooting angle in response to a shooting instruction input via the one or more input components, so as to display a corresponding real scene point cloud image via the display generation component.

[0105] A retake prompting module 520 is configured to present corresponding angle retake information in the real scene point cloud image according to a shot angle of the real scene, so as to acquire three-dimensional point cloud data of the real scene under a full angle and construct a corresponding real scene model.

[0106] In some implementable manners, the retake prompting module 520 can be specifically configured to:

[0107] determine a shot angle of each real object in the real scene according to a point cloud state of each real object in the real scene;

[0108] present corresponding angle retake information at an associated position of each real object in the real scene point cloud image according to the shot angle of each real object, so as to construct a real object model of each real object under a full angle and form a corresponding real scene model.

[0109] ​In some implementable manners, the point cloud data processing apparatus 500 can further comprise a spatial image shooting module and a spatial video shooting module. The spatial image shooting module can be used for:

[0110] If the shooting instruction is an image shooting instruction, the real scene model is displayed via the display generation component, and is stored as a corresponding spatial image.

[0111] The spatial video shooting module can be used for:

[0112] If the shooting instruction is a video shooting instruction, the point cloud index relationship of each rigid real object in the real scene is determined according to the three-dimensional point cloud data of the real scene under full view angle;

[0113] In response to a video shooting end instruction input via the one or more input components, the three-dimensional point cloud data of the real scene at each shooting time is completed under full view angle according to the point cloud index relationship of each rigid real object in the real scene, the real scene model at each shooting time is constructed, and is stored as a corresponding spatial video.

[0114] In some implementable manners, the spatial image and the spatial video support display under fixed view angle, following view angle and free view angle.

[0115] In some implementable manners, the point cloud data processing apparatus 500 can further comprise:

[0116] A model adjustment module is configured to update the three-dimensional point cloud data of the real object under full view angle in response to an adjustment operation input via the one or more input components on any real object model in the real scene model, so as to adjust the real object model.

[0117] In some implementable manners, the point cloud data processing apparatus 500 can further comprise:

[0118] A model segmentation module is configured to segment the real object model from the real scene model according to the three-dimensional point cloud data of the real object under full view angle in response to a marking operation input via the one or more input components on any real object model in the real scene model, and independently store the real object model.

[0119] In some implementable manners, the point cloud data processing apparatus 500 can further comprise:

[0120] A first sharing module is configured to share the real scene model or any real object model in the real scene model to a specified device in response to a first sharing instruction input via the one or more input components on the real scene model or any real object model in the real scene model.

[0121] In some implementations, the point cloud data processing apparatus 500 can further include:

[0122] The second sharing module is configured to, in response to a second sharing instruction input via the one or more input components for the real scene model or any real object model in the real scene model, fuse the real scene model or the real object model with a target virtual model, the target virtual model including a virtual scene model and a virtual object model.

[0123] In the embodiments of the present application, when a shooting instruction input via one or more input components is detected, three-dimensional point cloud data of the real scene under a real-time shooting angle can be acquired, and a corresponding real scene point cloud image can be displayed via a display generation component. Moreover, during real-time shooting of the real scene, corresponding angle supplement shooting information can be presented in the displayed real scene point cloud image according to the already shot angle of the real scene, so as to acquire three-dimensional point cloud data of the real scene under a full angle, construct a corresponding real scene model, and thus realize full angle shooting of the real scene, so that the real scene model supports browsing and viewing under any angle, ensures immersive browsing experience of the user for the real scene model, improves spatial visual effect of the displayed real scene after shooting, and enhances three-dimensional shooting performance of the real scene.

[0124] It should be understood that the apparatus embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no further description is given here. Specifically, Figure 5 The apparatus 500 shown can perform any method embodiment provided by the present application, and the foregoing and other operations and / or functions of each module in the apparatus 500 are respectively for realizing corresponding procedures in each method of the embodiments of the present application. To be brief, no further description is given here.

[0125] The apparatus 500 of the embodiments of the present application is described above in the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be realized in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in a processor and / or instructions of software form, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing performed and completed by a processor, or executed and completed by a combination of hardware and software modules in the code processing processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads information in the memory, and combines the hardware to complete the steps in the above method embodiments.

[0126] Figure 6 is a schematic block diagram of an electronic device provided by an embodiment of the present application.

[0127] As shown in Figure 6 , the electronic device 600 can include:

[0128] The memory 610 is configured to store a computer program and transmit the program code to the processor 620. In other words, the processor 620 can call and run the computer program from the memory 610 to implement the method in the embodiments of the present application.

[0129] For example, the processor 620 can be configured to execute the above-mentioned method embodiments according to the instructions in the computer program.

[0130] In some embodiments of the present application, the processor 620 can include but is not limited to:

[0131] a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like.

[0132] In some embodiments of the present application, the memory 610 includes but is not limited to:

[0133] The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0134] In some embodiments of the present application, the computer program can be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0135] As shown in Figure 6 The electronic device can further include:

[0136] The transceiver 630 can be connected to the processor 620 or the memory 610.

[0137] The processor 620 can control the transceiver 630 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 630 can include a transmitter and a receiver. The transceiver 630 can further include an antenna, and the number of antennas can be one or more.

[0138] It should be understood that the various components within the electronic device are connected by a bus system, which includes, in addition to a data bus, a power bus, a control bus, and a state signal bus.

[0139] The application also provides a computer storage medium, which stores a computer program, and the computer program enables a computer to execute the method of the method embodiment when executed by the computer. Alternatively, the application embodiment also provides a computer program product containing instructions, and the instructions enable the computer to execute the method of the method embodiment when executed by the computer.

[0140] When implemented by using software, the computer program product can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the flow or function of the embodiment of the application in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, digital video disc (DVD)), or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0141] Those of ordinary skill in the art can realize that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0142] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiment is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple devices or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0143] The modules described as separated parts can or can not be physically separated, and the parts shown as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. For example, the functional modules in the embodiments of the present application can be integrated in one processing module, or can be physically separated, or two or more modules can be integrated in one module.

[0144] The above provides only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for processing point cloud data, characterized in that, Applied on an electronic device in communication with a display generation component and one or more input components, the method comprises: in response to a shooting instruction input via the one or more input components, acquiring three-dimensional point cloud data of a real scene under a real-time shooting perspective to display a corresponding real scene point cloud image via the display generation component; according to the shot perspective of the real scene, presenting corresponding perspective fill-in information in the real scene point cloud image to acquire three-dimensional point cloud data of the real scene under a full perspective, and constructing a corresponding real scene model.

2. The method of claim 1, wherein, According to the shot perspective of the real scene, the corresponding perspective fill-in information is presented in the real scene point cloud image, comprising: determining the shot perspective of each real object in the real scene according to the point cloud state of each real object in the real scene; according to the shot perspective of each real object, presenting corresponding perspective fill-in information at the associated position of each real object in the real scene point cloud image to construct a real object model of each real object under a full perspective, and composing a corresponding real scene model.

3. The method of claim 1, wherein, The method further comprises: if the shooting instruction is an image shooting instruction, displaying the real scene model via the display generation component and storing it as a corresponding spatial image; if the shooting instruction is a video shooting instruction, determining the point cloud index relationship of each rigid real object in the real scene according to the three-dimensional point cloud data of the real scene under a full perspective; in response to a video shooting end instruction input via the one or more input components, according to the point cloud index relationship of each rigid real object in the real scene, the three-dimensional point cloud data of the real scene acquired at each shooting time is completed under a full perspective to construct a real scene model at each shooting time and store it as a corresponding spatial video.

4. The method of claim 3, wherein, The spatial image and the spatial video support display under fixed perspective, follow-up perspective and free perspective.

5. The method of claim 1, wherein, The method further comprises: in response to an adjustment operation input via the one or more input components on any real object model in the real scene model, updating the three-dimensional point cloud data of the real object under a full perspective to adjust the real object model.

6. The method of claim 1, wherein, The method further comprises: in response to a marking operation input via the one or more input components on any real object model in the real scene model, according to the three-dimensional point cloud data of the real object under a full perspective, the real object model is segmented from the real scene model and stored independently.

7. The method of claim 6, wherein, The method further comprises: in response to a first sharing instruction input via the one or more input components on the real scene model or any real object model in the real scene model, sharing the real scene model or the real object model to a designated device.

8. The method of claim 6, wherein, The method further comprises: In response to a second sharing instruction input via the one or more input components to the real scene model or any real object model within the real scene model, the real scene model or the real object model is fused with a target virtual model, the target virtual model including a virtual scene model and a virtual object model.

9. A point cloud data processing apparatus, characterized by comprising: configured on an electronic device in communication with a display generation component and one or more input components, the apparatus comprising: a point cloud acquisition module configured to, in response to a shooting instruction input via the one or more input components, acquire three-dimensional point cloud data of a real scene at a real-time shooting angle, to display a corresponding real scene point cloud image via the display generation component; a retake prompting module configured to present corresponding angle retake information in the real scene point cloud image according to the shot angle of the real scene, to acquire three-dimensional point cloud data of the real scene at a full angle, and to construct a corresponding real scene model.

10. An electronic device, comprising: comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the point cloud data processing method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, a computer program for storing, the computer program causing a computer to execute the point cloud data processing method of any one of claims 1-8.

12. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the point cloud data processing method of any one of claims 1-8.

Images