Point cloud window presentation method, device, computer readable medium and electronic device

By obtaining and parsing the information in the point cloud file, presenting the description content in the point cloud window based on the attached information, the problem of single presentation form of point cloud window in the existing technology is solved, diversified presentation forms are realized, and diversified presentation is met Requirements for application scenarios.

CN114661386BActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011528441.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-22
Publication Date
2024-11-22
Estimated Expiration
2040-12-22

AI Technical Summary

Technical Problem

The prior art is difficult to achieve diversified presentation forms in point cloud windows and cannot meet the business needs of different application scenarios.

Method used

通过获取和解析点云文件中的点云视窗信息和附属信息,根据附属信息在点云视窗中呈现相应的描述内容,支持不同类型的点云视窗呈现相应的描述内容。

Benefits of technology

It realizes that more diverse presentation forms are provided when point cloud window rendering, meeting the business needs of different application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114661386B_ABST
    Figure CN114661386B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a method, device, computer-readable medium and electronic device for presenting a point cloud window. The method for presenting a point cloud window includes: obtaining a point cloud file, wherein the point cloud file contains information about a point cloud window and auxiliary information associated with the point cloud window, wherein the auxiliary information corresponds to the type of the point cloud window; parsing the point cloud file to obtain information about the point cloud window and the auxiliary information; presenting the point cloud window according to the information about the point cloud window, and presenting corresponding description content in the point cloud window according to the auxiliary information. The technical solution of the embodiments of the present application can provide more diverse presentation forms when presenting a point cloud window, meeting the business needs of different application scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular, to a method, device, computer-readable medium and electronic device for presenting a point cloud window. Background Art

[0002] For point cloud media, the related technologies only involve the basic spatial position information of the point cloud window, which meets the basic requirements of point cloud window area rendering. However, for the application scenarios with rich point cloud media, the solutions in the related technologies are difficult to achieve diversified presentation forms and cannot meet the business needs of different application scenarios. Summary of the invention

[0003] The embodiments of the present application provide a point cloud window presentation method, device, computer-readable medium and electronic device, which can provide more diverse presentation forms when presenting the point cloud window, at least to a certain extent, and meet the business needs of different application scenarios.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a method for presenting a point cloud window is provided, including: obtaining a point cloud file, wherein the point cloud file contains information about a point cloud window and auxiliary information associated with the point cloud window, wherein the auxiliary information corresponds to the type of the point cloud window; parsing the point cloud file to obtain information about the point cloud window and the auxiliary information; presenting the point cloud window according to the information about the point cloud window, and presenting corresponding description content in the point cloud window according to the auxiliary information.

[0006] According to one aspect of an embodiment of the present application, a point cloud window presentation device is provided, comprising: an acquisition unit, configured to acquire a point cloud file, wherein the point cloud file contains information of the point cloud window and auxiliary information associated with the point cloud window, wherein the auxiliary information corresponds to the type of the point cloud window; a parsing unit, configured to parse the point cloud file to obtain the information of the point cloud window and the auxiliary information; and a processing unit, configured to present the point cloud window according to the information of the point cloud window, and present corresponding description content in the point cloud window according to the auxiliary information.

[0007] In some embodiments of the present application, based on the aforementioned solution, the point cloud file includes a first field, and the value of the first field is used to indicate the amount of auxiliary information associated with the point cloud window.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the auxiliary information includes a second field, and the second field is used to indicate the type of auxiliary information associated with the point cloud window, and different values ​​of the second field correspond to different types of auxiliary information.

[0009] In some embodiments of the present application, based on the aforementioned scheme, when the value of the second field is a first value, it indicates that the subsidiary information associated with the point cloud window is a subtitle type; when the value of the second field is a second value, it indicates that the subsidiary information associated with the point cloud window is a mark type; when the value of the second field is a third value, it indicates that the subsidiary information associated with the point cloud window is an annotation type.

[0010] In some embodiments of the present application, based on the aforementioned solution, the supplementary information includes a third field, and the third field is used to indicate an identifier of the supplementary information, and the identifier is used to uniquely identify the supplementary information.

[0011] In some embodiments of the present application, based on the aforementioned solution, the supplementary information includes a fourth field, and the fourth field is used to indicate the description content of the supplementary information.

[0012] In some embodiments of the present application, based on the above solution, the auxiliary information includes a fifth field, and the fifth field is used to indicate the area of ​​the point cloud window associated with the auxiliary information;

[0013] When the value of the fifth field is a first value, it indicates that the auxiliary information is associated with the entire area of ​​the point cloud window; when the value of the fifth field is a second value, it indicates that the auxiliary information is associated with a partial area of ​​the point cloud window.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the supplementary information includes a sixth field, and the sixth field is used to indicate the distance between the plane area associated with the supplementary information and the center position of the point cloud window, and the plane area is used to display the description content of the supplementary information.

[0015] In some embodiments of the present application, based on the aforementioned solution, the attribute information of the plane area matches the attribute information of the point cloud window, and the attribute information includes position information, rotation information, and plane type information.

[0016] In some embodiments of the present application, based on the aforementioned solution, the point cloud file includes a seventh field, and the seventh field is used to indicate the number of point cloud windows associated with auxiliary information.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the point cloud file includes an eighth field, and the eighth field is used to indicate an identifier of a point cloud window associated with auxiliary information, and the identifier is used to uniquely identify the point cloud window.

[0018] In some embodiments of the present application, based on the aforementioned scheme, the processing unit is configured to: obtain the description content for display in the point cloud window according to the field value contained in the auxiliary information, and determine the presentation position of the description content; and display the description content at the presentation position in the point cloud window.

[0019] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for presenting a point cloud window as described in the above embodiment is implemented.

[0020] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the point cloud window presentation method as described in the above embodiments.

[0021] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the point cloud window presentation method provided in the above various optional embodiments.

[0022] In the technical solutions provided in some embodiments of the present application, the point cloud file contains information about the point cloud window and auxiliary information associated with the point cloud window, and the auxiliary information corresponds to the type of the point cloud window. The point cloud window information and the auxiliary information are obtained by parsing the point cloud file, so that the point cloud window is presented according to the information of the point cloud window, and the corresponding description content is presented in the point cloud window according to the auxiliary information, so that the corresponding description content can be presented according to different types of point cloud windows, and different types of point cloud windows correspond to the application scenarios of the point cloud media. Therefore, the technical solutions of the embodiments of the present application can provide more diverse presentation forms when presenting the point cloud window, meeting the business needs of different application scenarios.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0026] Figure 2 A schematic diagram showing the placement of a video encoding device and a video decoding device in a streaming transmission system;

[0027] Figure 3 A flowchart of a method for presenting a point cloud window according to an embodiment of the present application is shown;

[0028] Figure 4 A flowchart of a method for presenting a point cloud window according to an embodiment of the present application is shown;

[0029] Figure 5 A block diagram of a point cloud window presentation device according to an embodiment of the present application is shown;

[0030] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0032] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0035] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0036] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0037] like Figure 1 As shown, the system architecture 100 includes a plurality of terminal devices, which can communicate with each other through, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected through the network 150. Figure 1 In the embodiment of the present invention, the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0038] For example, the first terminal device 110 may encode point cloud video data (e.g., a point cloud video stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded point cloud video data is transmitted in the form of one or more encoded video code streams. The second terminal device 120 may receive the encoded point cloud video data from the network 150, decode the encoded point cloud video data to restore the point cloud video data, and display a video image based on the restored point cloud video data.

[0039] In one embodiment of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded point cloud video data, which may occur, for example, during video communication. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode point cloud video data (e.g., a point cloud video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device 130 and the fourth terminal device 140 through the network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded point cloud video data transmitted by the other terminal device of the third terminal device 130 and the fourth terminal device 140, and may decode the encoded point cloud video data to restore the point cloud video data, and may display the point cloud video picture on an accessible display device according to the restored point cloud video data.

[0040] exist Figure 1 In the embodiment of the present invention, the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140 may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network 150 represents any number of networks that transmit encoded point cloud video data between the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140, including, for example, wired and / or wireless communication networks. The communication network 150 may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of the present application, unless explained below, the architecture and topology of the network 150 may be irrelevant to the operations disclosed in the present application.

[0041] In one embodiment of the present application, Figure 2 The placement of the video encoding device and the video decoding device in the streaming environment is shown. The subject matter disclosed in the present application can be equally applicable to other video-supported applications, including, for example, video conferencing, digital TV (television), storing compressed video on digital media including CD, DVD, memory stick, etc.

[0042] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, a point cloud media generation device, etc. The video source creates an uncompressed video picture stream 202. In an embodiment, the video picture stream 202 includes samples captured by a digital camera, or samples captured or generated by a point cloud media. Compared to the encoded video data 204 (or the encoded video code stream 204), the video picture stream 202 is depicted as a thick line to emphasize the high data volume of the video picture stream, and the video picture stream 202 can be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream 202, the encoded video data 204 (or the encoded video code stream 204) is depicted as a thin line to emphasize the lower amount of data of the encoded video data 204 (or the encoded video code stream 204), which can be stored on the streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2 , can access the streaming server 205 to retrieve the copies 207 and 209 of the encoded video data 204. The client subsystem 206 can include, for example, a video decoding device 210 in an electronic device 230. The video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a display screen) or another presentation device. In some streaming systems, the encoded video data 204, video data 207, and video data 209 (e.g., video bitstreams) can be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265.

[0043] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video file decoding device, and the electronic device 230 may also include a video file encoding device.

[0044] In one embodiment of the present application, the video data in the above embodiment may be point cloud media data. A point cloud is a set of discrete points that are irregularly distributed in space and express the spatial structure and surface properties of a three-dimensional object or scene. Each point in a point cloud has at least three-dimensional position information, and may also have color, material or other information depending on the application scenario. Typically, each point in a point cloud has the same number of additional attributes. Point cloud media can be divided into point cloud media compressed by VPCC (Video-based Point Cloud Compression) and point cloud media compressed by GPCC (Geometry-based Point Cloud Compression) based on the encoding method. In the file encapsulation of point cloud media, three-dimensional position information is usually referred to as the geometry component (Geometry Component) of a point cloud file, and attribute information is referred to as the attribute component (Attribute Component) of a point cloud file. A point cloud file has only one geometry component, but there may be one or more attribute components.

[0045] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes, so they are widely used. Their main application scenarios can be divided into two categories: machine-perceived point clouds and human-perceived point clouds. Machine-perceived point clouds include autonomous navigation systems, real-time inspection systems, geographic information systems, visual sorting robots, emergency rescue robots, etc.; human-perceived point clouds include digital cultural heritage, free viewpoint broadcasting, three-dimensional immersive communication, three-dimensional immersive interaction and other point cloud application scenarios.

[0046] Point clouds can be obtained mainly through the following methods: computer generation, 3D laser scanning, 3D photogrammetry, etc. Computers can generate point clouds of virtual three-dimensional objects and scenes; 3D scanning can obtain point clouds of static real-world three-dimensional objects or scenes, and can obtain millions of point clouds per second; 3D photography can obtain point clouds of dynamic real-world three-dimensional objects or scenes, and can obtain tens of millions of point clouds per second. In addition, in the medical field, point clouds of biological tissues and organs can be obtained by MRI (Magnetic Resonance Imaging), CT (Computed Tomography), and electromagnetic positioning information. These technologies reduce the cost and time cycle of point cloud data acquisition, improve the accuracy of data, and the change in the way point cloud data is acquired makes it possible to acquire a large amount of point cloud data. With the continuous accumulation of large-scale point cloud data, the efficient storage, transmission, publication, sharing and standardization of point cloud data have become the key to point cloud applications.

[0047] After encoding the point cloud media, the encoded data stream needs to be encapsulated and transmitted to the client. Correspondingly, on the point cloud media player side, the acquired point cloud file needs to be decapsulated first, then decoded, and finally the decoded data stream is presented.

[0048] In one embodiment of the present application, taking the point cloud media compressed in VPCC mode as an example, the meanings of the fields contained in the point cloud file are as follows:

[0049] viewport_type: indicates the point cloud viewport type. The meanings of the values ​​are shown in Table 1 below:

[0050] Value describe 0 Recommended viewports based on the content creator's shooting intent 1 Recommended viewing windows based on user viewing statistics 2 Recommended widgets created by other users 3 Recommended initial viewport 4-239 Reserved Fields 240-255 Undefined fields (may be customized by the application or other criteria)

[0051] Table 1

[0052] num_viewports: indicates the number of point cloud viewports.

[0053] viewport_description: indicates the description information of the point cloud viewport, which is a string terminated by a null byte.

[0054] int_camera_flag and camera_intrinsic_flag: Indicates whether the point cloud view contains the camera's intrinsic parameters.

[0055] ext_camera_flag and camera_extrinsic_flag: Indicates whether the point cloud view contains the external parameters of the camera.

[0056] IntCameraInfoStruct: used to indicate the internal parameters of the camera, including the camera ID, camera mapping type, camera mapping related parameters, and the depth of the near plane and far plane that constitute the window.

[0057] ExtCameraInfoStruct: used to indicate the external parameters of the camera, including window ID, window position, rotation angle, etc.

[0058] The technical solution of the above embodiment indicates the basic spatial position information of the point cloud window, meeting the basic requirements for rendering the point cloud window area. However, considering the rich application scenarios of point clouds, such as high-precision maps, video communications, etc., it is necessary to attach certain auxiliary information to the point cloud window area to support a variety of presentation forms. Therefore, the technical solution of the embodiment of the present application proposes a method for indicating the auxiliary information of the point cloud window area for the rendering and presentation of point cloud media, which is described in detail as follows:

[0059] Figure 3A flowchart of a method for presenting a point cloud window according to an embodiment of the present application is shown. The method for presenting a point cloud window can be performed by a point cloud media playback device, which can be a smart phone, a tablet computer, etc. Figure 3 As shown, the method for presenting the point cloud window at least includes steps S310 to S330, which are described in detail as follows:

[0060] In step S310, a point cloud file is obtained, wherein the point cloud file includes information of a point cloud window and auxiliary information associated with the point cloud window, wherein the auxiliary information corresponds to the type of the point cloud window.

[0061] In one embodiment of the present application, the point cloud file may be generated by a computer, or may be obtained by 3D laser scanning, or may be obtained by 3D photogrammetry. The point cloud file may be obtained by receiving the point cloud file transmitted by a sending end (the sending end may be, for example, a server end), wherein the sending end may add corresponding auxiliary information according to the type of the point cloud window, and then encapsulate the point cloud file.

[0062] Optionally, the type of the point cloud window corresponds to the type of the point cloud file. For example, the type of the point cloud file may be movie, high-precision map, video communication, etc. Then the type of the point cloud window is the point cloud window corresponding to the movie, high-precision map, video communication, etc.

[0063] In addition, a point cloud file may contain one or more point cloud windows, and a point cloud window may also be associated with one or more auxiliary information.

[0064] In one embodiment of the present application, the point cloud file may include a first field, the value of which is used to indicate the amount of ancillary information associated with the point cloud window. Optionally, for example, the first field may be num_associated_info, which is used to indicate the amount of ancillary information associated with the point cloud window.

[0065] In one embodiment of the present application, the auxiliary information may include a second field, and the second field is used to indicate the type of auxiliary information associated with the point cloud window, and different values ​​of the second field correspond to different auxiliary information types. Optionally, for example, the second field may be associated_info_type.

[0066] In one embodiment of the present application, when the value of the second field is the first value, it indicates that the auxiliary information associated with the point cloud window is of the subtitle type; when the value of the second field is the second value, it indicates that the auxiliary information associated with the point cloud window is of the tag type; when the value of the second field is the third value, it indicates that the auxiliary information associated with the point cloud window is of the annotation type. Optionally, the first value may be, for example, 0, the second value may be, for example, 1, and the third value may be, for example, 2. Of course, the second field may also be other values, such as a value between 3 and 255, which is used to indicate the type of auxiliary information customized by the application or other standards.

[0067] In an embodiment of the present application, the supplementary information may include a third field, and the third field is used to indicate an identifier of the supplementary information, and the identifier is used to uniquely identify the supplementary information. Optionally, the third field may be, for example, associated_info_id.

[0068] In one embodiment of the present application, the supplementary information may include a fourth field, and the fourth field is used to indicate the description content of the supplementary information. Optionally, the fourth field may be, for example, associated_info_description, which may be a string ending with a null character, used to indicate the description content of the supplementary information, and the description content is used to present the content in the point cloud window.

[0069] In one embodiment of the present application, the auxiliary information may include a fifth field, and the fifth field is used to indicate the area of ​​the point cloud window to which the auxiliary information is associated; wherein, when the value of the fifth field is a first value, it indicates that the auxiliary information is associated with the entire area of ​​the point cloud window; when the value of the fifth field is a second value, it indicates that the auxiliary information is associated with a partial area of ​​the point cloud window. Optionally, the fifth field may be, for example, vp_relative_pos_flag, the first value may be, for example, 0, and the second value may be, for example, 1, wherein the specific numerical values ​​are only examples, and in other embodiments of the present application, the first value may be, for example, 1, and the second value may be, for example, 0.

[0070] In one embodiment of the present application, the auxiliary information may include a sixth field, which is used to indicate the distance between the plane area associated with the auxiliary information and the center position of the point cloud window, and the plane area is used to display the description content of the auxiliary information. Optionally, the sixth field may be, for example, clipping_associated_plane. The attribute information of the plane area matches the attribute information of the point cloud window, and the attribute information may include, for example, position information, rotation information, plane type information, etc.

[0071] In one embodiment of the present application, the point cloud file may include a seventh field, and the seventh field is used to indicate the number of point cloud viewports associated with the auxiliary information. Optionally, the seventh field may be, for example, num_associated_viewports.

[0072] In one embodiment of the present application, the point cloud file may include an eighth field, which is used to indicate an identifier of a point cloud viewport associated with the auxiliary information, and the identifier is used to uniquely identify the point cloud viewport. Optionally, the eighth field may be, for example, viewport_id.

[0073] It should be noted that the point cloud file may include any one or more fields from the first to the eighth fields mentioned above.

[0074] Continue to refer to Figure 3 As shown, in step S320, the point cloud file is parsed to obtain the information of the point cloud window and the auxiliary information.

[0075] In one embodiment of the present application, the information of the point cloud window may be, for example, the information indicated by the various fields related to the point cloud window in the aforementioned embodiments, specifically, the information indicated by one or more of the following fields: viewport_type, num_viewports, viewport_description, int_camera_flag, camera_intrinsic_flag, ext_camera_flag, camera_extrinsic_flag, IntCameraInfoStruct, ExtCameraInfoStruct, etc.

[0076] In one embodiment of the present application, the ancillary information may be, for example, information indicated by any one or more fields of the first to eighth fields described in the aforementioned embodiments.

[0077] In step S330, a point cloud window is presented according to the information of the point cloud window, and corresponding description content is presented in the point cloud window according to the auxiliary information.

[0078] In one embodiment of the present application, presenting the point cloud window according to the information of the point cloud window may be specifically, for example: presenting the corresponding type of point cloud window according to the point cloud window type indicated by viewport_type; presenting the corresponding number of point cloud windows according to the number of point cloud windows indicated by num_viewports; and displaying the corresponding information according to the description information of the point cloud window indicated by viewport_description. In addition, it is also possible to determine whether the point cloud window contains the intrinsic parameters of the camera according to int_camera_flag and / or camera_intrinsic_flag; determine whether the point cloud window contains the external parameters of the camera according to ext_camera_flag and / or camera_extrinsic_flag; determine the intrinsic parameters of the camera contained in the point cloud window according to IntCameraInfoStruct; and determine the external parameters of the camera contained in the point cloud window according to ExtCameraInfoStruct. Then present the corresponding information according to the intrinsic parameters and / or external parameters of the camera.

[0079] In one embodiment of the present application, presenting corresponding description content in the point cloud window according to the auxiliary information includes: obtaining description content for display in the point cloud window according to the field value contained in the auxiliary information, determining the presentation position of the description content, and then displaying the description content at the presentation position of the point cloud window. For example, the description content displayed in the point cloud window can be determined according to the value of the fourth field, and the presentation position of the description content can be determined according to the value of the sixth field.

[0080] Of course, the amount of auxiliary information can also be determined according to the value of the first field, and the corresponding amount of auxiliary information can be displayed on the point cloud window based on the amount of auxiliary information; the type of auxiliary information can be determined according to the value of the second field, and the corresponding auxiliary content can be displayed on the point cloud window according to the type of auxiliary information; the identifier of the auxiliary information can be determined according to the third field; the area associated with the point cloud window can be determined according to the fifth field, and the description content can be displayed on the associated area; the number of point cloud windows associated with auxiliary information can also be determined according to the seventh field; the identifier of the point cloud window associated with the auxiliary information can be determined according to the eighth field.

[0081] In combination with the above embodiments, Figure 4 As shown, the server generates a point cloud file and the client consumes the point cloud file as an example. Specifically, the following steps may be included:

[0082] Step S401: The server encapsulates and generates a point cloud file.

[0083] In one embodiment of the present application, the server can add corresponding auxiliary information to the corresponding type of point cloud window on the server side according to the application scenario of the point cloud (such as movie scenes, high-precision map scenes, video communication scenarios, etc.), and then encapsulate and generate a point cloud file.

[0084] Step S402: The client requests a point cloud file from the server.

[0085] Step S403: the server transmits the point cloud file to the client.

[0086] Step S404: the client decapsulates the point cloud file and presents a point cloud window.

[0087] In one embodiment of the present application, after receiving the point cloud file, the client can obtain the auxiliary information of different point cloud windows by decapsulating it, and then provide corresponding presentation content according to the corresponding auxiliary information when presenting the corresponding point cloud window.

[0088] To achieve Figure 4 The technical solution of the embodiment shown in the figure, the embodiment of the present application adds some field information. The following takes the form of extending the ISOBMFF data box as an example, defining the point cloud window area ancillary information, which is applicable not only to the VPCC packaging technology, but also to the GPCC packaging technology. Among them, the extended fields are as follows:

[0089] num_associated_info: Indicates the number of associated information associated with the current viewport.

[0090] associated_info_type: indicates the type of associated auxiliary information. The values ​​are shown in Table 2 below:

[0091] Value describe 0 subtitle 1 mark 2 Notes 3-255 Information customized by the application or other standards

[0092] Table 2

[0093] associated_info_id: an identifier indicating the current associated information.

[0094] associated_info_description: indicates the description content of the associated information, which is a string ending with a null character.

[0095] vp_relative_pos_flag: indicates the window area to which the current auxiliary information is associated. A value of 0 indicates that the current auxiliary information is associated with the entire point cloud window area; a value of 1 indicates that the current auxiliary information is associated with a certain plane area in the point cloud window.

[0096] clipping_associated_plane: indicates the distance between the plane area associated with the auxiliary information and the center position of the point cloud window. The position, rotation information, plane type and other information of the plane area are the same as the corresponding information of the point cloud window.

[0097] num_associated_viewports: indicates the number of windows associated with auxiliary information.

[0098] viewport_id: Indicates the identifier of the viewport that contains the auxiliary information.

[0099] In a specific application scenario of this application, for example, there are three point cloud files F1 to F3 on the server side, corresponding to movies, high-precision maps, and video communications, respectively. The attached information encapsulated on the server side is as follows:

[0100] Point cloud file F1: Since the subtitle information changes over time, the associated information of the point cloud file F1 can be given in ViewportAssociatedInfoSample.

[0101] Examples of parameters in ViewportAssociatedInfoSample are as follows: {num_associated_viewports=1; viewport_id=001; num_associated_info=1; associated_info_type=0; associated_info_id=001; associated_info_description=""; vp_relative_pos_flag=1; clipping_associated_plane=2.0}

[0102] Point cloud file F2: Since the auxiliary information of the high-precision map also changes over time, the auxiliary information of the point cloud file F2 can also be given in ViewportAssociatedInfoSample.

[0103] Examples of each parameter in ViewportAssociatedInfoSample are as follows: {num_associated_viewports=1; viewport_id=001; num_associated_info=1; associated_info_type=1; associated_info_id=001; associated_info_description=“traffic jam”; vp_relative_pos_flag=0}

[0104] Point cloud file F3: The auxiliary information of the video communication does not change over time, so the auxiliary information of the point cloud file F3 can be given in ViewportAssociatedInfoSampleEntry, as follows:

[0105] {num_associated_viewports=1; viewport_id=001; num_associated_info=1; associated_info_type=2; associated_info_id=001; associated_info_description=“userId:01”; vp_relative_pos_flag=1; clipping_associated_plane=2.0}

[0106] For the above-mentioned point cloud files F1, F2 and F3, the client U1, U2 and U3 request the point cloud files F1 to F3 from the server respectively, and then the server transmits the point cloud files to the corresponding client. After receiving the point cloud files, the client obtains the attached information of different point cloud windows by decapsulation, and then provides the corresponding presentation content according to the attached information when presenting the corresponding point cloud window.

[0107] Specifically, after the client U1 receives the point cloud file F1, when viewing the corresponding point cloud window, it searches for the corresponding subtitle information in the subtitle track according to the associated_info_id in the auxiliary information of the point cloud window, and renders the subtitle information at the corresponding position in the window space according to the information of clipping_associated_plane.

[0108] After the client U2 receives the point cloud file F2, when using the high-precision map and parsing the corresponding window, it can know that the current window area (that is, a section of the high-precision map) is marked as traffic congestion based on the information in the associated_info_description and the vp_relative_pos_flag flag. Therefore, it can not only display the traffic congestion information on the current point cloud window, but also provide the traffic congestion information as input information to the autonomous driving system for decision-making.

[0109] After receiving the point cloud file F3, the client U3 parses the attached information and finds that a certain fixed window area of ​​the current video communication contains corresponding annotation information. The user ID can be obtained by parsing the annotation information, and then the personal business card information of the relevant user can be presented in the corresponding area according to the information of clipping_associated_plane.

[0110] The technical solution of the above-mentioned embodiments of the present application can add specific auxiliary information according to different application scenarios during point cloud media rendering and presentation, and associate the auxiliary information with the window area, so as to obtain the auxiliary information when the user consumes a specific window area, thereby enabling the presentation and consumption of point clouds to support more diverse application forms.

[0111] The following describes an embodiment of the device of the present application, which can be used to execute the method for presenting a point cloud window in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method for presenting a point cloud window in the above embodiment of the present application.

[0112] Figure 5 A block diagram of a point cloud window presentation device according to an embodiment of the present application is shown. The point cloud window presentation device can be set in a point cloud media playback device, which can be a smart phone, a tablet computer, etc.

[0113] Reference Figure 5 As shown, a point cloud window presentation device 500 according to an embodiment of the present application includes: an acquisition unit 502 , a parsing unit 504 and a processing unit 506 .

[0114] Among them, the acquisition unit 502 is configured to acquire a point cloud file, which contains information of a point cloud window and auxiliary information associated with the point cloud window, and the auxiliary information corresponds to the type of the point cloud window; the parsing unit 504 is configured to parse the point cloud file to obtain the information of the point cloud window and the auxiliary information; the processing unit 506 is configured to present the point cloud window according to the information of the point cloud window, and present corresponding description content in the point cloud window according to the auxiliary information.

[0115] In some embodiments of the present application, based on the aforementioned solution, the point cloud file includes a first field, and the value of the first field is used to indicate the amount of auxiliary information associated with the point cloud window.

[0116] In some embodiments of the present application, based on the aforementioned scheme, the auxiliary information includes a second field, and the second field is used to indicate the type of auxiliary information associated with the point cloud window, and different values ​​of the second field correspond to different types of auxiliary information.

[0117] In some embodiments of the present application, based on the aforementioned scheme, when the value of the second field is a first value, it indicates that the subsidiary information associated with the point cloud window is a subtitle type; when the value of the second field is a second value, it indicates that the subsidiary information associated with the point cloud window is a mark type; when the value of the second field is a third value, it indicates that the subsidiary information associated with the point cloud window is an annotation type.

[0118] In some embodiments of the present application, based on the aforementioned solution, the supplementary information includes a third field, and the third field is used to indicate an identifier of the supplementary information, and the identifier is used to uniquely identify the supplementary information.

[0119] In some embodiments of the present application, based on the aforementioned solution, the supplementary information includes a fourth field, and the fourth field is used to indicate the description content of the supplementary information.

[0120] In some embodiments of the present application, based on the above solution, the auxiliary information includes a fifth field, and the fifth field is used to indicate the area of ​​the point cloud window associated with the auxiliary information;

[0121] When the value of the fifth field is a first value, it indicates that the auxiliary information is associated with the entire area of ​​the point cloud window; when the value of the fifth field is a second value, it indicates that the auxiliary information is associated with a partial area of ​​the point cloud window.

[0122] In some embodiments of the present application, based on the aforementioned scheme, the supplementary information includes a sixth field, and the sixth field is used to indicate the distance between the plane area associated with the supplementary information and the center position of the point cloud window, and the plane area is used to display the description content of the supplementary information.

[0123] In some embodiments of the present application, based on the aforementioned solution, the attribute information of the plane area matches the attribute information of the point cloud window, and the attribute information includes position information, rotation information, and plane type information.

[0124] In some embodiments of the present application, based on the aforementioned solution, the point cloud file includes a seventh field, and the seventh field is used to indicate the number of point cloud windows associated with auxiliary information.

[0125] In some embodiments of the present application, based on the aforementioned scheme, the point cloud file includes an eighth field, and the eighth field is used to indicate an identifier of a point cloud window associated with auxiliary information, and the identifier is used to uniquely identify the point cloud window.

[0126] In some embodiments of the present application, based on the aforementioned scheme, the processing unit 506 is configured to: obtain the description content for display in the point cloud window according to the field value contained in the auxiliary information, and determine the presentation position of the description content; and display the description content at the presentation position in the point cloud window.

[0127] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0128] It should be noted that Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the method described in the above embodiment. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0130] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0131] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0132] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0133] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0135] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0136] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0137] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0138] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0139] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for presenting a point cloud window, characterized in that: include: Acquire a point cloud file, wherein the point cloud file includes information of a point cloud window, an eighth field for indicating a point cloud window associated with subsidiary information, and subsidiary information associated with the point cloud window, wherein the subsidiary information corresponds to the type of the point cloud window; wherein the point cloud window presents point cloud media data in a three-dimensional manner, and the information of the point cloud window includes a field for indicating the type of the point cloud window and a field for indicating the number of point cloud windows; the eighth field is used to indicate a unique identifier of the point cloud window associated with the subsidiary information; the subsidiary information includes a fifth field, and the fifth field is used to indicate a plane area of ​​the point cloud window associated with the subsidiary information; when the value of the fifth field is a first value, it indicates that the subsidiary information is associated with the entire plane area of ​​the point cloud window; and when the value of the fifth field is a second value, it indicates that the subsidiary information is associated with a partial plane area of ​​the point cloud window; Parsing the point cloud file to obtain the information of the point cloud window and the auxiliary information; Present a point cloud window of a corresponding type according to the field used to indicate the point cloud window type, present a point cloud window of a corresponding number according to the field used to indicate the number of point cloud windows, and obtain, based on the field value contained in the auxiliary information, the descriptive content for display in the plane area associated with the point cloud window indicated by the eighth field, and determine the presentation position of the descriptive content; display the descriptive content at the presentation position in the plane area associated with the point cloud window.

2. The method for presenting a point cloud window according to claim 1, characterized in that: The point cloud file includes a first field, and a value of the first field is used to indicate the amount of auxiliary information associated with the point cloud window.

3. The method for presenting a point cloud window according to claim 1, characterized in that: The auxiliary information includes a second field, where the second field is used to indicate the type of auxiliary information associated with the point cloud window, and different values ​​of the second field correspond to different types of auxiliary information.

4. The method for presenting a point cloud window according to claim 3, characterized in that: When the value of the second field is a first value, it indicates that the subsidiary information associated with the point cloud window is a subtitle type; when the value of the second field is a second value, it indicates that the subsidiary information associated with the point cloud window is a mark type; when the value of the second field is a third value, it indicates that the subsidiary information associated with the point cloud window is an annotation type.

5. The method for presenting a point cloud window according to claim 1, characterized in that: The supplementary information includes a third field, and the third field is used to indicate an identifier of the supplementary information, and the identifier is used to uniquely identify the supplementary information.

6. The method for presenting a point cloud window according to claim 1, characterized in that: The supplementary information includes a fourth field, and the fourth field is used to indicate the description content of the supplementary information.

7. The method for presenting a point cloud window according to claim 1, characterized in that: The supplementary information includes a sixth field, and the sixth field is used to indicate the distance between the plane area associated with the supplementary information and the center position of the point cloud window, and the plane area is used to display the description content of the supplementary information.

8. The method for presenting a point cloud window according to claim 7, characterized in that: The attribute information of the plane area matches the attribute information of the point cloud window, and the attribute information includes position information, rotation information, and plane type information.

9. The method for presenting a point cloud window according to claim 1, characterized in that: The point cloud file includes a seventh field, and the seventh field is used to indicate the number of point cloud windows associated with the auxiliary information.

10. A point cloud window presentation device, characterized in that: include: An acquisition unit is configured to acquire a point cloud file, wherein the point cloud file includes information of a point cloud window, an eighth field for indicating a point cloud window associated with subsidiary information, and subsidiary information associated with the point cloud window, wherein the subsidiary information corresponds to a type of the point cloud window; wherein the point cloud window presents point cloud media data in a three-dimensional manner, and the information of the point cloud window includes a field for indicating a point cloud window type and a field for indicating a number of point cloud windows; the eighth field is used to indicate a unique identifier of a point cloud window associated with subsidiary information; the subsidiary information includes a fifth field, and the fifth field is used to indicate a plane area of ​​the point cloud window associated with the subsidiary information; when the value of the fifth field is a first value, it indicates that the subsidiary information is associated with the entire plane area of ​​the point cloud window; and when the value of the fifth field is a second value, it indicates that the subsidiary information is associated with a partial plane area of ​​the point cloud window; A parsing unit configured to parse the point cloud file to obtain the information of the point cloud window and the auxiliary information; A processing unit is configured to present a point cloud window of a corresponding type according to a field used to indicate the type of point cloud window, present a corresponding number of point cloud windows according to a field used to indicate the number of point cloud windows, and obtain, according to the field value contained in the auxiliary information, a description content for display in a plane area associated with the point cloud window indicated by the eighth field, and determine a presentation position of the description content; and display the description content at the presentation position in the plane area associated with the point cloud window.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for presenting a point cloud window according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more computer programs, which, when executed by the one or more processors, enable the one or more processors to implement the point cloud window presentation method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. The processor of the electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the point cloud window presentation method described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Method and player for playing video

    CN104394478A

  • Playing method, system and terminal for information video file

    CN106791926A

  • Distributed structured three-dimensional point cloud image processing method and system

    CN106844610A

  • Image display method and device, display equipment and computer readable storage medium

    CN112037314A

  • Implemental method and application of personalized presentation of associated multimedia content

    US20180041820A1