Model data compression and decompression method and device

Abstract

The embodiment of the invention provides a model data compression and decompression method and device. In the compression method, after a compression device obtains a to-be-compressed target model, N key view angles can be determined in M shooting view angles, and projection and back projection are carried out on position information of point clouds in the target model under the N key view angles to obtain a first sparse model; and rendering the attribute information of each point cloud in the first sparse model to obtain a second sparse model, so that the scene characterization capability of the second sparse model tends to the scene characterization capability of the target model, and then projecting the position information and the attribute information of each point cloud in the second sparse model under N key perspectives to obtain a projection result. And obtaining a first projection image set, performing image compression on each projection image in the first projection image set to obtain a second projection image set, and taking the second projection image set as compression data corresponding to the target model.

Description

Technical Field

[0001] The present specification relates to the fields of three-dimensional graphics and computer vision, and in particular to a method and device for compressing and decompressing model data. Background Art

[0002] With the rapid development of science and technology, people have put forward higher requirements for high-quality 3D reconstruction and real-time rendering experience of scenes. 3D reconstruction technology can restore the 3D geometry and appearance information of objects or scenes from 2D data. Point cloud is a commonly used data representation form in 3D reconstruction technology, and the reconstructed 3D model can be composed of point clouds. Point cloud is composed of a large number of points, each of which contains information about an object or scene in 3D space. However, in the application of 3D reconstruction technology, the number of point clouds representing 3D models is too large and the representation parameters are too large, making it difficult to directly deploy the reconstructed model on the end-side device, and also puts forward higher requirements in terms of model storage and transmission.

[0003] For example, 3D Gaussian Splatting (3DGS) reconstruction technology has shown potential for application in multiple fields. 3DGS reconstruction technology can achieve low-cost and high-quality scene reconstruction by collecting a scene video or image and combining it with the implicit discrete Gaussian point cloud representation of the scene. However, during the application of this technology, there will be redundancy in the representation of 3D Gaussian point clouds, making it difficult to directly deploy the scene model reconstructed based on 3DGS reconstruction technology on the end-side device, and it will also consume more resources in terms of model storage and transmission.

[0004] Therefore, how to compress model data in three-dimensional reconstruction technology is a technical problem that needs to be solved urgently.

[0005] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0006] The present specification provides a method and device for compressing and decompressing model data, which can compress model data while ensuring the scene representation capability of the model, thereby reducing the resources consumed when storing and transmitting the model data.

[0007] In the first aspect, the present specification provides a method for compressing model data. In the compression method, after the compression device obtains the target model to be compressed, N key viewing angles are determined from M shooting viewing angles, and the first sparse model is obtained by projecting and back-projecting the position information of each point cloud in the target model at the N key viewing angles, and then the attribute information of each point cloud in the first sparse model is rendered to obtain a second sparse model, so that the scene representation capability of the second sparse model approaches the scene representation capability of the target model, the position information and attribute information of each point cloud in the second sparse model are projected at the N key viewing angles to obtain a first projection image set, and each projection image in the first projection image set is respectively compressed to obtain a second projection image set, and the second projection image set is used as the compressed data corresponding to the target model.

[0008] In some embodiments, determining N key viewing angles from the M shooting viewing angles includes the following steps: based on the amount of information respectively contained in the captured images under the M shooting viewing angles, selecting viewing angles whose information amounts meet the target conditions from the M shooting viewing angles as the N key viewing angles.

[0009] In some embodiments, a viewing angle difference between two adjacent key viewing angles among the N key viewing angles is greater than a preset threshold.

[0010] In some embodiments, the first sparse model is obtained by projecting and back-projecting the position information of each point cloud in the target model at N key viewing angles, comprising the following steps: projecting the position information of each point cloud in the target model at the N key viewing angles to obtain N projection images; and back-projecting the N projection images at the N key viewing angles to obtain the first sparse model.

[0011] In some embodiments, projecting the position information of each point cloud in the target model under the N key viewing angles to obtain N projection images includes the following steps: for each of the N key viewing angles, based on the camera observation matrix corresponding to the key viewing angle, projecting the position information of each point cloud in the target model onto the camera plane corresponding to the key viewing angle to obtain a projection image corresponding to the key viewing angle.

[0012] In some embodiments, the back-projecting of the N projection images at the N key viewing angles to obtain the first sparse model includes the following steps: for each of the N key viewing angles, based on the camera observation matrix corresponding to the key viewing angle, back-projecting each pixel point in the projection image corresponding to the key viewing angle into a three-dimensional space to obtain a point cloud subset corresponding to the key viewing angle; and generating the first sparse model based on the point cloud subsets corresponding to the N key viewing angles respectively.

[0013] In some embodiments, the rendering of the attribute information of each point cloud in the first sparse model to obtain the second sparse model includes the following steps: using the captured images at the N key perspectives as supervision, performing rendering training on the attribute information of each point cloud in the first sparse model to obtain the second sparse model, wherein the following training objective is adopted during the rendering training process: minimizing the difference between the projection image of the second sparse model at the N key perspectives and the captured images at the N key perspectives.

[0014] In some embodiments, the representation dimension of the attribute information of the point cloud in the second sparse model is higher than the representation dimension of the attribute information of the point cloud in the target model.

[0015] In some embodiments, each projection image in the first projection image set corresponds to an attribute dimension, and the image compression process of each projection image includes the following steps: based on the attribute dimension corresponding to the projection image, determining compression parameters suitable for compressing the projection image, and performing image compression processing on the projection image using the compression parameters, wherein different compression parameters are determined for different attribute dimensions.

[0016] In some embodiments, determining compression parameters suitable for compressing the projection image based on the attribute dimensions corresponding to the projection image, and performing image compression processing on the projection image through the compression parameters, includes the following steps: inputting the projection image and the attribute dimensions corresponding to the projection image into a pre-trained adaptive image compression network, so as to perform image compression processing on the projection image through the adaptive image compression network, wherein the adaptive image compression network is trained to use different compression parameters for image compression for projection images with different attribute dimensions.

[0017] In a second aspect, the specification also provides a compression device for model data. The compression device includes at least one storage medium and at least one processor. The at least one storage medium stores at least one instruction set for data compression. The at least one processor is in communication connection with the at least one storage medium, wherein when the compression device is running, the at least one processor reads the at least one instruction set and executes the method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0018] In a third aspect, the present specification also provides a method for decompressing model data. The decompression method comprises the following steps: obtaining compressed data corresponding to a target model, wherein the target model is a model obtained by three-dimensionally reconstructing images captured at M shooting angles, the target model includes location information and attribute information of multiple point clouds, M is an integer greater than 2, and the compressed data is obtained by compressing the target model by the method described in any one of the first aspects; obtaining a second projection image set from the compressed data, the second projection image set including compressed projection images corresponding to N key perspectives, the N key perspectives being partial perspectives among the M shooting angles, and N being an integer greater than 1; performing image decompression processing on each projection image in the second projection image set to obtain a first projection image set; and performing back-projection on the first projection image set at the N key perspectives to obtain decompressed data corresponding to the target model.

[0019] In some embodiments, each projection image in the second projection image set corresponds to an attribute dimension, and the decompression processing process corresponding to each projection image includes: determining a decompression parameter applicable to decompressing the projection image for the attribute dimension corresponding to the projection image, and performing image decompression processing on the projection image using the decompression parameter, wherein different decompression parameters are determined for different attribute dimensions.

[0020] In a fourth aspect, the present specification also provides a model data decompression device, comprising at least one storage medium and at least one processor. The at least one storage medium stores at least one instruction set for data decompression; the at least one processor is in communication connection with the at least one storage medium, wherein when the decompression device is running, the at least one processor reads the at least one instruction set and executes the method as described in any one of the third aspects according to the instructions of the at least one instruction set.

[0021] It can be seen from the above technical scheme that the model data compression method provided in this specification can determine N key perspectives from M shooting perspectives after obtaining the target model to be compressed, and obtain a first sparse model by projecting and back-projecting the position information of each point cloud in the target model at N key perspectives, and then rendering the attribute information of each point cloud in the first sparse model to obtain a second sparse model, so that the scene representation capability of the second sparse model approaches the scene representation capability of the target model, and then, project the position information and attribute information of each point cloud in the second sparse model at N key perspectives to obtain a first projection image set, and perform image compression on each projection image in the first projection image set to obtain a second projection image set, and use the second projection image set as compressed data corresponding to the target model. The above method selects some key perspectives from multiple shooting perspectives, and reconstructs a first sparse model with a reduced number of point clouds using the position projection information under the key perspectives, and then renders the point cloud attribute information in the first sparse model so that the scene representation capability of the rendered second sparse model is equivalent to that of the target model, that is, the second sparse model can also achieve efficient scene representation using a limited number of point clouds. Furthermore, the compression method projects the second sparse model at a key perspective and converts the second sparse model into structured two-dimensional image data, which can reduce the amount of model data. Furthermore, the compression method further reduces the amount of model data by performing image compression on the projected image of the second sparse model. The compression method provided in this specification can facilitate the deployment of the model on the device on the end side by compressing the model data, while also reducing the resources required for model storage and transmission.

[0022] Other functions of the compression and decompression method and device of model data provided in this specification will be partially listed in the following description. The creative aspects of the compression and decompression method and device of model data provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of an application scenario of model data processing provided according to an embodiment of this specification is shown; Figure 2 A schematic diagram of a computing device provided according to an embodiment of the present specification is shown; Figure 3 A flowchart of a model data compression method provided according to some embodiments of this specification is shown; Figure 4 A schematic diagram of a target model provided according to some embodiments of this specification is shown; Figure 5 A schematic diagram of a first sparse model provided according to some embodiments of this specification is shown; Figure 6 A schematic diagram of a second sparse model provided according to some embodiments of this specification is shown; Figure 7 A schematic diagram of a first projection image set provided according to some embodiments of this specification is shown; Figure 8 A schematic diagram showing a second projection image set provided according to some embodiments of the present specification; and Fig. 9 A flowchart of a method for decompressing model data provided according to some embodiments of the present specification is shown. DETAILED DESCRIPTION

[0025] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0026] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0027] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0028] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0029] For the convenience of description, the terms involved in this specification are first explained as follows.

[0030] Attributes: In the field of 3D graphics and computer vision, attributes usually refer to various features and parameters related to objects, scenes or images in three-dimensional space, also known as 3D attributes. These attributes can be physical, geometric, and visual, and they determine the representation of objects in the three-dimensional world. For example, attributes can include geometric attributes (position, shape, size, direction, normal, etc.), visual attributes (color, texture, lighting, shadow, etc.), physical attributes (material, density, etc.), image attributes (pixel value, edge, etc.), and spatial relationship attributes (distance, occlusion relationship, etc.).

[0031] Channel: In the field of 3D graphics and computer vision, channels can be considered as a form of expression and storage of attributes. Attributes are the information content carried by channels. For example, in an image, color attributes can be stored through three channels of RGB (Red, Green, Blue), and each channel represents a color component. For another example, the normal information of a 3D object can be expressed through a normal channel, while the color information can be expressed through the RGB channel. Channels and attributes together form the basis for describing and processing the 3D world.

[0032] 3D reconstruction technology is a process of converting 2D images or data into 3D models. This technology is widely used in many fields such as computer vision, graphics, robotics, medical imaging, archeology, etc. The goal of 3D reconstruction technology is to restore the 3D geometry and appearance information of an object or scene from given data. The following is an introduction to 3DGS reconstruction technology. And for the sake of convenience, objects and scenes are collectively referred to as scenes.

[0033] 3DGS reconstruction technology mainly obtains 3D Gaussian point cloud representation of the scene by collecting multi-view images of the scene and combining them with cutting-edge differentiable neural radiation field training. Each Gaussian point cloud data contains the 3D properties of the scene under different viewing angles. 3DGS reconstruction technology is based on a differentiable rendering process supervised by multiple view images, and trains and optimizes 3D Gaussian point clouds through the network, thereby achieving high-quality reconstruction and representation of 3D scenes.

[0034] This specification provides a method for processing model data. This method can realize the compression and reconstruction of any three-dimensional model. Specifically, on the one hand, this specification provides a method for compressing model data. This method can realize the compression of any three-dimensional model. When this method is applied to 3DGS reconstruction technology, it can realize the compression of any 3DGS model. On the other hand, this specification provides a method for decompressing model data. This method can decompress the compressed data obtained by the above compression method to realize the reconstruction of the three-dimensional model. When this method is applied to 3DGS reconstruction technology, it can realize the reconstruction of the 3DGS model.

[0035] Those skilled in the art should understand that the model data compression method and / or model data decompression method described in this specification applied to other three-dimensional reconstruction technologies, such as voxel rendering algorithms based on Neural Radiance Field (NeRF), etc. are also within the scope of protection of this specification.

[0036] Figure 1 FIG. 1 is a schematic diagram showing an application scenario of model data processing provided according to some embodiments of this specification. Figure 1 As shown, the scene 100 may include a compression device 110 for model data (hereinafter referred to as compression device 110 ), a decompression device 120 for model data (hereinafter referred to as decompression device 120 ), and a transmission medium 130 .

[0037] The compression device 110 may receive a target model to be compressed, and compress the target model using the model data compression method proposed in this specification to obtain corresponding compressed data. The compression device 110 may store data or instructions for executing the compression method described in this specification, and execute the data and / or instructions to obtain compressed data corresponding to the target model.

[0038] In some embodiments, the compression device 110 may also be a device for generating a target model. After the compression device 110 generates the target model, it compresses the target model to facilitate subsequent transmission and storage of the model data.

[0039] The decompression device 120 may receive compressed data corresponding to the target model, and decompress the compressed data using the model data decompression method proposed in this specification to obtain decompressed data corresponding to the target model. The decompression device 120 may store data or instructions for executing the decompression method described in this specification, and execute the data and / or instructions to obtain decompressed data corresponding to the target model.

[0040] The compression device 110 and the decompression device 120 may include a wide range of devices. For example, the compression device 110 and the decompression device 120 may include a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a handheld device such as a smart phone, a television, a camera, a display device, a digital media player, a video game console, a car computer, or the like.

[0041] like Figure 1 As shown, the compression device 110 and the decompression device 120 can be connected via a transmission medium 130. The transmission medium 130 can facilitate the transmission of information and / or data. The transmission medium 130 can be any data carrier that can transmit compressed data from the compression device 110 to the decompression device 120. For example, the transmission medium 130 can be a storage medium (e.g., an optical disk), a wired or wireless communication medium. The communication medium can be a network. In some embodiments, the transmission medium 130 can be any type of wired or wireless network, or a combination thereof. For example, the transmission medium 130 can include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or a similar network.

[0042] One or more components in the compression device 110 and the decompression device 120 may be connected to the transmission medium 130 to transmit data and / or information. The transmission medium 130 may include a router, a switch, a base station, or other devices that facilitate communication from the compression device 110 to the decompression device 120. In other embodiments, the transmission medium 130 may be a storage medium, such as a mass storage, a removable memory, a volatile read-write memory, a read-only memory (ROM), or the like, or any combination thereof. Exemplary mass storage may include non-transitory storage mediums such as disks, optical disks, and solid-state drives. Removable storage may include flash drives, floppy disks, optical disks, memory cards, zip disks, tapes, and the like. Typical volatile read-write memory may include random access memory (RAM). RAM may include dynamic RAM (DRAM), double date rate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero capacitance RAM (Z-RAM), and the like. ROM may include mask ROM (MROM), programmable ROM (PROM), virtual programmable ROM (PEROM), electronic programmable ROM (EEPROM), compact disk (CD-ROM) and digital versatile disk ROM, etc. In some embodiments, the transmission medium 130 may be a cloud platform. Just as an example, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, cloud-to-cloud, etc., or forms similar to the above forms, or any combination of the above forms.

[0043] like Figure 1 As shown, the compression device 110 receives the target model to be compressed, and executes the instructions of the model data compression method described in this specification, compresses the target model data, and obtains compressed data corresponding to the target model; the compressed data is transmitted to the decompression device 120 via the transmission medium 130; the decompression device 120 executes the instructions of the model data decompression method described in this specification, decompresses the compressed data, and obtains decompressed data corresponding to the target model.

[0044] In some embodiments, the compression device 110 and the decompression device 120 may be different devices. Figure 1 As shown, the target model to be compressed is compressed by a dedicated compression device 110, and decompressed by another dedicated decompression device 120 at the receiving end. In some embodiments, the compression device 110 and the decompression device 120 can be the same device. This means that one device can complete both the compression of the target model and the decompression of the compressed data corresponding to the target model. In this case, compressing the model data can facilitate the storage of the model data.

[0045] Figure 2 FIG. 2 shows a schematic diagram of a computing device 200 provided according to some embodiments of the present specification. The compression device 110 and the decompression device 120 may have the following features: Figure 2 The structure of computing device 200 is shown.

[0046] like Figure 2 As shown, the computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, the computing device 200 may further include a communication port 250 and an internal communication bus 210. Meanwhile, the computing device 200 may further include an I / O component 260.

[0047] Internal communication bus 210 may connect various system components, including storage media 230 and processor 220. I / O components 260 support input / output between computing device 200 and other components.

[0048] The storage medium 230 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. The storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the compression method or the decompression method provided in this specification.

[0049] The communication port 250 is used for data communication between the compression device 110 or the decompression device 120 and the outside world. For example, the data compression device 110 can be connected to the transmission medium 130 through the communication port 250 .

[0050] The processor 220 may perform all steps included in the compression method or the decompression method. The processor 220 may be in the form of one or more processors. The processor 220 may issue execution instructions. The processor 220 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0051] For illustrative purposes only, only one processor 220 is shown in the computing device 200 in the drawings of this specification. However, it should be noted that the computing device 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as described in this specification, or may be performed jointly by multiple processors. For example, if the processor 220 of the computing device 200 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 220 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0052] Figure 3 FIG. 1 is a flowchart of a method P300 for compressing model data according to some embodiments of the present specification. As described above, the compression device 110 may execute the compression method P300. Specifically, the processor 220 in the compression device 110 may read an instruction set stored in its local storage medium, and then execute the method P300 for compressing model data according to the provisions of the instruction set. Figure 3 As shown, method P300 may include: P310: Obtain a target model to be compressed. The target model may be a model obtained by three-dimensionally reconstructing images captured at M shooting angles, where M is an integer greater than 2. The target model may include location information and attribute information of multiple point clouds.

[0053] Among them, three-dimensional reconstruction refers to the technology of converting two-dimensional image data into a three-dimensional model.

[0054] The target model is a 3D model constructed by using 3D reconstruction technology to construct multiple images of the target object. The target object can be any object or scene that the user expects to obtain or reconstruct, such as buildings, cars, people, seascapes, etc.

[0055] Figure 4 A schematic diagram of a target model is provided according to some embodiments of this specification. Figure 4 As shown in , the target object is an excavator, and the target model is a three-dimensional model of the excavator.

[0056] The target model can be composed of a bunch of points or point clouds. Therefore, the target model can include the location information and attribute information of multiple point clouds. Among them, the location information can include the coordinates of each point in the point cloud in three-dimensional space. The attribute information can include other information of each point in the point cloud in three-dimensional space in addition to the location information. For example, the aforementioned color, intensity, normal (direction of the surface), texture, etc. The attribute information can provide more details about the surface characteristics of the target model.

[0057] The target model utilizes multiple captured images of the target object during reconstruction. The shooting orientation / angle of the camera when shooting each captured image is called the shooting angle of view. By shooting from multiple different shooting angles, relatively rich captured images are obtained, which can provide visual information about the target object from different perspectives, thereby helping to capture different sides and details of the target object. The above-mentioned shooting angles can be selected arbitrarily or arranged according to a certain rule or pattern. For example, the shooting angles differ by a fixed angle. In some embodiments, in order to obtain more information about the target object, the shooting angles can be as many as possible. Further, in some embodiments, in order not to miss information about the target object, the field of view corresponding to different shooting angles can overlap with each other. It can be understood that when the shooting angles are relatively rich, the reconstructed target model has a stronger ability to represent the target object, making the target model more realistic.

[0058] For example, Figure 4 , eight shooting angles, View1 to View8, are shown. Where M=8. Eight or eight groups of shooting pictures P1 to P8 (P8 is not shown) are obtained from the eight shooting angles. A three-dimensional model of the excavator, that is, a target model, can be obtained by three-dimensional reconstruction of the eight shooting images.

[0059] As an example, the above three-dimensional reconstruction process to obtain the target model can be implemented by 3DGS reconstruction technology. Specifically, after acquiring the photographic images of multiple shooting angles, the light rendering and differentiable 3D Gaussian point cloud training process are performed based on the photographic images to obtain a dense point cloud target model.

[0060] Continue to see Figure 3 , method P300 may also include step P320.

[0061] P320: Determine N key viewing angles among the M shooting viewing angles, and obtain a first sparse model by projecting and back-projecting the position information of each point cloud in the target model under the N key viewing angles, wherein the first sparse model includes the position information of some point clouds in the target model, and N is an integer less than M.

[0062] Among them, the key perspective can also be called the supervisory perspective, which can refer to the most or relatively representative or key perspective among all shooting perspectives. For example, the key perspective may provide a more comprehensive coverage of the scene, and the key perspective may contain more important information about the scene, so that it can better represent the object or scene and play an important role in reconstructing the target model.

[0063] Among multiple shooting angles, some shooting angles may contain repeated information, which may lead to data redundancy. When resources are limited, processing data from all shooting angles will take up more computing resources. By selecting N key angles, data redundancy can be reduced and the use of computing resources can be optimized.

[0064] For example, Figure 4 As shown, four key perspectives are determined from the eight shooting perspectives, View1, View3, View5, and View7. Taking the bucket of the excavator model as the front of the excavator, these four key perspectives can be the forward perspective View1 facing the front of the excavator model, the backward perspective View5 facing the back of the excavator model, the left perspective View7 facing the left side of the excavator model, and the right perspective View3 facing the right side of the excavator model. Through these four key perspectives, the four views of the excavator can be obtained, thereby determining the overall structure of the excavator. It is worth noting that the actual direction of the key perspective can be different from the actual direction of the key perspective. Figure 4 The viewing angles shown are in different orientations.

[0065] In some embodiments, N key viewing angles can be determined from M shooting viewing angles by the following steps: based on the amount of information contained in the captured images at the M shooting viewing angles, viewing angles whose information amounts meet the target conditions are selected from the M shooting viewing angles as N key viewing angles.

[0066] The more information a captured image contains, the better it can be considered to represent the scene, so as to ensure that the scene can be efficiently represented in the future with a limited number of point clouds. For example, a captured image with high information content may contain feature points, textures or structures that are not in other captured images (i.e., captured images with low information content).

[0067] In some embodiments, the information entropy of an image can be used to measure the amount of information contained in the image. The higher the amount of information contained in a captured image, the higher the information entropy of the captured image. For example, the resolution of the captured image is 256×256. The captured image can be composed of 256 rows and 256 columns of pixels. The information entropy of each pixel can be calculated by the information entropy calculation formula, and the information entropy of all pixels can be added to obtain the information entropy of the entire image.

[0068] In some embodiments, the target condition may include: the information entropy of the captured image is greater than a preset information entropy threshold. For example, the perspectives whose information entropy exceeds the information entropy threshold are selected from M shooting perspectives as N key perspectives. The information entropy threshold may be set based on manual experience or determined by machine learning, which is not limited in this specification. For another example, the first N perspectives with the highest information entropy are selected from M shooting perspectives as key perspectives. In this case, N may be a value pre-set based on actual conditions.

[0069] In some embodiments, the difference in perspective between two adjacent key perspectives among the N key perspectives is greater than a preset threshold. The difference in perspective may refer to the spatial angle from one key perspective to another key perspective. In other words, the N key perspectives may be distributed relatively evenly. The relatively even distribution of the N key perspectives may enable the scene to be covered as comprehensively as possible within a limited perspective, and the perspectives may complement each other to provide relatively complete scene information. For example, Figure 4 As shown, the four key perspectives View1, View3, View5, and View7 can be distributed relatively evenly in the three-dimensional space.

[0070] After determining the N key viewing angles, the processor 220 may obtain a first sparse model by projecting and back-projecting the position information of each point cloud in the target model under the N key viewing angles.

[0071] The first sparse model may represent the basic shape and structure of the scene, but does not have other details of the scene. For example, the first sparse model only includes the shape of the excavator model, but does not include other attribute details such as the color, texture, and material of the excavator.

[0072] Since the first sparse model is generated by projecting and back-projecting the position information of each point cloud data in the target model under N key viewing angles. During the above projection and back-projection process, the processor 220 can filter and retain the point cloud data related to the N key viewing angles, and discard the point cloud data irrelevant to the N key viewing angles. Therefore, the number of point clouds (data) included in the first sparse model is smaller than that of the target model. In other words, the first sparse model only includes part of the point cloud data in the target model.

[0073] Figure 5 A schematic diagram of a first sparse model provided according to some embodiments of this specification is shown below. Figure 5 The generation process of the first sparse model is described.

[0074] (1) Project the position information of each point cloud in the target model at N key perspectives to obtain N projection images.

[0075] Through projection, the information of high-dimensional space can be mapped to low-dimensional space. For example, through projection, the information of three-dimensional space can be converted into the information of two-dimensional space. Through projection, the two-dimensional projection image can reflect the information of the target model in three-dimensional space.

[0076] In some embodiments, for each of the N key perspectives, based on the camera observation matrix corresponding to the key perspective, the position information of each point cloud in the target model is projected onto the camera plane corresponding to the key perspective to obtain a projection image corresponding to the key perspective.

[0077] The camera observation matrix can characterize the position and orientation of the camera when shooting. The camera observation matrix can transform the point cloud in the three-dimensional space into the camera coordinate system. Through the camera observation matrix, the point cloud in the camera coordinate system can eventually be projected onto the camera plane to form a two-dimensional image. In other words, the processor 220 can transform each point cloud in the target model into the camera coordinate system through the camera observation matrix, and then project the points in the camera coordinate system to the camera plane to obtain the corresponding projected image. In some embodiments, the camera observation matrix performs matrix operations on the three-dimensional coordinates of the point cloud to obtain the projection coordinates of the point cloud on the camera plane.

[0078] In some embodiments, the camera observation matrix may be obtained by the processor 220 in real time calculation after the processor 220 determines the key viewing angle. In some embodiments, the camera observation matrices of all shooting viewing angles may be pre-stored in a memory, and after the processor 220 determines the key viewing angle, the processor 220 may directly obtain the corresponding camera observation matrix.

[0079] The camera plane may refer to a plane where an image is formed. For example, the camera plane may refer to a plane where a camera sensor is located.

[0080] The projected image can reflect the position information of each point cloud in the target model. In some embodiments, the projected image can be a depth map to show the distance of each point cloud to the camera plane. In some embodiments, the projected image can also be a grayscale map.

[0081] For example, Figure 5 As shown in Figure 1, under the key perspective View1, the position information of each point cloud in the target model is projected onto its corresponding camera plane, and the projection image can be obtained. Similarly, under the key perspective View3, after projecting the position information of each point cloud in the target model onto its corresponding camera plane, the projected image can be obtained. Under the key perspective View5, after projecting the position information of each point cloud in the target model onto its corresponding camera plane, the projected image can be obtained. Under the key perspective View7, after projecting the position information of each point cloud in the target model onto its corresponding camera plane, the projected image can be obtained. .

[0082] In some embodiments, the position information of each point cloud in the target model can also be projected onto other planes. As long as the plane information of the plane (such as normal information, coordinates of the midpoint of the plane, etc.) or the observation matrix corresponding to the viewing angle of the plane is known, the processor 220 can also perform subsequent compression operations based on the projected image projected onto the plane.

[0083] (2) Back-project the N projected images at N key viewing angles to obtain the first sparse model.

[0084] Back projection can be considered as the inverse process of projection. Through back projection, the information of low-dimensional space can be restored to high-dimensional space. Through back projection, the information of two-dimensional space can be converted into the information of three-dimensional space. Through back projection, the two-dimensional projection image is reconstructed into the first sparse model in three-dimensional space.

[0085] In some embodiments, the first sparse model can be obtained in the following manner: for each of the N key perspectives, based on the camera observation matrix corresponding to the key perspective, each pixel point in the projection image corresponding to the key perspective is back-projected into the three-dimensional space to obtain a point cloud subset corresponding to the key perspective; and the first sparse model is generated based on the point cloud subsets corresponding to the N key perspectives respectively.

[0086] As can be seen from the above, the camera observation matrix can be used to convert the three-dimensional data of the point cloud to the projection image. Therefore, by performing an inverse transformation operation through the camera observation matrix, each pixel point in the projection image can be back-projected into the three-dimensional space, thereby obtaining a point cloud subset. In some embodiments, the camera observation matrix and the two-dimensional coordinates of the projection image are subjected to a matrix inverse operation to obtain the three-dimensional data (coordinates) of the point cloud in the three-dimensional space, thereby obtaining a point cloud subset.

[0087] The point cloud subset can directly constitute the first sparse model. The point cloud subset can also generate the first sparse model through a training optimization process. The process of generating the first sparse model can refer to the process of generating the target model, which will not be repeated here. As an example, the first sparse model can be generated by 3DGS reconstruction technology.

[0088] Continue to see Figure 3 , step P300 may further include: P330: Rendering the attribute information of each point cloud in the first sparse model to obtain a second sparse model, so that the scene representation capability of the second sparse model approaches the scene representation capability of the target model.

[0089] As before, since the first sparse model is a model obtained by projecting and back-projecting the position information of each point cloud in the target model at N key viewing angles, the first sparse model only includes the position information of some point clouds in the target model, but does not include the attribute information of the point clouds. In other words, the scene expression capability of the first sparse model obtained through P320 is weaker than that of the target model. Therefore, in P330, the processor 220 can obtain a second sparse model by rendering the attribute information of each point cloud in the first sparse model, so that the scene representation capability of the second sparse model tends to the scene representation capability of the target model.

[0090] The attribute information may include, but is not limited to, the color information, transparency information, curvature information, density information, material information, etc. mentioned above. As mentioned above, the attribute information can provide more details about the surface characteristics of the model. Rendering the first sparse model with the attribute information can improve the representation capability of the point cloud in the model, thereby obtaining a more realistic and detailed second sparse model.

[0091] The scene representation capability may refer to the ability to represent a real scene in a three-dimensional space, or may refer to the degree of realism of restoring the scene. For example, when a user observes a target model and a second sparse model, visually, the user believes that the scene presented by the second sparse model is the same or approximately the same as the scene presented by the target model. In this case, it can be considered that the scene representation capability of the second sparse model tends to the scene representation capability of the target model. For another example, the difference between the amount of information contained in the target model and the amount of information contained in the second sparse model is less than a preset value. In this case, it can be considered that the scene representation capability of the second sparse model tends to the scene representation capability of the target model.

[0092] The number of point clouds in the second sparse model is less than the number of point clouds in the target model. When the scene representation capability of the second sparse model approaches that of the target model, it means that the second sparse model uses a smaller number of point clouds to achieve the same or similar scene representation capability as the target model. In this way, the second sparse model reduces the amount of model data compared to the target model.

[0093] Figure 6 A schematic diagram of a second sparse model provided according to some embodiments of the present specification is shown. Figure 6 , a second sparse model obtained after rendering training by the first sparse model is shown. The second sparse model has completely or almost presented all the features of the excavator. The excavator presented by the second sparse model is completely or almost the same as the excavator presented by the target model visually.

[0094] In some embodiments, the second sparse model can be obtained by the following steps: using the images taken at N key viewing angles as supervision, the attribute information of each point cloud in the first sparse model is rendered and trained to obtain the second sparse model. During the rendering training, the training goal is to minimize the difference between the projection images of the second sparse model at the N key viewing angles and the images taken at the N key viewing angles.

[0095] Since the images captured under critical viewing angles include rich and critical scene or object information. By using these captured images as supervision to perform rendering training on the first sparse model, the obtained second sparse model can be made closer to the real scene. Rendering training on the attribute information of each point cloud in the first sparse model can be understood as "assigning" the attributes of the pixels in the captured image to the corresponding point cloud. For example, rendering training includes processes such as "coloring", "lighting estimation", and "material rendering" of each point cloud in the first sparse model to obtain a second sparse model close to the real scene.

[0096] As an example, the above-mentioned process of rendering and training the first sparse model to obtain the second sparse model can be implemented by 3DGS reconstruction technology. The 3DGS reconstruction technology uses the point cloud in the first sparse model, the captured image (that is, the true value image at the key perspective) and the rendered image (that is, the projected image of the rendered sparse model at the key perspective) to train the model by minimizing the difference between the rendered image and the true value image. By calculating the loss function, the point cloud data in the rendered sparse model is updated by back propagation, so that the scene expression capability of the rendered sparse model gradually approaches the scene expression capability of the target model.

[0097] In order to improve the representation ability of point cloud data, in some embodiments, the representation dimension of the attribute information of the point cloud in the second sparse model is higher than the representation dimension of the attribute information of the point cloud in the target model. The representation dimension may refer to the number of vector dimensions or parameters required to describe or represent the attribute information. For example, a point cloud in the target model uses a ten-dimensional vector to represent the color, and the corresponding point cloud in the second sparse model uses a twenty-dimensional vector to represent the color. Therefore, the color information of the point cloud in the second sparse model trained with the captured image as the supervisory information is richer, thereby improving the representation ability of the point cloud data, and then improving the representation ability of the limited point cloud data for the scene. In other words, since the representation dimension left in the second sparse model is relatively high, when using the captured image to render the second sparse model, the processor 220 can extract more and richer information from the captured image, so as to obtain more information about the scene, so that the rendered second sparse model is closer to the real scene.

[0098] In some embodiments, the number of point clouds included in the second sparse model is different from the number of point clouds included in the first sparse model. For example, the number of point clouds included in the second sparse model is greater than the number of point clouds included in the first sparse model. During the rendering training process, the scene representation capability of the model is enhanced by increasing the number of point clouds by a small amount. For example, in order to make the scene expression capability of the second sparse model comparable to that of the target model, a small amount of point cloud data can be added to the second sparse model during the training process if it is very necessary. It can be understood that the number of point clouds added during the rendering process is much smaller than the number of point clouds discarded in P320, that is, the number of point clouds added during the rendering process can be ignored.

[0099] In some embodiments, the number of point clouds included in the second sparse model is the same as the number of point clouds included in the first sparse model. That is, during the rendering process, the processor 220 only increases the representation dimension of the attribute information without increasing the point cloud data, so that the second sparse model uses a smaller number of point clouds to achieve efficient representation of the scene.

[0100] Continue to see Figure 3 , the method P300 further includes: P340: Project the position information and attribute information of each point cloud in the second sparse model at N key viewing angles to obtain a first projection image set.

[0101] Specifically, the processor 220 can project the position information and attribute information of each point cloud in the second sparse model onto the camera plane through the camera observation matrix corresponding to the key view, thereby obtaining a first projection image set. The projection method of the attribute information is similar to the projection method of the position information. For the specific projection method and process, please refer to the projection of the target model mentioned above.

[0102] The attribute information of the point cloud in the second sparse model may involve multiple attribute dimensions. The attribute dimensions may be the aforementioned attributes or more specific divisions of the aforementioned attributes. For example, the color attribute of the point cloud in the second sparse model may be represented by RGB (Red, Green, Blue), and the attribute dimensions involved in the attribute information of the point cloud in the second sparse model include at least three dimensions: R, G, and B.

[0103] When the second sparse model is projected, the processor can project the information of each attribute dimension in the second sparse model at the N key viewing angles, thereby obtaining a first set of projection images. For example, taking the attribute information of the point cloud in the second sparse model involving five attribute dimensions as an example, it is assumed that the five attribute dimensions are attribute dimension A, attribute dimension B, attribute dimension C, attribute dimension D, and attribute dimension E. The processor 220 can project the attribute information of each point cloud in the second sparse model on the attribute dimension A at the N key viewing angles, respectively, to obtain N projection images. These N projection images correspond to the attribute dimension A. Similarly, the processor 220 can project the attribute information of each point cloud in the second sparse model on the attribute dimension B at the N key viewing angles, respectively, to obtain N projection images. These N projection images correspond to the attribute dimension B. By analogy, the projection process of the processor 220 for the attribute dimension C, attribute dimension D, and attribute dimension E is similar.

[0104] Figure 7 A schematic diagram of a first projection image set provided according to some embodiments of this specification is shown. Figure 7 As shown, the first projection image set includes projection images under four key viewing angles. Specifically, the first projection image set includes the projection images obtained under View1 , projected under View3 , projected under View5 , projected under View7 The first projection image under each key viewing angle also includes projection images of different attribute dimensions. Different attribute dimensions are represented by different colors. For example, the projection image obtained under View1 It includes five projection images corresponding to different attribute dimensions (attribute dimensions A, B, C, D, E). Similarly, the projection images obtained under View3, View5, and View7 , , It also includes projection images corresponding to the above five attribute dimensions.

[0105] By converting the three-dimensional model structure into a two-dimensional image through the above steps, the discrete point cloud data can be represented in a structured manner, which can reduce the amount of model data and make the subsequent compression of the two-dimensional image simpler, and more image compression algorithms can be used.

[0106] In some embodiments, the first projection image set may be directly used as compressed data corresponding to the target model.

[0107] In some embodiments, the projection images in the first projection image set may be further compressed, thereby further improving storage efficiency. Figure 3 , method P300 may further include: P350: Perform image compression on each projection image in the first projection image set to obtain a second projection image set, and use the second projection image set as compressed data corresponding to the target model.

[0108] By further compressing the first projection image to obtain the second projection image, the model data can be further compressed, thereby further improving the storage efficiency while ensuring that the scene can be efficiently represented by the model data.

[0109] As described above, each projection image in the first projection image set may correspond to an attribute dimension. In some embodiments, the image compression process of each projection image may include: determining compression parameters applicable to compressing the projection image based on the attribute dimension corresponding to the projection image, and performing image compression processing on the projection image using the compression parameters.

[0110] The compression parameters determined for different attribute dimensions are different. For example, the compression parameter corresponding to attribute dimension A is a, the compression parameter corresponding to attribute dimension B is b, the compression parameter corresponding to attribute dimension C is c, the compression parameter corresponding to attribute dimension D is d, and the compression parameter corresponding to attribute dimension E is e.

[0111] In some embodiments, the same attribute dimension under different perspectives may correspond to the same compression parameter. Figure 7 The compression parameters corresponding to View1, View3, View5 and View7 in the attribute dimension A are the same. Similarly, the compression parameters corresponding to View1, View3, View5 and View7 in the attribute dimensions B~D are also the same.

[0112] In some embodiments, the compression parameter may be a quantization matrix. The quantization matrix may reduce the amount of image data by quantizing discrete cosine transform coefficients.

[0113] In the attribute information of the three-dimensional model, the physical meanings of the information representation of different attribute dimensions are different. Therefore, the physical features that need to be compressed in different attribute dimensions are also different during data compression. Therefore, in the above scheme, for the attribute dimension corresponding to the projection image, the compression parameters suitable for the attribute dimension are adaptively adopted, so that the projection images of different attribute dimensions can be compressed to the maximum extent while ensuring that no useful information is lost.

[0114] In some embodiments, the determination of compression parameters and the image compression processing of the projection image through the compression parameters can be achieved through the following steps: the projection image and the attribute dimension corresponding to the projection image are input into a pre-trained adaptive image compression network to perform image compression processing on the projection image through the adaptive image compression network.

[0115] Among them, the adaptive image compression network is trained to use different compression parameters for image compression for projected images of different attribute dimensions. The adaptive image compression network can determine the corresponding compression parameters based on the input attribute dimensions, and then compress the projected image based on the compression parameters. For example, the adaptive image compression network can be designed as a differentiable compression network of adaptive channels. The compression network can be combined with the algorithm network architecture of Diff-JPEG, adaptively optimize the quantization matrix according to the data characteristics of different feature channels (attribute dimensions), and use different quantization matrices to compress different feature channels, so as to maximize the compression and sparse representation of the image, thereby effectively improving the compression efficiency.

[0116] In some embodiments, the adaptive image compression network may be pre-trained and deployed on the compression device 110. The training data of the adaptive image compression network may include sample projection images of the sample model under different attribute dimensions and corresponding sample compression parameters. The training goal of the adaptive image compression network may include constraining the difference between the compressed image output by it and the pre-labeled compressed image to be within a second preset difference threshold. The second preset difference threshold may be determined based on experience or based on a network training process.

[0117] Figure 8 A schematic diagram of a second projection image set provided according to some embodiments of the present specification is shown. Figure 8 The process of obtaining the second projection image set corresponding to each attribute dimension based on the adaptive image compression network and the second projection image set are shown. Figure 8 As shown, the second projection image set includes projection images The projected image after compression , for the projected image The projected image after compression , for the projected image The projected image after compression , for the projected image The projected image after compression .

[0118] In the above scheme, the adaptive image compression network is obtained by pre-training, so that the adaptive image compression network has the ability to compress the projection image using different compression parameters based on different attribute dimensions. In this way, when compressing each projection image in the first projection image set, it is only necessary to input the projection image into the pre-trained adaptive image compression network, and the compression processing can be realized by adaptively selecting appropriate compression parameters according to the attribute dimension of the projection image, which can improve the processing efficiency of data compression.

[0119] At this point, the processor 220 has completed the compression of the model data.

[0120] In the process of compressing the model data, the processor 220 first selects some key perspectives from multiple shooting perspectives, and uses the projection information under the key perspectives to reconstruct the first sparse model with a reduced number of point clouds, and then renders the point cloud attribute information in the first sparse model so that the scene representation capability of the rendered second sparse model is equivalent to that of the target model, that is, the second sparse model can also achieve efficient representation of the scene using a limited number of point clouds. Furthermore, the compression method projects the second sparse model at the key perspective and converts the second sparse model into structured two-dimensional image data, which can reduce the amount of model data. Furthermore, the compression method further reduces the amount of model data by compressing the projected image of the second sparse model. The compression method provided in this specification can facilitate the deployment of scene models on end-side devices by compressing model data, and also reduces the resources required for model storage and transmission.

[0121] After compressing the target model, when the target model needs to be used, the decompression device 120 can decompress the compressed data of the target model to obtain decompressed data corresponding to the target model. The following is a detailed description of the decompression method of model data provided in the embodiments of this specification.

[0122] Fig. 9 1 shows a flow chart of a decompression method P900 of model data provided according to some embodiments of the present specification. As mentioned above, the decompression device 120 can execute the decompression method P900. Specifically, the processor 220 in the decompression device 120 can read the instruction set stored in its local storage medium, and then execute the decompression method P900 of the present specification according to the provisions of the instruction set. Fig. 9 As shown, method P900 may include: P910: Obtain compressed data corresponding to a target model, wherein the target model is a model obtained by three-dimensionally reconstructing images captured at M shooting angles, the target model includes location information and attribute information of multiple point clouds, M is an integer greater than 2, and the compressed data is obtained by compressing the target model using a compression method.

[0123] P920: Obtain a second projection image set from the compressed data, wherein the second projection image set includes compressed projection images corresponding to N key perspectives, where the N key perspectives are partial perspectives of the M shooting perspectives, and N is an integer greater than 1.

[0124] For example, the second projection image set may be the image set obtained in the above step P350.

[0125] P930: Perform image decompression processing on each projection image in the second projection image set to obtain a first projection image set.

[0126] The first projection image set is the projection images corresponding to the N key viewing angles. For example, the first projection image set may be the image set obtained in the above step P340.

[0127] As mentioned above, each projection image in the second projection image set corresponds to an attribute dimension. Therefore, in some embodiments, the decompression processing corresponding to each projection image can be implemented by the following steps: for the attribute dimension corresponding to the projection image, determining a decompression parameter applicable to decompressing the projection image, and performing image decompression processing on the projection image using the decompression parameter.

[0128] Among them, similar to the compression parameters, the decompression parameters determined for different attribute dimensions are different. In some embodiments, in order to ensure that the compressed data can be completely restored, the decompression parameters should maintain a one-to-one correspondence with the compression parameters. For example, when the compression parameter is a quantization matrix, the decompression parameter is an inverse quantization matrix, that is, an inverse matrix of the quantization matrix.

[0129] In some embodiments, the decompression of the projected image can be achieved through a network. For example, an adaptive image decompression network is pre-trained, and the adaptive image decompression network is trained to determine different decompression parameters for different attribute dimensions, and then decompress the projected image based on the decompression parameters. For example, the projected image and the attribute dimension corresponding to the projected image are input into the adaptive image decompression network, and the adaptive image decompression network can determine the required inverse quantization matrix based on the attribute dimension, and then use the inverse quantization matrix to decompress the projected image.

[0130] P940: Back-project the first projection image set at N key viewing angles to obtain decompressed data corresponding to the target model.

[0131] The information of the low-dimensional space can be restored to the high-dimensional space through back-projection. The information of the two-dimensional space can be converted into the information of the three-dimensional space through back-projection. The decompressed data obtained by back-projection can be a three-dimensional model. The three-dimensional model (decompressed data) can be generated by the images in the first projection image set. The process of generating the three-dimensional model (decompressed data) can refer to the process of generating the target model, which will not be repeated here. As an example, the three-dimensional model (decompressed data) can be generated by the 3DGS reconstruction technology.

[0132] The above decompression process can completely restore or approximately restore the decompressed data to the state of the second sparse model without considering other calculation errors. Since the scene representation capability of the second sparse model is equivalent to that of the target model, the second sparse model can be used as the decompressed data corresponding to the target model.

[0133] In some embodiments, the processor 220 may also perform other processing on the decompressed data, such as enhancing the point cloud in the decompressed data, etc., which is not limited in this specification.

[0134] At this point, the processor 220 has completed decompression of the model data.

[0135] In the process of decompressing the model data, by decompressing the projected image and performing a back-projection operation, the decompressed data of the second sparse model can be restored or substantially restored, so that a realistic scene can be restored, thereby ensuring the user's immersion and experience.

[0136] This specification further provides a non-temporary storage medium storing at least one set of executable instructions for model data compression. When the executable instructions are executed by a processor, the executable instructions guide the processor to implement the steps of method P300.

[0137] This specification also provides a non-temporary storage medium storing at least one set of executable instructions for decompressing model data. When the executable instructions are executed by a processor, the executable instructions guide the processor to implement the steps of method P900.

[0138] In some possible implementations, various aspects of this specification may also be implemented in the form of a program product, which includes program code. For example, when the program product is run on the compression device 110, the program code is used to enable the compression device 110 to perform the steps of the compression method P300 described in this specification. For another example, when the program product is run on the decompression device 120, the program code is used to enable the decompression device 120 to perform the steps of the decompression method P900 described in this specification.

[0139] The program product for implementing the above method can be stored in a portable compact disk read-only memory (CD-ROM) and run on the compression device 110 or the decompression device 120. However, the program product of this specification is not limited to this. In this specification, a readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operations of this specification can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0140] The program code for implementing the compression method of the model data may be executed completely on the compression device 110, partially on the compression device 110, as an independent software package, partially on the compression device 110 and partially on a remote computing device, or completely on the remote computing device. The program code for implementing the decompression method of the model data may be executed completely on the decompression device 120, partially on the compression device 120, as an independent software package, partially on the compression device 120 and partially on a remote computing device, or completely on the remote computing device. In the case of a remote computing device, the remote computing device may be connected to the compression device 110 or the decompression device 120 via the transmission medium 130, or may be connected to an external computing device.

[0141] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0143] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0144] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0145] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0146] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for compressing model data, comprising: Obtaining a target model to be compressed, wherein the target model is a model obtained by performing three-dimensional reconstruction on images captured at M shooting angles, the target model includes position information and attribute information of multiple point clouds, and M is an integer greater than 2; Determine N key viewing angles from the M shooting viewing angles, and obtain a first sparse model by projecting and back-projecting the position information of each point cloud in the target model under the N key viewing angles, wherein the first sparse model includes the position information of part of the point cloud in the target model, and N is an integer less than M; Rendering the attribute information of each point cloud in the first sparse model to obtain a second sparse model, so that the scene representation capability of the second sparse model approaches the scene representation capability of the target model; Projecting the position information and attribute information of each point cloud in the second sparse model at the N key viewing angles to obtain a first projection image set; as well as Perform image compression on each projection image in the first projection image set to obtain a second projection image set, and use the second projection image set as compressed data corresponding to the target model.

2. The method according to claim 1, wherein: The determining of N key viewing angles from the M shooting viewing angles includes: Based on the information amounts respectively contained in the captured images under the M shooting perspectives, perspectives whose information amounts satisfy the target condition are selected from the M shooting perspectives as the N key perspectives.

3. The method according to claim 2, wherein: A viewing angle difference between two adjacent key viewing angles among the N key viewing angles is greater than a preset threshold.

4. The method according to claim 1, wherein: The first sparse model is obtained by projecting and back-projecting the position information of each point cloud in the target model at N key viewing angles, including: Projecting the position information of each point cloud in the target model at the N key viewing angles to obtain N projection images; and The first sparse model is obtained by back-projecting the N projection images at the N key viewing angles.

5. The method according to claim 4, wherein: The projecting the position information of each point cloud in the target model under the N key viewing angles to obtain N projection images includes: For each of the N key perspectives, based on the camera observation matrix corresponding to the key perspective, the position information of each point cloud in the target model is projected onto the camera plane corresponding to the key perspective to obtain a projection image corresponding to the key perspective.

6. The method according to claim 4, wherein: The back-projecting the N projection images at the N key viewing angles to obtain the first sparse model includes: For each key perspective of the N key perspectives, based on a camera observation matrix corresponding to the key perspective, back-project each pixel point in the projection image corresponding to the key perspective into a three-dimensional space to obtain a point cloud subset corresponding to the key perspective; and The first sparse model is generated based on the point cloud subsets corresponding to the N key perspectives respectively.

7. The method according to claim 1, wherein: The rendering of the attribute information of each point cloud in the first sparse model to obtain a second sparse model includes: Using the captured images at the N key viewing angles as supervision, the attribute information of each point cloud in the first sparse model is rendered and trained to obtain a second sparse model, wherein the following training objective is adopted during the rendering training process: minimizing the difference between the projection image of the second sparse model at the N key viewing angles and the captured images at the N key viewing angles.

8. The method according to claim 7, wherein: The representation dimension of the attribute information of the point cloud in the second sparse model is higher than the representation dimension of the attribute information of the point cloud in the target model.

9. The method according to claim 1, wherein: Each projection image in the first projection image set corresponds to an attribute dimension, and the image compression process of each projection image includes: Based on the attribute dimension corresponding to the projection image, a compression parameter suitable for compressing the projection image is determined, and image compression processing is performed on the projection image using the compression parameter, wherein different compression parameters are determined for different attribute dimensions.

10. The method according to claim 9, wherein: The step of determining a compression parameter applicable to compressing the projection image based on the attribute dimension corresponding to the projection image, and performing image compression processing on the projection image by using the compression parameter, includes: The projection image and the attribute dimension corresponding to the projection image are input into a pre-trained adaptive image compression network so as to perform image compression processing on the projection image through the adaptive image compression network, wherein the adaptive image compression network is trained to use different compression parameters for image compression for projection images with different attribute dimensions.

11. A model data compression device, comprising: at least one storage medium storing at least one instruction set for data compression; as well as at least one processor, in communication with the at least one storage medium, Wherein, when the compression device is running, the at least one processor reads the at least one instruction set and executes the method according to any one of claims 1 to 10 according to the instructions of the at least one instruction set.

12. A method for decompressing model data, comprising: Obtaining compressed data corresponding to a target model, wherein the target model is a model obtained by three-dimensionally reconstructing images captured at M shooting angles, the target model includes location information and attribute information of a plurality of point clouds, M is an integer greater than 2, and the compressed data is obtained by compressing the target model by the method according to any one of claims 1 to 10; Obtaining a second projection image set from the compressed data, the second projection image set including compressed projection images corresponding to N key viewing angles, where the N key viewing angles are partial viewing angles of the M shooting viewing angles, and N is an integer greater than 1; performing image decompression processing on each projection image in the second projection image set to obtain a first projection image set; and The first projection image set is back-projected at the N key viewing angles to obtain decompressed data corresponding to the target model.

13. The method according to claim 12, wherein: Each projection image in the second projection image set corresponds to an attribute dimension, and the decompression processing process corresponding to each projection image includes: Decompression parameters applicable to decompressing the projection image are determined for the attribute dimensions corresponding to the projection image, and image decompression processing is performed on the projection image using the decompression parameters, wherein different decompression parameters are determined for different attribute dimensions.

14. A decompression device for model data, comprising: at least one storage medium storing at least one instruction set for decompressing data; as well as at least one processor, in communication with the at least one storage medium, Wherein, when the decompression device is running, the at least one processor reads the at least one instruction set and executes the method according to any one of claims 12-13 according to the instructions of the at least one instruction set.

Images