System and method for LLM-based quality assessment for 3D media

Abstract

Systems and methods for quality assessment of three-dimensional visual content are provided. An exemplary method includes: receiving a point cloud comprising a plurality of data points representing; obtaining, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encoding the input datasets for the modalities via separate encoders for the modalities to generate encoded datasets for the modalities; projecting the encoded datasets for the modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the modalities; providing the embeddings of the encoded datasets for the modalities to a language model; and obtaining, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

Description

SYSTEM AND METHOD FOR LLM-BASED QUALITY ASSESSMENT FOR 3D MEDIATECHNICAL FIELD

[0001] The present disclosure related to Large Language Model (LLM)-based assessment of perceptual quality of 3-Dimensional (3D) content (point clouds or meshes).BACKGROUND

[0002] Quality Assessment (QA) algorithms aim to predict the perceptual quality of media, i.e. how ‘good’ they look to humans. Currently, QA algorithms for 3-Dimensional (3D) content (point clouds or meshes) only provide a numeric score, and do not provide context behind the score. As an example, a point cloud with color issues and another with capture artifacts might receive the same score on an axis of ‘goodness’ but would have to be handled quite differently by a post-processing (e.g. compression or enhancement) pipeline.SUMMARY

[0003] While there are some quality 2-Dimensional (2D), there’s a need for improve 3D content quality assessment.

[0004] An implementation of the present disclosure includes a computer-implemented method including any one or more of receiving a point cloud including a plurality of data points representing a 3D object; obtaining, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities to provide or generate encoded datasets for the two or more modalities; projecting the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model to provide or generate embeddings of the encoded datasets for the two or more modalities; providing the embeddings of the encoded datasets for the two or more modalities to a language model; and obtaining, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0005] In a related implementation, the one or more outputs related to the perceptual quality assessment for the point cloud comprise any one or more of the following: a continuous quality metric for the point cloud, a qualitative description of one or more dominant perceptual distortions for the point cloud, a localized area where the one or more dominant perceptual distortions are likely located within the point cloud, a chain of reasoning for the continuous quality metric for thepoint cloud and / or the qualitative description of the one or more dominant perceptual distortions for the point cloud.

[0006] In a related implementation, the method further includes providing the one or more outputs to a system or function responsible for processing of the point cloud. The point cloud includes a number N points, and the input datasets for the two or more modalities includes one or more input datasets each comprising less than N points. The point cloud include N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising a subset of the N points of the point cloud.

[0007] In a related implementation, the point cloud comprises N points, and the input datasets for the two or more modalities comprise any one or more of the following: a first input dataset comprising n<N samples from the N points of the point cloud selected via a furthest-point sampling technique; a second input dataset comprising n<N samples from the N points of the point cloud, the n samples being obtained via full scale sampling of two (or more) local clusters of the point cloud; and a third input dataset comprising n<N samples from the N points of the point cloud, the n samples being obtained via half-scale sampling of two (or more) local clusters of the point cloud.

[0008] Encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities includes encoding the first input dataset, the second input dataset, and / or the third input dataset via a point cloud encoder. The input datasets for the two or more modalities comprise one or more image projections of the point cloud. The one or more image projections are comprised of two-dimensional rendered cube views. Encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the one or more image projections via a vision encoder. The input datasets for the two or more modalities comprise textual information.

[0009] The textual information comprises any one or more of the following: information about one or more psycho visual aspects; taxonomy information including information about one or more types of dominant perceptually relevant information likely to affect visual perception of the point cloud; localization information that indicates where perceived low-quality areas are likely to be located in the point cloud; display type; distance of the point cloud; and / or textual descriptions of other projection types including how they were obtained.

[0010] Encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities includes encoding the textual information via a vision encoder. Projecting the encoded datasets for the two or more modalities into the common embedding space for processing in the large language model includes projecting each encoded dataset into the common embedding space via a projection function for a respective one of the two or moremodalities.

[0011] Another implementation includes computing system adapted to perform any one or more of the following: receive a point cloud comprising a plurality of data points representing a 3D object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0012] In related implementation, the system is further adapted to perform the method described herein.

[0013] Another implementation includes a computing system including one or more processors and memory configured to store instructions executable by the one or more processors. The instructions cause the computing system to: receive a point cloud comprising a plurality of data points representing a 3D object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0014] Another implementation includes a computer program including instructions that, when executed on at least one processor, cause the processor to carry out the method according to any of implementations above.

[0015] Another implementation includes a carrier containing the computer program of the methods described herein, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium.

[0016] Another implementation includes a non-transitory computer-readable medium includinginstructions executable by processing circuitry of a computing system. By executing the instructions the computing system is operable to: receive a point cloud comprising a plurality of data points representing a 3D object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawing figures incorporated in and forming a part of this specification illustrate several aspects of the disclosure, and together with the description serve to explain the principles of the disclosure.

[0018] Figure 1 illustrates a multi-modal Large Language Model (LLM)-based quality assessment procedure for accessing a perceptual quality of a 3-Dimensional (3D) media, which in the illustrated example is a point cloud, in accordance with embodiments of the present disclosure;

[0019] Figure 2 is a flow chart that illustrates the operation of a computing system to perform the procedure of Figure 1, in accordance with embodiments of the present disclosure;

[0020] Figure 3 shows a computing system in accordance with some embodiments of the present disclosure; and

[0021] Figure 4 is a block diagram illustrating a virtualization environment in which functions implemented by some embodiments may be virtualized in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0022] The embodiments set forth below represent information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts andapplications fall within the scope of the disclosure.

[0023] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0024] There currently exist certain challenge(s). Recent work in the image quality assessment domain has leveraged Large Multimodal Models (LMMs) to not only achieve State of the Art (SOTA) metrics on traditional score-based quality assessment, but also offer context for their decision via a description of the issues. However, this work has not yet been translated to 3- Dimensional (3D) content.

[0025] Existing technologies for Quality Assessment (QA) of media provide quality metrics for 2-Dimensional (2D) visual data across modalities, but these metrics suffer at detecting problems specific to the 3D domain - particularly at scale and with the breadth available with modern Large Multimodal Model (LLM)-based techniques. In addition, these technologies are not well-suited to provide explainable quality scores for perception - particularly across different modalities of inputs.

[0026] Some of the existing technologies are:• Q-Align: (see https: / / github.com / Q-Future / Q-Align / tree / main71ora_finetune) o Provides an LLM perceptual quality indicator for projections of 3D assets. Does not provide a strong explainability component in its evaluation and is not suited for multi-modal evaluation - particularly with the inclusion of 3D data.• LS-PCQA: (see https: / / smt.sjtu.edu.cn / database / large-scale-point-cloud-quality- assessment-dataset-ls-pcqa / ) o Provides a data set for assessing quality by providing distortions across a number of reference point clouds.• Quality assessment in 3D meshes (see Yana Nehme, Johanna Delanoy, Florent Dupont, Jean-Philippe Farrugia, Patrick Le Callet, & Guillaume Lavoue. (2023). Textured Mesh Quality Assessment: Large-Scale Dataset and Deep Learning-based Quality Metric.) o Provides a quality assessment method based on a Deep Neural Network (DNN) architecture for textured 3D meshes. Does not handle multi-modal inputs and also does not work on point clouds.

[0027] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. Embodiments of systems and methods are disclosed herein that provide LLM- based perceptual quality assessment for point cloud data. To achieve this, several different modalities of inputs are combined using a multi-modal modeling architecture and results areproduced that, in one embodiment, provide a continuous quality metric alongside a qualitative description of the dominant perceptual distortions, a localized area where these distortions are likely located, and a chain of reasoning for this score and categorization. Note however that the produced results may include any one or more of the following: a continuous quality metric alongside a qualitative description of the dominant perceptual distortions, a localized area where these distortions are likely located, and a chain of reasoning for this score and categorization.

[0028] Embodiments of the present disclosure may include any one or more of the following aspects:• The integration of complex point cloud, textual, and 2D projection data into a multi-modal modeling architecture• The use of a multi-modal LLM architecture capable of simultaneously processing geometric, projection-based, and text-based inputs to assess the quality of point cloud information• The use of modality-aware projection of embeddings in the multi-modal modeling process for downstream training of an ensemble LLM• The combination of feature inputs into scored, localized, hierarchically classified, and reasoned quality assessments for perception

[0029] Certain embodiments may provide one or more of the following technical advantage(s). Information of the kind produced by embodiments of the present disclosure could be helpful in several ways, including any one or more of the following:• Individual analysis: A downstream process (say a codec) can decide how to handle a piece of media based on its specific characteristics or distortions and detect issues after processing.• Systematic analysis: An entire dataset can be quickly analyzed for systemic issues, which can inform decisions about how to handle it, such as in data curation.• Explainability: It can be verified if the scoring process of the model matches that of a human, such as by checking if it is able to identify the correct root cause and location of a distortion.

[0030] Figure 1 illustrates a multi-modal LLM-based quality assessment (QA) procedure for 3D media, in accordance with embodiments of the present disclosure. Here, the 3D media is a point cloud. As illustrated, given a point cloud of arbitrary size (say, N) and number of features (say, D) such that each of the N points in the point cloud is associated with D features where these features may be, for example, RBG values, intensity / reflectance, curvature, sematic labels, and / orthe like , the following inputs segments (also referred to herein as “input datasets”) are obtained for the point cloud, e.g., derived from the point cloud (e.g., derived from a dataset consisting of the raw collection of points of point cloud data (three dimensional of three varieties):

[0031] a. Fixed-size smaller segments of the larger point cloud, say of size n (e.g., n = 8192), where n « N, of three types: n-dimensional segments from three different point cloud samples:1. Furthest-point sampling of the entire point cloud, which captures a global, high-level representation of the point cloud2. Two (or more) local clusters sampled at full scale (resolution), which capture local, low- level variations at the finest level of the point cloud

[0032] Note that the two (or more) local clusters may be selected in any desired manner. As a non-limiting example, two clusters may be selected by randomly sampling two points, then doing k-nearest neighbors around these two points with or without intersection to form two local clusters.3. Two (or more) local clusters at half scale (resolution), which capture local variations at a higher level of the point cloud• Note that the two (or more) local clusters may be selected in any desired manner. As a non-limiting example, two clusters may be selected by randomly sampling two points (seed points), then doing k-nearest neighbors around these two points with or without intersection to form two local clusters. Note that the two (or more) local clusters may be selected using the same two (or more) randomly sampled seed points as used for the full scale sampling above or may be selected using different randomly sampled seed points.

[0033] Sampling in this way not only allows us to extract information at multiple levels, but also addresses the challenge of processing highly dense point clouds efficiently.

[0034] b. Image projections comprised of two-dimensional rendered cube views, where point size, projection types, and camera locations are set parameters. Known techniques for cubemap projections can be used. Note that point size determines the rendered size of each point on the 2D image based on the point size in the 3D image. Projection type refers to the type of projection — orthogonal or perspective projections can be used. Camera location refers to where the projections are “centered” (from where the mathematical projections should be computed), and it is typically the origin of the cube.

[0035] c. Textual information:1. Psychovisual aspects, such as those that might be provided by human studies and those that might be relevant for perceptual quality, such as the type of experiment (e.g., fullreference absolute category rating, no-reference two-alternative forced choice, etc.), thelevel of interactivity allowed to the human subject etc.2. Taxonomy information, including the type of dominant perceptually relevant information likely to affect perception of the point cloud. Taxonomical information that could be included for point cloud would be a description of what the point cloud might be intended to do, or be used for, in what type of media, for use among what kind of audience, etc.3. Localization information indicating where perceived low-quality areas are likely to be located• Localization information is most likely available in synthetically generated or human labeled data4. Display type (e.g., type of display on which the point cloud is to be rendered)5. Distance of point cloud (i.e., measure of the distance between points in the point cloud such as, e.g., average pairwise distance, Hausdorff distance, etc.).6. Textual descriptions of the other two projection types (i.e., image projections and fixed- size smaller segments), including how they were obtained

[0036] Feed input data into distinct submodules:

[0037] a. Point cloud processing module, comprised of a point cloud encoder (e.g., PointBERT or PointMamba) transforming each point cloud into encoded outputs, e.g., 25 encoded outputs of size 784. As will be appreciated by those of ordinary skill in the art of point clouds, the point cloud encoder takes a collection of points in 3D space (a point cloud) and converts it into a format that a computer can understand and work with more easily. Think of a point cloud as a big scatter of dots floating in space, each dot representing a specific spot on an object or in an environment. The encoder’s job is to take all these dots and figure out how to organize them into a simpler, more structured form. This could involve figuring out patterns, grouping points together, or capturing important details like shapes and surfaces. The encoded output is a feature vector that can be more easily used in the learning process. Note that the particular point cloud encoder used may vary depending on the particular implementation, but in general a desired point cloud encoder can be used.

[0038] b. A Vision Encoder (e.g., CLIP) transforming the image projections into encoded outputs, e.g., 6 encoded outputs of size 784.

[0039] c. A text encoder transforming textual data into n encoded tokens, e.g., of size 784.

[0040] Project outputs into a common embedding space for processing in a language model(using, e.g., a common off-the-shelf linear projection method). The common embedding space is, in one embodiment, a series of projected vectors where different kinds of data (e.g., words,sentences, or images) are plotted in such a way that similar or related items are close to each other in the corresponding vector space, making it easier for the model to connect and understand them across different forms of input. Techniques of this projection are known by those of ordinary skill in the art of LLMs and LMMs. For example, in one embodiment, this project is performed by: (1) converting text and image data into embeddings, (2) normalize the embeddings, (3) apply a linear transform to project the embeddings into a common embedding space. Optionally, the projection may be trained to improve alignment. The projected embeddings (in the common embedding space) may then be evaluated and used for multi-modal tasks.

[0041] a. Modality-aware projection - Each encoder has its own projection module, all of which are trained jointly to improve alignment.

[0042] Feed projected outputs from submodules into a large language model (e.g., a transformer-based model like LLaMA), with minor modifications to handle specific modalities:

[0043] a. Modality-aware projection -> k and v weights differ between modalities. More specifically, the model handles each type of data (modality) differently when transforming it. This means that the process of projecting text data might be different from projecting image data. This is important because different modalities have different characteristics (e.g., text is sequential, while images are spatial), so the model needs to account for these differences to process each type of data effectively. In transformer models, the attention mechanism is based on training a series of Query, Key, and Value weights. In this method, the weights used in the attention mechanism (k and v) are not the same for each modality. For example, the way the model "attends to" or focuses on parts of an image (using the k and v weights) will differ from how it attends to parts of a text. This ensures that the model can capture the unique aspects of each modality. For instance, the model might need to focus on different image regions for understanding an image, compared to how it focuses on words in a sentence.

[0044] Feed the model a prompt designed to extract outputs from the decoder. This prompt may include as many or as few of the outcomes below as are desired:

[0045] a. The outputs of this model will be: i. A quality score, which is represented by a probability distribution on ordinal quality categories (e.g., a five-point Likert scale). To obtain the final score, values are assigned to each ordinal category, and a smoothing function is applied where the contribution of each category is proportional to its probability. Example: Excellent, Very Good, Average, Below Average, Poor. This scale could be coarsened or refined depending on the needs of the user. ii. An identified category of perceptual distortions based on taxonomical inputs (see detailsabout the taxonomical inputs above). iii. A chain-of-reasoning-based text response outlining the qualitative reasoning motivating the quality score. iv. A localized area (e.g., an identified quadrant or octant) in the point cloud identifying the region most significantly associated with the distortion. Note that this can be done is multiple ways. For example, textual information about perceived areas of low quality associated with input data into the model can provide learning that allows the model to identify this information efficiently. One could also train the model with synthetically generated distortions and teach the model to localize based on a localized area approach as described here.

[0046] Here, feeding the model a prompt means giving the model a prompt to extract the desired output. This prompt may be proceeded by a fine-tuning step teaching the model to refine some of the outputs to the purposes of the user. In the case of the quality score for example, this means providing a fine tuning prompt instructing the model first to rate each input across the given Likert scale, then assign a value to each category representing the relevance of each category to the input. The model will then transform this output across the ordinal categories (Likert scale) into a single figure using a smoothing function (e.g., softmax, etc.) to produce an output. This fine- tuning clears the way for a decoder-only prompt for the first output, for example, here asking “What is the quality of this point cloud?” This would trigger scores based on this process.

[0047] Figure 2 is a flow chart that illustrate the procedure of Figure 1. In this example, the procedure is performed by a computing system (e.g., a server computer, a personal computer, or the like). Note that not all of the details provided above with respect to Figure 1 are repeated here. However, it is to be understood that the details provided above are equally applicable to the corresponding steps of the flowchart of Figure 2.

[0048] As illustrated, computing system receives a point cloud comprising data points representing a 3D object (action 200). The computing system obtains, for the point cloud, input datasets for two or more modalities (action 202). The two or more modalities comprising any two or more (and preferably all) of the following modalities: a point cloud modality, a visual modality, and a textual modality. In one embodiment, the point cloud comprises N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising less than N points. In another embodiment, the point cloud comprises N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising a subset of the N points of the point cloud. In one embodiment, the point cloud comprises N points, and the input datasets for the two or more modalities comprise any one or more of the following:• a first input dataset (202A) comprising n<N samples from the N points of the point cloud selected via a furthest-point sampling technique;• a second input dataset (202B) comprising n<N samples from the N points of the point cloud, the n samples being obtained via full scale sampling of two (or more) local clusters of the point cloud;• a third input dataset (202C) comprising n<N samples from the N points of the point cloud, the n samples being obtained via half-scale sampling of two (or more) local clusters of the point cloud.In one embodiment, the input datasets additionally or alternatively includes a fourth input dataset (202D) including one or more image projections (e.g., two-dimensional rendered cube views) of the point cloud and / or a fifth input dataset (202E) including textual information (see, for example, the examples given above).

[0049] The computing system encodes the input datasets for the two or more modalities via separate encoders for the two or more modalities (action 204). For example, the first, second, and third input datasets may be encoded via a point cloud encoder(s) (e.g., PointBERT or PointMamba), the fourth input dataset may be encoded by a vision encoder (e.g., CLIP), and the fifth input dataset may be encoded via a text encoder.

[0050] The computing system projects the encoded input datasets for the two or more modalities into a common embedding space for processing in a large language model (action 206). This common embedding space is known and may vary depending on the particular application. The embeddings of the encoded input datasets for the two or more modalities are provided to a large language model (action 208), as described above. The computing system then obtains (e.g., based on one or more prompts) one or more outputs from the large language model related to a perceptual quality assessment for the point cloud (action 212). As discussed above, the one or more outputs may include any one or more of the following: a quality metric (e.g., a continuous quality metric, i.e., a quality metric that is not limited to being an integer) for the point cloud, a qualitative description of one or more dominant perceptual distortions for the point cloud, a localized area where the one or more dominant perceptual distortions are likely located within the point cloud, a chain of reasoning for the continuous quality metric for the point cloud and / or the qualitative description of the one or more dominant perceptual distortions for the point cloud. Note that

[0051] The computing system may provide the one or more outputs to another entity (e.g., another computing system or function) (action 212). The other entity may, for example, take these outputs input account when processing the point cloud.

[0052] It should be noted that while the description above describes the process of Figure 2 as being performed by a computing system, the actions illustrated in Figure 2 may alternatively be performed by two or more computing systems in any desired manner. For example, some of the actions may be performed by a first computing system and some other actions may be performed by a second computing system that is separate from the first computing system.

[0053] LoRA Implementation

[0054] Due to the complexity of fine-tuning several models within this architecture, we propose the use of low rank adaptation for each model training step to reduce the computational complexity of the training process. Low-rank adaptation (LoRA) is a technique that fine-tunes large models by updating only a small set of low -rank matrices, significantly reducing the number of trainable parameters. This approach is efficient in terms of both memory and computation, making it ideal for adapting large models to specific tasks with limited resources.

[0055] Figure 3 shows a computing system 300 in which embodiments of the present disclosure may be implemented. For example, the computing system 300 may be a server computer, a personal computer, or the like. The computing system 300 includes processing circuitry 302 that is operatively coupled via a bus 304 to an input / output interface 306, a power source 308, memory 310, a communication interface 312, and / or any other component, or any combination thereof. Certain computing system may utilize all or a subset of the components shown in Figure 3. The level of integration between the components may vary from one computing system to another computing system . Further, certain computing system may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0056] The processing circuitry 302 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 310. The processing circuitry 302 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general purpose processors, such as a microprocessor or Digital Signal Processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 302 may include one or more Central Processing Units (CPUs).

[0057] In the example, the input / output interface 306 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor,a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the computing system 300. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0058] In some embodiments, the power source 308 includes power circuitry for delivering power from the power source 308 itself (e.g., in case that the power source 308 is a battery or the like), and / or an external power source, to the various parts of the computing system 300 via input circuitry or an interface such as an electrical power cable. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 308 to make the power suitable for the respective components of the computing system 300 to which power is supplied.

[0059] The memory 310 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid state memory, remotely mounted memory, magnetic media, optical media, RAM, ROM, mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD), or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable, and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 302. The memory 310 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 302 and utilized by the computing system 300. The memory 310 may be used to store any calculations made by the processing circuitry 302 and / or any data received via the communication interface 312. In some embodiments, the processing circuitry 302 and the memory 310 are integrated. In an example embodiment, the memory 310 stores software configured to cause the computing system 300 to perform the process of Figure 1 and / or Figure 2, or any subset of the actions comprised in that process.

[0060] The processing circuitry 302 may be configured to communicate with a network using the communication interface 312. The communication interface 312 may comprise one or morecommunication subsystems for wired and / or wireless communication. The communication interface 312 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wired or wireless communication (e.g., another computing system or a network node in an access network).

[0061] In the illustrated embodiment, communication functions of the communication interface 312 may include cellular communication, WiFi communication, LPWAN communication, data communication, voice communication, multimedia communication, short- range communications such as Bluetooth, NFC, location-based communication such as the use of the Global Positioning System (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband CDMA (WCDMA), GSM, LTE, NR, UMTS, WiMax, Ethernet, Transmission Control Protocol / Internet Protocol (TCP / IP), Synchronous Optical Networking (SONET), Asynchronous Transfer Mode (ATM), Quick User Datagram Protocol Internet Connection (QUIC), Hypertext Transfer Protocol (HTTP), and so forth.

[0062] Figure 4 is a block diagram illustrating a virtualization environment 400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices, and networking resources. As used herein, virtualization can be applied to any device described herein (e.g., the computing system), or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more Virtual Machines (VMs) implemented in one or more virtualization environments 400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a computing system. Further, in some embodiments, the node may be entirely virtualized. Virtualization may facilitate distributed implementations of the computing system.

[0063] Applications 402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0064] Hardware 404 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, an input / output interface, and so forth. Softwaremay be executed by the processing circuitry to instantiate one or more virtualization layers 406 (also referred to as hypervisors or Virtual Machine Monitors (VMMs)), provide VMs 408A and 408B (one or more of which may be generally referred to as VMs 408), and / or perform any of the functions, features, and / or benefits described in relation with some embodiments described herein. The virtualization layer 406 may present a virtual operating platform that appears like networking hardware to the VMs 408.

[0065] The VMs 408 comprise virtual processing, virtual memory, virtual networking, or interface and virtual storage, and may be run by a corresponding virtualization layer 406. Different embodiments of the instance of a virtual appliance 402 may be implemented on one or more of VMs 408, and the implementations may be made in different ways.

[0066] The hardware 404 may be implemented in a standalone network node with generic or specific components. The hardware 404 may implement some functions via virtualization. Alternatively, the hardware 404 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 410, which, among others, oversees lifecycle management of the applications 402. In some embodiments, some signaling can be provided with the use of a control system 412 which may alternatively be used for communication between hardware nodes and other nodes.

[0067] Although the computing system described herein may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions, and methods disclosed herein. Determining, calculating, obtaining, or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non- computationally intensive functions of any of such components may be implemented in softwareor firmware and computationally intensive functions may be implemented in hardware.

[0068] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0069] The following enumerated clauses provide some examples of embodiments of the present disclosure.

[0070] Embodiment 1. A computer-implemented method comprising any one or more of the following: receiving (200) a point cloud comprising a plurality of data points representing a 3- dimensional, 3D, object; obtaining (202), for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encoding (204) the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; projecting (206) the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; providing (208) the embeddings of the encoded datasets for the two or more modalities to a language model; and obtaining (210), from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0071] Embodiment 2. The method of embodiment 1 , wherein the one or more outputs related to the perceptual quality assessment for the point cloud comprise any one or more of the following: a continuous quality metric for the point cloud, a qualitative description of one or more dominant perceptual distortions for the point cloud,a localized area where the one or more dominant perceptual distortions are likely located within the point cloud, a chain of reasoning for the continuous quality metric for the point cloud and / or the qualitative description of the one or more dominant perceptual distortions for the point cloud.

[0072] Embodiment 3. The method of embodiment 1 or 2, further comprising providing (212) the one or more outputs to a system or function responsible for processing of the point cloud.

[0073] Embodiment 4. The method of any of embodiments 1 to 3, wherein the point cloud comprises N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising less than N points.

[0074] Embodiment 5. The method of any of embodiments 1 to 3, wherein the point cloud comprises N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising a subset of the N points of the point cloud.

[0075] Embodiment 6. The method of any of embodiments 1 to 5, wherein the point cloud comprises N points, and the input datasets for the two or more modalities comprise any one or more of the following: a first input dataset comprising n<N samples from the N points of the point cloud selected via a furthest-point sampling technique; a second input dataset comprising n<N samples from the N points of the point cloud, the n samples being obtained via full scale sampling of two (or more) local clusters of the point cloud; a third input dataset comprising n<N samples from the N points of the point cloud, the n samples being obtained via half-scale sampling of two (or more) local clusters of the point cloud.

[0076] Embodiment 7. The method of embodiment 6, wherein encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the first input dataset, the second input dataset, and / or the third input dataset via a point cloud encoder.

[0077] Embodiment 8. The method of any of embodiments 1 to 7, wherein the input datasets for the two or more modalities comprise one or more image projections of the point cloud.

[0078] Embodiment 9. The method of embodiment 8, wherein the one or more image projections are comprised of two-dimensional rendered cube views.

[0079] Embodiment 10. The method of embodiment 8 or 9, wherein encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the one or more image projections via a vision encoder.

[0080] Embodiment 11. The method of any of embodiments 1 to 10, wherein the input datasets for the two or more modalities comprise textual information.

[0081] Embodiment 12. The method of embodiment 11, wherein the textual information comprises any one or more of the following: information about one or more psycho visual aspects; taxonomy information including information about one or more types of dominant perceptually relevant information likely to affect visual perception of the point cloud; localization information that indicates where perceived low-quality areas are likely to be located in the point cloud; display type; distance of the point cloud; textual descriptions of other projection types including how they were obtained.

[0082] Embodiment 13. The method of embodiment 11 or 12, wherein encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the textual information via a vision encoder.

[0083] Embodiment 14. The method of any of embodiments 1 to 13, wherein projecting the encoded datasets for the two or more modalities into the common embedding space for processing in the large language model comprises projecting each encoded dataset into the common embedding space via a projection function for a respective one of the two or more modalities.

[0084] Embodiment 15. A computing system adapted to perform any one or more of the following: receive a point cloud comprising a plurality of data points representing a 3 -dimensional, 3D, object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0085] Embodiment 16. The computing system of embodiment 15, further adapted to performthe method any of embodiments 2 to 14.

[0086] Embodiment 17. A computing system comprising: one or more processors; and memory configured to store instructions executable by the one or more processors whereby the computing system is caused to perform any one or more of the following: receive a point cloud comprising a plurality of data points representing a 3- dimensional, 3D, object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0087] Embodiment 18. A computer program comprising instructions which, when executed on at least one processor, cause the processor to carry out the method according to any of embodiments 1 to 14.

[0088] Embodiment 19. A earner containing the computer program of embodiment 18, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium.

[0089] Embodiment 20. A non-transitory computer-readable medium comprising instructions executable by processing circuitry of a computing system, whereby the computing system is operable to: receive a point cloud comprising a plurality of data points representing a 3-dimensional, 3D, object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality;encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

[0090] Those skilled in the art will recognize improvements and modifications to the embodiments of the present disclosure. All such improvements and modifications are considered within the scope of the concepts disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: receiving (200) a point cloud comprising a plurality of data points representing a 3- dimensional, 3D, object; obtaining (202), for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encoding (204) the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; projecting (206) the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; providing (208) the embeddings of the encoded datasets for the two or more modalities to a language model; and obtaining (210), from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

2. The method of claim 1 , wherein the one or more outputs related to the perceptual quality assessment for the point cloud comprise any one or more of the following: a continuous quality metric for the point cloud, a qualitative description of one or more dominant perceptual distortions for the point cloud, a localized area where the one or more dominant perceptual distortions are likely located within the point cloud, a chain of reasoning for the continuous quality metric for the point cloud and / or the qualitative description of the one or more dominant perceptual distortions for the point cloud.

3. The method of claim 1 or 2, further comprising providing (212) the one or more outputs to a system or function responsible for processing of the point cloud.

4. The method of any of claims 1 to 3, wherein the point cloud comprises N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising less than N points.

5. The method of any of claims 1 to 3, wherein the point cloud comprises N points, and the input datasets for the two or more modalities comprise one or more input datasets each comprising a subset of the N points of the point cloud.

6. The method of any of claims 1 to 5, wherein the point cloud comprises N points, and the input datasets for the two or more modalities comprise any one or more of the following: a first input dataset including n<N samples from the N points of the point cloud selected via a furthest-point sampling technique; a second input dataset including n<N samples from the N points of the point cloud, the n samples being obtained via full scale sampling of two (or more) local clusters of the point cloud; a third input dataset including n<N samples from the N points of the point cloud, the n samples being obtained via half-scale sampling of two (or more) local clusters of the point cloud.

7. The method of claim 6, wherein encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the first input dataset, the second input dataset, and / or the third input dataset via a point cloud encoder.

8. The method of any of claims 1 to 7, wherein the input datasets for the two or more modalities comprise one or more image projections of the point cloud.

9. The method of claim 8, wherein the one or more image projections are comprised of two- dimensional rendered cube views.

10. The method of claim 8 or 9, wherein encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the one or more image projections via a vision encoder.

11. The method of any of claims 1 to 10, wherein the input datasets for the two or more modalities comprise textual information.

12. The method of claim 11 , wherein the textual information comprises any one or more of the following: information about one or more psycho visual aspects; taxonomy information including information about one or more types of dominantperceptually relevant information likely to affect visual perception of the point cloud; localization information that indicates where perceived low-quality areas are likely to be located in the point cloud; display type; distance of the point cloud; textual descriptions of other projection types including how they were obtained.

13. The method of claim 11 or 12, wherein encoding the input datasets for the two or more modalities via separate encoders for the two or more modalities comprises encoding the textual information via a vision encoder.

14. The method of any of claims 1 to 13, wherein projecting the encoded datasets for the two or more modalities into the common embedding space for processing in the large language model comprises projecting each encoded dataset into the common embedding space via a projection function for a respective one of the two or more modalities.

15. A computing system adapted to perform any one or more of the following: receive a point cloud comprising a plurality of data points representing a 3-dimensional, 3D, object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

16. The computing system of claim 15, further adapted to perform the method any of embodiments 2 to 14.

17. A computing system comprising: one or more processors; and memory configured to store instructions executable by the one or more processors whereby the computing system is caused to perform any one or more of the following: receive a point cloud comprising a plurality of data points representing a 3- dimensional, 3D, object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality; encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

18. A computer program comprising instructions which, when executed on at least one processor, cause the processor to carry out the method according to any of claims 1 to 14.

19. A carrier containing the computer program of embodiment 18, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium.

20. A non-transitory computer-readable medium comprising instructions executable by processing circuitry of a computing system, whereby the computing system is operable to: receive a point cloud comprising a plurality of data points representing a 3-dimensional, 3D, object; obtain, for the point cloud, input datasets for two or more modalities, the two or more modalities comprising any two or more of the following modalities: a point cloud modality, a visual modality, and a textual modality;encode the input datasets for the two or more modalities via separate encoders for the two or more modalities, thereby providing encoded datasets for the two or more modalities; project the encoded datasets for the two or more modalities into a common embedding space for processing in a large language model, thereby providing embeddings of the encoded datasets for the two or more modalities; provide the embeddings of the encoded datasets for the two or more modalities to a language model; and obtain, from the large language model, one or more outputs related to a perceptual quality assessment for the point cloud.

Images