A color point cloud JND threshold prediction method, device and equipment and storage medium

Abstract

The application discloses a color point cloud JND threshold prediction method and device, equipment and a storage medium. The method comprises the following steps: extracting two-dimensional features and three-dimensional features of a reference color point cloud and a distorted color point cloud respectively, extracting two-dimensional projection features and color point cloud features through a multi-modal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set; fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multi-modal fusion features, performing perception distortion prediction according to the multi-modal fusion features to obtain a geometric and texture perception distortion prediction label sequence set; and searching the perception distortion prediction label sequence set by using a two-part block JND search strategy to obtain geometric JND threshold and texture JND threshold of a target color point cloud. The application uses a visual perception mechanism to assist in the perception prediction of the color point cloud, further excavates visual perception redundancy, and improves the efficiency of point cloud compression and more efficient visual representation.

Description

Technical Field

[0001] This application relates to the field of point cloud compression technology, and in particular to a method, apparatus, device, and storage medium for predicting the JND threshold of color point clouds. Background Technology

[0002] With the rapid development of multimedia technology and computer graphics, VR (Virtual Reality) has become increasingly popular due to its high level of immersion and six degrees of freedom (6DoF) interaction. Point clouds can accurately and completely reproduce the real world, and are therefore widely used in VR. High compression ratio and low loss 3D point cloud compression methods are key to improving data transmission efficiency. The MPEG international standards organization has developed two point cloud compression standards: G-PCC and V-PCC. G-PCC is more friendly to sparse point cloud encoding. When the number of points in the point cloud is too large, in order to preserve rich texture details, V-PCC borrows from video coding methods, projecting the point cloud separately to obtain texture image frames, geometric image frames, and placeholder image frames, and then encoding and reconstructing the point cloud through video encoding and decoding methods.

[0003] Point cloud quality assessment (PCQA) plays a crucial role in data processing for quality-of-experience (QoE). To reflect visual degradation caused by various distortions, numerous PCQA metrics have been proposed in recent years. However, these metrics often indiscriminately consider all distortions, ignoring the fact that some distortions are below a significant threshold and therefore do not affect subjective perception. To address this issue, Wang et al. incorporated a two-dimensional JND (just noticeable difference) feature into PCQA. First, they repeatedly rotated the reference and distorted point clouds to obtain multiple viewpoints. Then, they used an existing two-dimensional JND model to calculate a three-dimensional JND map, inferring the probability that each point could perceive distortion. Using the generated three-dimensional JND map, they modified the current point-level and structure-level quality metrics to help the point cloud better correlate with subjective visual perception.

[0004] Secondly, video-based point cloud coding not only contains color information but also spatial location information, typically involving a large amount of data. Typical rate-distortion optimization algorithms based on the human visual system only consider color information, which limits coding performance. Zhang et al. proposed a coding tree unit (CTU) quantization level parameter (QP) adjustment algorithm based on JND and spatial complexity to improve the subjective and objective quality of video-based point cloud compression (V-PCC). First, they found that the JND model for attribute videos degrades at the CTU level due to the pixel-filling strategy of V-PCC, and designed an improved JND model using occupancy maps. Second, they designed a spatial complexity detection metric to measure the visual importance of each CTU. Finally, they proposed a CTU-level QP adjustment scheme based on JND and visual importance for geometric and attribute videos. Liu et al. proposed a fast V-PCC coding unit (CU) partitioning algorithm with adaptive CU depth prediction. In video-based point cloud compression (V-PCC), the partitioning of the control unit (CU) has extremely high computational complexity. Juxtaposition of pixel density (JND) prediction methods are effective metrics for guiding this process, but the performance of traditional JND models degrades in V-PCC. For attribute videos, pixel padding operations reduce the brightness perception capability of JND models. For geometric videos, depth-based JND models (JNDD) experience a decrease in depth perception capability in boundary regions due to depth padding operations. Typically, colored point clouds contain a large number of samples, reaching millions of points. To achieve better results, video-based point cloud compression (V-PCC) suffers from significant visual redundancy.

[0005] In summary, existing point cloud compression methods are all based on experiments on point cloud quality assessment datasets and do not directly explore the visual perception characteristics in point cloud compression. The exploration of the visual perception characteristics of 3D point clouds is still far from sufficient. Secondly, the mechanism of 2D visual perception differs from that of 3D visual perception. 3D vision includes an additional depth dimension, making it impossible to directly predict the JND threshold of 3D point clouds using existing 2D image / video JND prediction methods. Furthermore, 3D color point cloud features lack texture and semantic information, and 2D projection features also suffer from occlusion issues. Summary of the Invention

[0006] In view of this, this application provides a method, apparatus, device, and storage medium for predicting the JND threshold of color point clouds, aiming to at least partially solve one of the aforementioned technical problems in the prior art.

[0007] To address the above problems, this application provides the following technical solution:

[0008] A method for predicting the JND threshold of a color point cloud includes:

[0009] Two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud are extracted respectively. The pixel-level JND map is calculated using the reference projection image. The three-dimensional visual mechanism features are calculated based on the original color point cloud. The visual mechanism features and data features of different dimensions are combined to construct a JND training sample set driven by mechanism and data.

[0010] The JND training sample set is input into the multimodal feature extraction module, and the multimodal feature extraction module extracts two-dimensional projection features and color point cloud features respectively to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set.

[0011] The two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain multimodal fusion features. Perceptual distortion prediction is performed based on the multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture.

[0012] A binary block JND search strategy is used to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

[0013] The technical solution adopted in this application embodiment further includes: before extracting the two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, it further includes:

[0014] The reference color point cloud and the distorted color point cloud are projected in two dimensions to obtain projected images from at least six perspectives: top, bottom, left, right, front, and back. The projected images include geometric projection images, texture projection images, and reference projection images.

[0015] The technical solution adopted in this application embodiment further includes: extracting the two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, and constructing a JND training sample set jointly driven by mechanism and data based on the three-dimensional visual mechanism features of the original color point cloud, including:

[0016] Two-dimensional features are extracted from the geometric projection image and the texture projection image, and the pixel-level JND image corresponding to the reference projection image is calculated;

[0017] The reference color point cloud and the distorted color point cloud are subjected to color space conversion and segmentation processing to extract three-dimensional features from the reference color point cloud and the distorted color point cloud.

[0018] Based on the visual mechanism features and data features of different dimensions, a JND training sample set {I,P} jointly driven by mechanism and data is constructed, where I represents the projected image, and I∈{I... R ,I D ,I JND}, I RFor reference projection image, I D For geometric projection images and texture projection images, I JND This is a pixel-level JND map for generating two-dimensional visual mechanisms from a reference projection image; P represents a three-dimensional feature, P∈{P R ,P D ,P V}, where P R For reference, color point cloud, P D For geometrically and texturally distorted color point clouds, P V The three-dimensional visual mechanism features are generated from the original color point cloud, and the three-dimensional visual mechanism features include factors such as brightness, chromaticity, and curvature; when the QP of the distorted color point cloud is greater than or equal to the JND threshold, the JND training sample subset {I i ,P i If} is a negative sample pair, then the JND training sample subset {I} is used. i ,P i} represents positive sample pairs, where i = [1, 2, 3, ..., n], and n is the size of the entire sample space.

[0019] The technical solution adopted in this application embodiment further includes: inputting the JND training sample set into the multimodal feature extraction module, and extracting two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set, specifically:

[0020] The multimodal feature extraction module includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The two-dimensional feature extraction network is used to extract two-dimensional projection features from the geometric projection image, texture projection image, and reference projection image to obtain a two-dimensional projection feature set F1, where F1∈{F 1R ,F 1D ,F JND}, F 1R For the features of the reference projection image, F 1D For features of geometric projection images and texture projection images, F JND Features of the pixel-level JND image; the three-dimensional feature extraction network is used for the three-dimensional features {P R ,P D ,P V} Perform 3D point cloud feature extraction to obtain a 3D point cloud feature set F2, F2∈{F 2R ,F 2D ,F V}, where F 2R To reference the characteristics of colored point clouds, F 2D For the characteristics of geometrically and texturally distorted colored point clouds, F V These are characteristics of the three-dimensional vision mechanism.

[0021] The technical solution adopted in this application embodiment further includes: fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, and performing perceptual distortion prediction based on the multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture, specifically:

[0022] The attention-guided feature fusion module aligns the two-dimensional projection feature set and the three-dimensional point cloud feature set, and then fuses the aligned two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features.

[0023] The multimodal fusion features are input into the trained perceptual distortion discriminator. The perceptual distortion discriminator performs perceptual distortion prediction on the test color point cloud and the corresponding color point cloud set with all compression rates to obtain a set of perceptual distortion prediction label sequences for geometry and texture.

[0024] The technical solution adopted in this application embodiment further includes: performing perceptual distortion prediction on the test color point cloud and the corresponding set of color point clouds with all compression rates through the perceptual distortion discriminator to obtain a set of perceptual distortion prediction label sequences for geometry and texture, specifically:

[0025] Fusing the {F} in the two-dimensional projection feature set F1 1R ,F 1D} and {F in the 3D point cloud feature set F2 2R ,F 2D Predict the overall quality weights FQ of the colored point cloud, and fuse {F} from the two-dimensional projection feature set F1. 1D ,F JND} and {F in the 3D point cloud feature set F2 2D ,F V The distribution weights FW for predicting geometric and texture distortion in distorted color point clouds;

[0026] The quality weight FQ and the distortion distribution weight FW are weighted and the perceived distortion prediction label L is output.

[0027] The technical solution adopted in this application embodiment further includes: the step of using a binary block JND search strategy to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud further includes:

[0028] Within the V-PCC framework, the quantization level parameters QP of the geometric and texture images are controlled based on the geometric JND threshold and texture JND threshold of the target colored point cloud.

[0029] Another technical solution adopted in this application embodiment is: a color point cloud JND threshold prediction device, comprising:

[0030] Preprocessing module: used to extract two-dimensional and three-dimensional features from the reference color point cloud and the distorted color point cloud respectively, calculate pixel-level JND map using the reference projection image, and calculate three-dimensional visual mechanism features based on the original color point cloud, constructing a mechanism and data jointly driven JND training sample set from the visual mechanism features and data features of different dimensions.

[0031] Feature extraction module: Used to input the JND training sample set into the multimodal feature extraction module, and extract two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain two-dimensional projection feature set and three-dimensional point cloud feature set;

[0032] Feature fusion module: used to fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, and perform perceptual distortion prediction based on the multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture;

[0033] JND threshold prediction module: used to search the set of perceptual distortion prediction label sequences using a binary block JND search strategy to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

[0034] Another technical solution adopted in this application embodiment is: a computer device, the computer device including a processor and a memory coupled to the processor, wherein,

[0035] The memory stores program instructions for implementing the color point cloud JND threshold prediction method.

[0036] The processor is used to execute the program instructions stored in the memory to control the color point cloud JND threshold prediction method.

[0037] Another technical solution adopted in this application embodiment is: a storage medium storing processor-executable program instructions, the program instructions being used to execute the color point cloud JND threshold prediction method.

[0038] Compared to existing technologies, the beneficial effects of the embodiments of this application are as follows: The color point cloud JND threshold prediction method, apparatus, device, and storage medium of the embodiments of this application fully explore the three-dimensional visual mechanism features in the color point cloud by combining visual mechanism factors. A self-attention feature fusion module is used to fuse two-dimensional and three-dimensional point cloud features to jointly predict the texture JND threshold and geometric JND threshold of the color point cloud. The texture JND threshold and geometric JND threshold are used to control the quality factor (QP) of the texture and geometric images of the color point cloud, thereby exploring visual perceptual redundancy in the color point cloud and improving the coding performance of the color point cloud. The embodiments of this application utilize visual perceptual mechanisms to assist in the perceptual prediction of color point clouds, compensating for the lack of texture and semantics in three-dimensional point clouds. While further exploring visual perceptual redundancy, the efficiency of point cloud compression is improved. Attached Figure Description

[0039] Figure 1 This is a flowchart of the color point cloud JND threshold prediction method according to an embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the attention-guided feature fusion module in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the structure of the color point cloud JND threshold prediction device according to an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the device structure according to an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the structure of the storage medium according to an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0045] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0046] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0047] Specifically, please refer to Figure 1 This is a flowchart of a color point cloud JND threshold prediction method according to an embodiment of this application. The color point cloud JND threshold prediction method according to an embodiment of this application includes the following steps:

[0048] S100: Obtain the reference color point cloud and the distorted color point cloud;

[0049] In this step, a reference color point cloud and a distorted color point cloud can be obtained based on the JND dataset (PC-JND) of the color point cloud.

[0050] S110: Project the reference color point cloud and the distorted color point cloud in two dimensions to obtain projected images from multiple perspectives;

[0051] In this step, by projecting the reference color point cloud and the distorted color point cloud onto a two-dimensional plane from multiple perspectives, at least six projected images from the top, bottom, left, right, front, and back perspectives are obtained. The projected images include geometric projected images, texture projected images, and reference projected images.

[0052] S120: Extract two-dimensional features from the projected image and extract three-dimensional features from the reference color point cloud and the distorted color point cloud. Calculate the pixel-level JND map using the reference projected image and calculate the three-dimensional visual mechanism features based on the original color point cloud. Construct a mechanism and data jointly driven JND training sample set by combining the visual mechanism features and data features of different dimensions.

[0053] In this step, the two-dimensional features include edge, texture, and color information extracted from geometric projection images and texture projection images, and the pixel-level JND map corresponding to the reference projection image is calculated. Then, the reference color point cloud and the distorted color point cloud undergo color space conversion and segmentation, and the 3D visual mechanism features such as kurtosis, curvature, brightness, and chromaticity in the reference color point cloud and the distorted color point cloud are calculated. This constructs a mechanism- and data-driven JND training sample set {I, P}, where I represents the projection image, I ∈ {I... R ,I D ,I JND}, I R For reference projection image, I D For geometric projection images and texture projection images, I JND This is a pixel-level JND map for generating two-dimensional visual mechanisms from a reference projection image; P represents a three-dimensional feature, P∈{P R ,P D ,P V}, where P R For reference, color point cloud, P D For geometrically and texturally distorted color point clouds, P V The 3D visual mechanism features are generated from the original color point cloud. These features include factors such as brightness, chromaticity, and curvature. When the QP of the distorted color point cloud is greater than or equal to the JND threshold, the JND training sample subset {I}... i ,P i If} is a negative sample pair, then the JND training sample subset {I} is used. i ,P i} represents positive sample pairs, where i = [1, 2, 3, ..., n], and n is the size of the entire sample space.

[0054] S130: Input the JND training sample set into the multimodal feature extraction module, and extract the two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain the two-dimensional projection feature set and the three-dimensional point cloud feature set;

[0055] In this step, the JND training sample set, driven by both mechanism and data, is input into the multimodal feature extraction network. The multimodal feature extraction module includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The two-dimensional feature extraction network is used to extract two-dimensional projection features from the geometric projection image, texture projection image, and reference projection image, obtaining a two-dimensional projection feature set F1, where F1∈{F... 1R ,F 1D ,F JND}, where F 1R For the features of the reference projection image, F 1D For features of geometric projection images and texture projection images, F JND Features of the pixel-level JND image; the three-dimensional feature extraction network is used for the three-dimensional features {P R ,P D ,P V} Perform 3D point cloud feature extraction to obtain a 3D point cloud feature set F2, F2∈{F 2R ,F 2D ,F V}, where F 2R To reference the characteristics of colored point clouds, F 2D For the characteristics of geometrically and texturally distorted colored point clouds, F V These are characteristics of the three-dimensional vision mechanism.

[0056] S140: Align the two-dimensional projection feature set and the three-dimensional point cloud feature set through the attention-guided feature fusion module, and fuse the aligned two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features;

[0057] In this step, based on the perceptual characteristics of human vision for 3D point clouds, and fully utilizing the 3D visual mechanism features in the point cloud, an attention-guided feature fusion module is used to align and fuse the 2D projection feature set F1 and the 3D point cloud feature set F2, jointly predicting the texture JND threshold and geometric JND threshold of the colored point cloud. Specifically, as follows... Figure 2 The diagram shown is a schematic of the attention-guided feature fusion module in an embodiment of this application.

[0058] S150: Input the multimodal fusion features into the trained perceptual distortion discriminator, and use the perceptual distortion discriminator to perform perceptual distortion prediction on the test color point cloud and the corresponding color point cloud set with all compression rates to obtain a set of perceptual distortion prediction label sequences for geometry and texture.

[0059] In this step, by fusing {F} in F1 1R ,F 1D} and {F in F2 2R ,F 2D Predict the overall quality weights FQ of the colored point cloud, and fuse {F} from F1.1D ,F 1JND} and {F in F2 2D ,F V The distribution weights FW for geometric and texture distortion of the predicted distorted color point cloud are calculated, and finally the quality weight FQ and the distortion distribution weight FW are weighted to output the perceptual distortion prediction label L.

[0060] S160: A binary block JND search strategy is used to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud. The quantization level parameters of the geometric image and texture image in the V-PCC framework are controlled according to the geometric JND threshold and texture JND threshold.

[0061] In this step, a binary block JND threshold search strategy is used to find the perceptual distortion prediction label sequence, thereby obtaining the geometric JND threshold and texture JND threshold of the target color point cloud. Under the V-PCC framework, the quantization level parameters of its geometric and texture images are controlled according to the geometric and texture JND thresholds of the target color point cloud to mine the visual perceptual redundancy in the color point cloud, control the encoding process of geometric and texture image frames in V-PCC, and improve the encoding performance of the color point cloud.

[0062] Based on the above, the JND threshold prediction method for color point clouds in this application fully mines the three-dimensional visual mechanism features in color point clouds by combining visual mechanism factors. It employs a self-attention feature fusion module to fuse two-dimensional and three-dimensional point cloud features, jointly predicting the texture JND threshold and geometric JND threshold of the color point cloud. These thresholds are then used to control the quality factor (QP) of the texture and geometric images of the color point cloud, thereby mining visual perceptual redundancy in the color point cloud and improving its encoding performance. This application utilizes visual perceptual mechanisms to assist in the perceptual prediction of color point clouds, compensating for the lack of texture and semantics in three-dimensional point clouds. While further mining visual perceptual redundancy, it also improves the efficiency of point cloud compression.

[0063] Please see Figure 3 This is a schematic diagram of the structure of a color point cloud JND threshold prediction device according to an embodiment of this application. The color point cloud JND threshold prediction device 40 according to an embodiment of this application includes:

[0064] Preprocessing module 41: used to extract two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, calculate pixel-level JND map using the reference projection image, and calculate three-dimensional visual mechanism features based on the original color point cloud, constructing a mechanism and data jointly driven JND training sample set by combining visual mechanism features and data features of different dimensions.

[0065] Feature extraction module 42: used to input the JND training sample set into the multimodal feature extraction module, and extract two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain two-dimensional projection feature set and three-dimensional point cloud feature set;

[0066] Feature fusion module 43: used to fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, and perform perceptual distortion prediction based on the multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture;

[0067] JND threshold prediction module 44: Used to search the set of perceptual distortion prediction label sequences using a binary block JND search strategy to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

[0068] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0069] The apparatus provided in this application embodiment can be applied in the foregoing method embodiment. For details, please refer to the description of the above method embodiment, which will not be repeated here.

[0070] Please see Figure 4 This is a schematic diagram of the device structure according to an embodiment of this application. The device 50 includes:

[0071] Memory 51 storing executable program instructions;

[0072] Processor 52 connected to memory 51;

[0073] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: extract the two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively; calculate the pixel-level JND map using the reference projection image; and construct a mechanism- and data-driven JND training sample set based on the three-dimensional visual mechanism features of the original color point cloud. The JND training sample set is input into the multimodal feature extraction module, and the two-dimensional projection features and color point cloud features are extracted by the multimodal feature extraction module to obtain the two-dimensional projection feature set and the three-dimensional point cloud feature set. The two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain the multimodal fusion features. Perceptual distortion prediction is performed based on the multimodal fusion features to obtain the geometric and texture perceptual distortion prediction label sequence set. The perceptual distortion prediction label sequence set is searched using a binary block JND search strategy to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

[0074] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0075] Please see Figure 5 This is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores program instructions 61 capable of implementing the following steps: extracting two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively; calculating pixel-level JND maps using the reference projection image; and constructing a mechanism- and data-driven JND training sample set by combining the visual mechanism features and data features of different dimensions; inputting the JND training sample set into a multimodal feature extraction module; extracting two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set; fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain a multimodal fusion feature; performing perceptual distortion prediction based on the multimodal fusion feature to obtain a set of geometric and texture perceptual distortion prediction label sequences; and using a binary block JND search strategy to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

[0076] The program instructions 61 can be stored in the aforementioned storage medium in the form of a software product, including several instructions to cause a device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program instructions, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for predicting the JND threshold of a color point cloud, characterized in that, include: Two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud are extracted respectively. The pixel-level JND map is calculated using the reference projection image. The three-dimensional visual mechanism features are calculated based on the original color point cloud. The visual mechanism features and data features of different dimensions are combined to construct a JND training sample set driven by mechanism and data. The JND training sample set is input into the multimodal feature extraction module, and the multimodal feature extraction module extracts two-dimensional projection features and color point cloud features respectively to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set. The two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain multimodal fusion features. Perceptual distortion prediction is performed based on the multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture. A binary block JND search strategy is used to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

2. The color point cloud JND threshold prediction method according to claim 1, characterized in that, Before extracting the two-dimensional and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, the process also includes: The reference color point cloud and the distorted color point cloud are projected in two dimensions to obtain projected images from at least six perspectives: top, bottom, left, right, front, and back. The projected images include geometric projection images, texture projection images, and reference projection images.

3. The color point cloud JND threshold prediction method according to claim 2, characterized in that, The process involves extracting two-dimensional and three-dimensional features from the reference color point cloud and the distorted color point cloud, respectively, and then constructing a mechanism- and data-driven JND training sample set based on the three-dimensional visual mechanism features from the original color point cloud. This includes: Two-dimensional features are extracted from the geometric projection image and the texture projection image, and the pixel-level JND image corresponding to the reference projection image is calculated; The reference color point cloud and the distorted color point cloud are subjected to color space conversion and segmentation processing to extract three-dimensional features from the reference color point cloud and the distorted color point cloud. Based on the visual mechanism features and data features of different dimensions, a JND training sample set {I,P} jointly driven by mechanism and data is constructed, where I represents the projected image, and I∈{I... R ,I D ,I JND }, I R For reference projection image, I D For geometric projection images and texture projection images, I JND This is a pixel-level JND map for generating two-dimensional visual mechanisms from a reference projection image; P represents a three-dimensional feature, P∈{P R ,P D ,P V }, where P R For reference, color point cloud, P D For geometrically and texturally distorted color point clouds, P V The three-dimensional visual mechanism features are generated from the original color point cloud, and the three-dimensional visual mechanism features include factors such as brightness, chromaticity, and curvature; when the QP of the distorted color point cloud is greater than or equal to the JND threshold, the JND training sample subset {I i ,P i If} is a negative sample pair, then the JND training sample subset {I} is used. i ,P i } represents positive sample pairs, where i = [1, 2, 3, ..., n], and n is the size of the entire sample space.

4. The color point cloud JND threshold prediction method according to claim 3, characterized in that, The process involves inputting the JND training sample set into a multimodal feature extraction module, which extracts two-dimensional projection features and color point cloud features to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set. Specifically: The multimodal feature extraction module includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The two-dimensional feature extraction network is used to extract two-dimensional projection features from the geometric projection image, texture projection image, and reference projection image to obtain a two-dimensional projection feature set F1, where F1∈{F 1R ,F 1D ,F JND }, where F 1R For the features of the reference projection image, F 1D For features of geometric projection images and texture projection images, F JND Features of the pixel-level JND image; the three-dimensional feature extraction network is used for the three-dimensional features {P R ,P D ,P V } Perform 3D point cloud feature extraction to obtain a 3D point cloud feature set F2, F2∈{F 2R ,F 2D ,F V }, where F 2R To reference the characteristics of colored point clouds, F 2D For the characteristics of geometrically and texturally distorted colored point clouds, F V These are characteristics of the three-dimensional vision mechanism.

5. The color point cloud JND threshold prediction method according to claim 4, characterized in that, The process involves fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features. Perceptual distortion prediction is then performed based on these multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture. Specifically: The attention-guided feature fusion module aligns the two-dimensional projection feature set and the three-dimensional point cloud feature set, and then fuses the aligned two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features. The multimodal fusion features are input into the trained perceptual distortion discriminator. The perceptual distortion discriminator performs perceptual distortion prediction on the test color point cloud and the corresponding color point cloud set with all compression rates to obtain a set of perceptual distortion prediction label sequences for geometry and texture.

6. The color point cloud JND threshold prediction method according to claim 5, characterized in that, The process of performing perceptual distortion prediction on the test color point cloud and the corresponding set of color point clouds with all compression rates using the perceptual distortion discriminator to obtain a set of perceptual distortion prediction label sequences for geometry and texture is as follows: Fusing the {F} in the two-dimensional projection feature set F1 1R ,F 1D } and {F in the 3D point cloud feature set F2 2R ,F 2D Predict the overall quality weights FQ of the colored point cloud, and fuse {F} from the two-dimensional projection feature set F1. 1D ,F JND } and {F in the 3D point cloud feature set F2 2D ,F V The distribution weights FW for predicting geometric and texture distortion in distorted color point clouds; The quality weight FQ and the distortion distribution weight FW are weighted and the perceived distortion prediction label L is output.

7. The color point cloud JND threshold prediction method according to any one of claims 1 to 6, characterized in that, The step of using a binary block JND search strategy to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud further includes: Within the V-PCC framework, the quantization level parameters QP of the geometric and texture images are controlled based on the geometric JND threshold and texture JND threshold of the target colored point cloud.

8. A color point cloud JND threshold prediction device, characterized in that, include: Preprocessing module: used to extract two-dimensional and three-dimensional features from the reference color point cloud and the distorted color point cloud respectively, calculate pixel-level JND map using the reference projection image, and calculate three-dimensional visual mechanism features based on the original color point cloud, constructing a mechanism and data jointly driven JND training sample set from the visual mechanism features and data features of different dimensions. Feature extraction module: Used to input the JND training sample set into the multimodal feature extraction module, and extract two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain two-dimensional projection feature set and three-dimensional point cloud feature set; Feature fusion module: used to fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, and perform perceptual distortion prediction based on the multimodal fusion features to obtain a set of perceptual distortion prediction label sequences for geometry and texture; JND threshold prediction module: used to search the set of perceptual distortion prediction label sequences using a binary block JND search strategy to obtain the geometric JND threshold and texture JND threshold of the target color point cloud.

9. A device, characterized in that, The device includes a processor and a memory coupled to the processor, wherein, The memory stores program instructions for implementing the color point cloud JND threshold prediction method according to any one of claims 1-7; The processor is used to execute the program instructions stored in the memory to control the machine to perform the color point cloud JND threshold prediction method.

10. A storage medium, characterized in that, The device stores processor-executable program instructions for performing the color point cloud JND threshold prediction method according to any one of claims 1 to 7.

Images