Color point cloud JND threshold prediction method and device, equipment and storage medium

Abstract

The invention discloses a color point cloud JND threshold prediction method, device and equipment and a storage medium, and the method comprises the steps: extracting two-dimensional features and three-dimensional features of a reference color point cloud and a distorted color point cloud respectively, and extracting two-dimensional projection features and color point cloud features through a multi-modal feature extraction module, obtaining a two-dimensional projection feature set and a three-dimensional point cloud feature set; fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain a multi-modal fusion feature, and performing perceptual distortion prediction according to the multi-modal fusion feature to obtain a geometric and texture perceptual distortion prediction tag sequence set; and searching the perceptual distortion prediction label sequence set by adopting a binary block JND search strategy to obtain a geometric JND threshold value and a texture JND threshold value of the target color point cloud. The visual perception mechanism is used for assisting perception prediction of the color point cloud, visual perception redundancy is further mined, and meanwhile, the point cloud compression efficiency and more efficient visual representation are improved.

Description

Technical Field

[0001] The present application relates to the field of point cloud compression technology, and in particular to a method, device, equipment and storage medium for predicting a JND threshold of a color point cloud. Background Art

[0002] With the rapid development of multimedia technology and computer graphics, VR (Virtual Reality) has become increasingly popular due to its high level of immersion and six degrees of freedom (6DoF) interaction. Point clouds can accurately and completely reproduce the real world, so they are widely used in VR. High compression ratio and low loss three-dimensional point cloud compression methods are the key to improving data transmission efficiency. The International Organization for Standardization (MPEG) has developed two point cloud compression standards, G-PCC and V-PCC. G-PCC is more friendly to sparse point cloud coding. When the number of points in the point cloud is too large, in order to retain rich texture detail information, V-PCC draws on the video coding method, projects the point cloud to obtain texture image frames, geometric image frames and placeholder image frames, and encodes and reconstructs the point cloud through video coding and decoding.

[0003] Point cloud quality assessment (PCQA) plays a vital role in quality of experience (QoE) oriented data processing. Many PCQA metrics have been proposed in recent years to reflect the visual degradation caused by various distortions. However, these metrics often consider all distortions indiscriminately, ignoring the fact that some distortions are below a noticeable threshold and therefore do not affect subjective perception. To address this issue, Wang et al. added a 2D JND (just noticeable difference) feature to PCQA. First, the reference and distorted point clouds are repeatedly rotated to obtain multiple viewpoints, and then a 3D JND map is calculated using the existing 2D JND model to infer the probability that each point can perceive the distortion. Using the generated 3D JND map, the current point-level and structure-level quality metrics are modified to help point clouds better correlate with subjective visual perception.

[0004] Secondly, video-based point cloud coding contains not only color information but also spatial location information, which usually has a large amount of data. Typical rate-distortion optimization algorithms based on the human visual system only consider color information, which limits the coding performance. Zhang et al. proposed a coding tree unit (CTU) quantization level parameter (QP) adjustment algorithm based on JND and spatial complexity to improve the subjective and objective quality of video-based point cloud compression (V-PCC). First, it is found that the JND model of attribute video is degraded at the CTU level due to the pixel padding strategy of V-PCC, and an improved JND model is designed using occupancy maps. Secondly, a spatial complexity detection metric is designed to measure the visual importance of each CTU. Finally, a CTU-level QP adjustment scheme for geometric and attribute videos based on JND and visual importance is proposed. Liu et al. proposed a fast V-PCC coding unit (CU) partitioning algorithm with adaptive CU depth prediction. In video-based point cloud compression (V-PCC), the division of CU has extremely high computational complexity. The JND prediction method is an effective metric to guide this process, while the performance of the traditional JND model in V-PCC has declined. For attribute videos, the brightness perception ability of the JND model is reduced due to the pixel filling operation. For geometric videos, the depth perception ability of the depth-based JND model (JNDD) in the boundary area is reduced due to the depth filling operation. Generally, the number of samples of the point cloud with color is large, which can reach millions of points. In order to obtain better results, the video-based point cloud compression technology (V-PCC) has a lot of visual redundancy.

[0005] In summary, existing point cloud compression methods are all based on experiments on point cloud quality evaluation datasets, and do not directly explore the visual perception characteristics of point cloud compression. The mining of the visual perception characteristics of three-dimensional point clouds is far from enough. Secondly, the two-dimensional visual perception mechanism is different from the three-dimensional visual perception mechanism. In three-dimensional vision, there is an additional dimension of depth information, and it is impossible to directly use the existing two-dimensional image / video JND prediction method to directly predict the JND threshold of three-dimensional point clouds. In addition, the three-dimensional color point cloud features lack texture, semantics and other information, and the two-dimensional projection features also have occlusion problems. Summary of the invention

[0006] In view of this, the present application provides a color point cloud JND threshold prediction method, device, equipment and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0007] In order to solve the above problems, this application provides the following technical solutions:

[0008] A color point cloud JND threshold prediction method, comprising:

[0009] The 2D and 3D features of the reference color point cloud and the distorted color point cloud are extracted respectively, and the pixel-level JND map is calculated using the reference projection image. Based on the 3D visual mechanism features of the original color point cloud, the visual mechanism features and data features of different dimensions are combined to construct a JND training sample set driven by the mechanism and data.

[0010] Inputting the JND training sample set into a multimodal feature extraction module, respectively extracting two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set;

[0011] The two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain a multimodal fusion feature, and perceptual distortion prediction is performed according to the multimodal fusion feature to obtain a set of geometric and texture perceptual distortion prediction label sequences;

[0012] A binary JND search strategy is used to search the perceptual distortion prediction label sequence set to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

[0013] The technical solution adopted by the embodiment of the present application also includes: before respectively extracting the two-dimensional features and the three-dimensional features of the reference color point cloud and the distorted color point cloud, it also includes:

[0014] The reference color point cloud and the distorted color point cloud are two-dimensionally projected to obtain projection images of at least six viewing angles, namely, top, bottom, left, right, front, and back; wherein the projection images include geometric projection images, texture projection images, and reference projection images.

[0015] The technical solution adopted by the embodiment of the present application also includes: extracting the two-dimensional features and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, and constructing a JND training sample set driven by the mechanism and data jointly based on the three-dimensional visual mechanism features of the original color point cloud, using the visual mechanism features and data features of different dimensions, including:

[0016] Extracting two-dimensional features from the geometric projection image and the texture projection image, and calculating a pixel-level JND map corresponding to the reference projection image;

[0017] Performing color space conversion and slicing processing on the reference color point cloud and the distorted color point cloud, and extracting three-dimensional features from the reference color point cloud and the distorted color point cloud;

[0018] According to the visual mechanism features and data features of different dimensions, a JND training sample set {I,P} jointly driven by mechanism and data is constructed, where I represents the projected image, I∈{I R ,I D ,I JND}, I Ris the reference projection image, I D are the geometric projection image and texture projection image, I JND is the pixel-level JND map of the two-dimensional visual mechanism generated by the reference projection image; P represents the three-dimensional feature, P∈{P R ,P D ,P V}, where P R is the reference color point cloud, P D Color point cloud with geometric and texture distortion, P V is a 3D visual mechanism feature generated from the original color point cloud, wherein the 3D visual mechanism feature includes factors such as brightness, chromaticity and curvature; when the QP of the distorted color point cloud is greater than or equal to the JND threshold, the JND training sample subset {I i ,P i} is a negative sample pair, otherwise, the JND training sample subset {I i ,P i} is a positive sample pair, where i = [1, 2, 3....n], and n is the size of the entire sample space.

[0019] The technical solution adopted by the embodiment of the present application also includes: the JND training sample set is input into the multimodal feature extraction module, and the two-dimensional projection features and the color point cloud features are respectively extracted by the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set, specifically:

[0020] The multimodal feature extraction module includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The two-dimensional feature extraction network is used to extract two-dimensional projection features from the geometric projection image, the texture projection image and the reference projection image to obtain a two-dimensional projection feature set F1, F1∈{F 1R ,F 1D ,F JND}, F 1R is the feature of the reference projection image, F 1D is the feature of geometric projection image and texture projection image, F JND is the feature of the pixel-level JND map; the three-dimensional feature extraction network is used to extract the three-dimensional feature {P R ,P D ,P V} to extract the three-dimensional point cloud features and obtain the three-dimensional point cloud feature set F2, F2∈{F 2R ,F 2D ,F V}, where F 2R is the feature of the reference color point cloud, F 2D is the feature of the color point cloud with geometric and texture distortion, F V It is a characteristic of the three-dimensional vision mechanism.

[0021] The technical solution adopted by the embodiment of the present application also includes: fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain a multimodal fusion feature, performing perceptual distortion prediction based on the multimodal fusion feature, and obtaining a set of geometric and texture perceptual distortion prediction label sequences, specifically:

[0022] The two-dimensional projection feature set and the three-dimensional point cloud feature set are aligned by an attention-guided feature fusion module, and the aligned two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain a multimodal fusion feature;

[0023] The multimodal fusion features are input into a trained perceptual distortion discriminator, and the perceptual distortion discriminator is used to perform perceptual distortion prediction on a test color point cloud and a corresponding color point cloud set of all compression rates to obtain a set of perceptual distortion prediction label sequences of geometry and texture.

[0024] The technical solution adopted by the embodiment of the present application also includes: performing perceptual distortion prediction on the test color point cloud and the corresponding color point cloud set of all compression rates through the perceptual distortion discriminator to obtain a set of geometric and texture perceptual distortion prediction label sequences, specifically:

[0025] Fusion of the two-dimensional projection feature set F1 {F 1R ,F 1D} and {F 2R ,F 2D} predict the overall quality weight FQ of the color point cloud, integrating {F 1D ,F JND} and {F 2D ,F V}Predict the distribution weights FW of geometric distortion and texture distortion of distorted color point cloud;

[0026] The quality weight FQ and the distortion distribution weight FW are weighted, and a perceptual distortion prediction label L is output.

[0027] The technical solution adopted by the embodiment of the present application also includes: the binary block JND search strategy is used to search the set of perceptual distortion prediction label sequences to obtain the geometric JND threshold and texture JND threshold of the target color point cloud, and further includes:

[0028] In the V-PCC framework, the quantization level parameter QP of the geometric image and the texture image is controlled according to the geometric JND threshold and the texture JND threshold of the target color point cloud.

[0029] Another technical solution adopted by the embodiment of the present application is: a color point cloud JND threshold prediction device, comprising:

[0030] Preprocessing module: used to extract the 2D and 3D features of the reference color point cloud and the distorted color point cloud respectively, calculate the pixel-level JND map using the reference projection image, and calculate the 3D visual mechanism features of the original color point cloud, and construct the JND training sample set driven by the mechanism and data jointly by the visual mechanism features and data features of different dimensions;

[0031] Feature extraction module: used for inputting the JND training sample set into the multimodal feature extraction module, and extracting two-dimensional projection features and color point cloud features respectively through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set;

[0032] Feature fusion module: used to fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, perform perceptual distortion prediction based on the multimodal fusion features, and obtain a set of geometric and texture perceptual distortion prediction label sequences;

[0033] The JND threshold prediction module is used to search the perceptual distortion prediction label sequence set by adopting a binary block JND search strategy to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

[0034] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0035] The memory stores program instructions for implementing the color point cloud JND threshold prediction method;

[0036] The processor is used to execute the program instructions stored in the memory to control the color point cloud JND threshold prediction method.

[0037] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the color point cloud JND threshold prediction method.

[0038] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the color point cloud JND threshold prediction method, device, equipment and storage medium of the embodiments of the present application fully exploit the three-dimensional visual mechanism features in the color point cloud by combining visual mechanism factors, and use the self-attention feature fusion module to fuse the two-dimensional point cloud features and the three-dimensional point cloud features, and jointly predict the texture JND threshold and geometric JND threshold of the color point cloud, and use the texture JND threshold and geometric JND threshold to control the quality factor (QP) of the texture image and geometric image of the color point cloud to exploit the visual perception redundancy in the color point cloud and improve the encoding performance of the color point cloud. The embodiments of the present application use the visual perception mechanism to assist the perceptual prediction of the color point cloud, which makes up for the lack of texture and semantics of the three-dimensional point cloud, and improves the efficiency of point cloud compression while further exploiting the visual perception redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of a color point cloud JND threshold prediction method according to an embodiment of the present application;

[0040] Figure 2 This is a schematic diagram of an attention-guided feature fusion module according to an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the structure of a color point cloud JND threshold prediction device according to an embodiment of the present application;

[0042] Figure 4 This is a schematic diagram of the device structure of an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0046] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0047] Specifically, see Figure 1 , is a flow chart of the color point cloud JND threshold prediction method of the embodiment of the present application. The color point cloud JND threshold prediction method of the embodiment of the present application comprises the following steps:

[0048] S100: Acquire a reference color point cloud and a distorted color point cloud;

[0049] In this step, a reference color point cloud and a distorted color point cloud may be obtained based on a JND dataset of color point clouds (PC-JND).

[0050] S110: performing two-dimensional projection on the reference color point cloud and the distorted color point cloud to obtain projection images from multiple perspectives;

[0051] In this step, by projecting the reference color point cloud and the distorted color point cloud onto a two-dimensional plane from multiple perspectives, projection images of at least six perspectives, namely, top, bottom, left, right, front, and back, are obtained, wherein the projection images include a geometric projection image, a texture projection image, and a reference projection image.

[0052] S120: extracting two-dimensional features from the projection image, and extracting three-dimensional features from the reference color point cloud and the distorted color point cloud, using the reference projection image to calculate the pixel-level JND map, and calculating the three-dimensional visual mechanism features based on the original color point cloud, and constructing a JND training sample set jointly driven by the mechanism and data by combining the visual mechanism features and data features of different dimensions;

[0053] In this step, the two-dimensional features include the edge, texture, color information and other features extracted from the geometric projection image and the texture projection image, and the pixel-level JND map corresponding to the reference projection image is calculated. Then, the reference color point cloud and the distorted color point cloud are subjected to color space conversion and segmentation, and the three-dimensional visual mechanism features such as kurtosis, curvature, brightness and chromaticity in the reference color point cloud and the distorted color point cloud are calculated to construct the JND training sample set {I,P} driven by the mechanism and data, where I represents the projection image, I∈{I R ,I D ,I JND}, I R is the reference projection image, I D are the geometric projection image and texture projection image, I JND is the pixel-level JND map of the two-dimensional visual mechanism generated by the reference projection image; P represents the three-dimensional feature, P∈{P R ,P D ,P V}, where P R is the reference color point cloud, P D Color point cloud with geometric and texture distortion, P V is the 3D visual mechanism feature generated by the original color point cloud. The 3D visual mechanism feature includes factors such as brightness, chromaticity and curvature. When the QP of the distorted color point cloud is greater than or equal to the JND threshold, the JND training sample subset {I i ,P i} is a negative sample pair, otherwise, the JND training sample subset {I i ,P i} is a positive sample pair, where i = [1, 2, 3....n], and n is the size of the entire sample space.

[0054] S130: inputting the JND training sample set into the multimodal feature extraction module, extracting two-dimensional projection features and color point cloud features respectively through the multimodal feature extraction module, and obtaining a two-dimensional projection feature set and a three-dimensional point cloud feature set;

[0055] In this step, the JND training sample set jointly driven by the mechanism and data is input into the multimodal feature extraction network. The multimodal feature extraction module includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The two-dimensional feature extraction network is used to extract two-dimensional projection features of the geometric projection image, the texture projection image and the reference projection image to obtain a two-dimensional projection feature set F1, F1∈{F 1R ,F 1D ,F JND}, where F 1R is the feature of the reference projection image, F 1D is the feature of geometric projection image and texture projection image, F JND is the feature of the pixel-level JND map; the three-dimensional feature extraction network is used to extract the three-dimensional feature {P R ,P D ,P V} to extract the three-dimensional point cloud features and obtain the three-dimensional point cloud feature set F2, F2∈{F 2R ,F 2D ,F V}, where F 2R is the feature of the reference color point cloud, F 2D is the feature of the color point cloud with geometric and texture distortion, F V It is a characteristic of the three-dimensional vision mechanism.

[0056] S140: aligning the two-dimensional projection feature set and the three-dimensional point cloud feature set through an attention-guided feature fusion module, and fusing the aligned two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain a multimodal fusion feature;

[0057] In this step, according to the human visual perception characteristics of 3D point clouds, the 3D visual mechanism characteristics in the point clouds are fully utilized, and the 2D projection feature set F1 and the 3D point cloud feature set F2 are aligned and fused using the attention-guided feature fusion module to jointly predict the texture JND threshold and geometric JND threshold of the color point cloud. Figure 2 As shown, it is a schematic diagram of the attention-guided feature fusion module of an embodiment of the present application.

[0058] S150: input the multimodal fusion features into the trained perceptual distortion discriminator, and perform perceptual distortion prediction on the test color point cloud and the corresponding color point cloud set of all compression rates through the perceptual distortion discriminator to obtain a set of geometric and texture perceptual distortion prediction label sequences;

[0059] In this step, by integrating {F 1R ,F 1D} and {F in F2 2R ,F 2D} predict the overall quality weight FQ of the color point cloud, integrating {F1D ,F 1JND} and {F in F2 2D ,F V}Predict the distribution weights FW of the geometric distortion and texture distortion of the distorted color point cloud, and finally weight the quality weight FQ and the distortion distribution weight FW to output the perceptual distortion prediction label L.

[0060] S160: using a binary JND search strategy to search for a set of perceptual distortion prediction label sequences, obtaining a geometric JND threshold and a texture JND threshold of the target color point cloud, and controlling quantization level parameters of the geometric image and the texture image in the V-PCC framework according to the geometric JND threshold and the texture JND threshold;

[0061] In this step, a binary JND threshold search strategy is used to find the perceptual distortion prediction label sequence to obtain the geometric JND threshold and texture JND threshold of the target color point cloud. Under the V-PCC framework, the quantization level parameters of the geometric image and texture image are controlled according to the geometric JND threshold and texture JND threshold of the target color point cloud to mine the visual perceptual redundancy in the color point cloud, control the encoding process of the geometric image frame and texture image frame in V-PCC, and improve the encoding performance of the color point cloud.

[0062] Based on the above, the color point cloud JND threshold prediction method of the embodiment of the present application fully exploits the three-dimensional visual mechanism characteristics in the color point cloud by combining visual mechanism factors, and uses the self-attention feature fusion module to fuse the two-dimensional point cloud features and the three-dimensional point cloud features, and jointly predicts the texture JND threshold and geometric JND threshold of the color point cloud. The texture JND threshold and geometric JND threshold are used to control the quality factor QP of the texture image and geometric image of the color point cloud to exploit the visual perception redundancy in the color point cloud and improve the encoding performance of the color point cloud. The embodiment of the present application uses the visual perception mechanism to assist the perceptual prediction of the color point cloud, making up for the lack of texture and semantics of the three-dimensional point cloud, and while further exploiting the visual perception redundancy, it improves the efficiency of point cloud compression.

[0063] See also Figure 3 , is a schematic diagram of the structure of a color point cloud JND threshold prediction device according to an embodiment of the present application. The color point cloud JND threshold prediction device 40 according to an embodiment of the present application comprises:

[0064] Preprocessing module 41: used to extract the two-dimensional features and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, calculate the pixel-level JND map using the reference projection image, and calculate the three-dimensional visual mechanism features of the original color point cloud, and construct the JND training sample set driven by the mechanism and data jointly by the visual mechanism features and data features of different dimensions;

[0065] Feature extraction module 42: used for inputting the JND training sample set into the multimodal feature extraction module, and extracting the two-dimensional projection features and the color point cloud features respectively through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set;

[0066] Feature fusion module 43: used to fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, perform perceptual distortion prediction based on the multimodal fusion features, and obtain a set of geometric and texture perceptual distortion prediction label sequences;

[0067] The JND threshold prediction module 44 is used to search the perceptual distortion prediction label sequence set by adopting a binary block JND search strategy to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

[0068] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0069] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0070] See also Figure 4 , is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

[0071] A memory 51 storing executable program instructions;

[0072] A processor 52 connected to the memory 51;

[0073] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: respectively extract the two-dimensional features and the three-dimensional features of the reference color point cloud and the distorted color point cloud, use the reference projection image to calculate the pixel-level JND map, and according to the original color point cloud three-dimensional visual mechanism features, construct the visual mechanism features and data features of different dimensions into a JND training sample set jointly driven by the mechanism and data; input the JND training sample set into the multimodal feature extraction module, and respectively extract the two-dimensional projection features and the color point cloud features through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set; fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain a multimodal fusion feature, perform perceptual distortion prediction based on the multimodal fusion feature, and obtain a set of geometric and texture perceptual distortion prediction label sequences; use a binary block JND search strategy to search the perceptual distortion prediction label sequence set to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

[0074] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0075] See also Figure 5 , which is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program instruction 61 capable of implementing the following steps: extracting two-dimensional features and three-dimensional features of the reference color point cloud and the distorted color point cloud respectively, using the reference projection image to calculate the pixel-level JND map, and according to the original color point cloud three-dimensional visual mechanism features, visual mechanism features and data features of different dimensions are constructed into a JND training sample set jointly driven by mechanism and data; inputting the JND training sample set into the multimodal feature extraction module, extracting two-dimensional projection features and color point cloud features respectively through the multimodal feature extraction module, and obtaining a two-dimensional projection feature set and a three-dimensional point cloud feature set; fusing the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain a multimodal fusion feature, performing perceptual distortion prediction based on the multimodal fusion feature, and obtaining a set of geometric and texture perceptual distortion prediction label sequences; using a binary block JND search strategy to search the perceptual distortion prediction label sequence set to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

[0076] Among them, the program instruction 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions to enable a device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program instructions, or terminal devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0077] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0078] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A color point cloud JND threshold prediction method, characterized in that: include: The 2D and 3D features of the reference color point cloud and the distorted color point cloud are extracted respectively, and the pixel-level JND map is calculated using the reference projection image. Based on the 3D visual mechanism features of the original color point cloud, the visual mechanism features and data features of different dimensions are combined to construct a JND training sample set driven by the mechanism and data. Inputting the JND training sample set into a multimodal feature extraction module, respectively extracting two-dimensional projection features and color point cloud features through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set; The two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain a multimodal fusion feature, and perceptual distortion prediction is performed according to the multimodal fusion feature to obtain a set of geometric and texture perceptual distortion prediction label sequences; A binary JND search strategy is used to search the perceptual distortion prediction label sequence set to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

2. The color point cloud JND threshold prediction method according to claim 1, characterized in that: Before respectively extracting the two-dimensional features and the three-dimensional features of the reference color point cloud and the distorted color point cloud, the method further includes: The reference color point cloud and the distorted color point cloud are two-dimensionally projected to obtain projection images of at least six viewing angles, namely, top, bottom, left, right, front, and back; wherein the projection images include geometric projection images, texture projection images, and reference projection images.

3. The color point cloud JND threshold prediction method according to claim 2, characterized in that: The two-dimensional features and three-dimensional features of the reference color point cloud and the distorted color point cloud are extracted respectively, and according to the three-dimensional visual mechanism features of the original color point cloud, the visual mechanism features and data features of different dimensions are combined to construct a JND training sample set driven by the mechanism and data, including: Extracting two-dimensional features from the geometric projection image and the texture projection image, and calculating a pixel-level JND map corresponding to the reference projection image; Performing color space conversion and slicing processing on the reference color point cloud and the distorted color point cloud, and extracting three-dimensional features from the reference color point cloud and the distorted color point cloud; According to the visual mechanism features and data features of different dimensions, a JND training sample set {I,P} jointly driven by mechanism and data is constructed, where I represents the projected image, I∈{I R ,I D ,I JND }, I R is the reference projection image, I D are the geometric projection image and texture projection image, I JND is the pixel-level JND map of the two-dimensional visual mechanism generated by the reference projection image; P represents the three-dimensional feature, P∈{P R ,P D ,P V }, where P R is the reference color point cloud, P D Color point cloud with geometric and texture distortion, P V is a 3D visual mechanism feature generated from the original color point cloud, wherein the 3D visual mechanism feature includes factors such as brightness, chromaticity and curvature; when the QP of the distorted color point cloud is greater than or equal to the JND threshold, the JND training sample subset {I i ,P i } is a negative sample pair, otherwise, the JND training sample subset {I i ,P i } is a positive sample pair, where i = [1, 2, 3....n], and n is the size of the entire sample space.

4. The color point cloud JND threshold prediction method according to claim 3, characterized in that: The JND training sample set is input into the multimodal feature extraction module, and the two-dimensional projection features and the color point cloud features are respectively extracted by the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set, specifically: The multimodal feature extraction module includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The two-dimensional feature extraction network is used to extract two-dimensional projection features from the geometric projection image, the texture projection image and the reference projection image to obtain a two-dimensional projection feature set F1, F1∈{F 1R ,F 1D ,F JND }, where F 1R is the feature of the reference projection image, F 1D is the feature of geometric projection image and texture projection image, F JND is the feature of the pixel-level JND map; the three-dimensional feature extraction network is used to extract the three-dimensional feature {P R ,P D ,P V } to extract the three-dimensional point cloud features and obtain the three-dimensional point cloud feature set F2, F2∈{F 2R ,F 2D ,F V }, where F 2R is the feature of the reference color point cloud, F 2D is the feature of the color point cloud with geometric and texture distortion, F V It is a characteristic of the three-dimensional vision mechanism.

5. The color point cloud JND threshold prediction method according to claim 4, characterized in that: The two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain a multimodal fusion feature, and perceptual distortion prediction is performed according to the multimodal fusion feature to obtain a set of geometric and texture perceptual distortion prediction label sequences, specifically: The two-dimensional projection feature set and the three-dimensional point cloud feature set are aligned by an attention-guided feature fusion module, and the aligned two-dimensional projection feature set and the three-dimensional point cloud feature set are fused to obtain a multimodal fusion feature; The multimodal fusion features are input into a trained perceptual distortion discriminator, and the perceptual distortion discriminator is used to perform perceptual distortion prediction on a test color point cloud and a corresponding color point cloud set of all compression rates to obtain a set of perceptual distortion prediction label sequences of geometry and texture.

6. The color point cloud JND threshold prediction method according to claim 5, characterized in that: The perceptual distortion discriminator is used to perform perceptual distortion prediction on the test color point cloud and the corresponding color point cloud set of all compression rates to obtain a set of perceptual distortion prediction label sequences of geometry and texture, specifically: Fusion of the two-dimensional projection feature set F1 {F 1R ,F 1D } and {F 2R ,F 2D } predict the overall quality weight FQ of the color point cloud, integrating {F 1D ,F JND } and {F 2D ,F V }Predict the distribution weights FW of geometric distortion and texture distortion of distorted color point cloud; The quality weight FQ and the distortion distribution weight FW are weighted, and a perceptual distortion prediction label L is output.

7. The color point cloud JND threshold prediction method according to any one of claims 1 to 6, characterized in that: The binary JND search strategy is used to search the perceptual distortion prediction label sequence set to obtain the geometric JND threshold and texture JND threshold of the target color point cloud, and further includes: In the V-PCC framework, the quantization level parameter QP of the geometric image and the texture image is controlled according to the geometric JND threshold and the texture JND threshold of the target color point cloud.

8. A color point cloud JND threshold prediction device, characterized in that: include: Preprocessing module: used to extract the 2D and 3D features of the reference color point cloud and the distorted color point cloud respectively, calculate the pixel-level JND map using the reference projection image, and calculate the 3D visual mechanism features of the original color point cloud, and construct the JND training sample set driven by the mechanism and data jointly by the visual mechanism features and data features of different dimensions; Feature extraction module: used for inputting the JND training sample set into the multimodal feature extraction module, and extracting two-dimensional projection features and color point cloud features respectively through the multimodal feature extraction module to obtain a two-dimensional projection feature set and a three-dimensional point cloud feature set; Feature fusion module: used to fuse the two-dimensional projection feature set and the three-dimensional point cloud feature set to obtain multimodal fusion features, perform perceptual distortion prediction based on the multimodal fusion features, and obtain a set of geometric and texture perceptual distortion prediction label sequences; The JND threshold prediction module is used to search the perceptual distortion prediction label sequence set by adopting a binary block JND search strategy to obtain a geometric JND threshold and a texture JND threshold of the target color point cloud.

9. A device, characterized in that: The device comprises a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the color point cloud JND threshold prediction method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control the machine to perform a color point cloud JND threshold prediction method.

10. A storage medium, characterized in that: The device stores program instructions executable by a processor, wherein the program instructions are used to execute the color point cloud JND threshold prediction method according to any one of claims 1 to 7.

Images