A method and device for sending and receiving three-dimensional point cloud-assisted video semantic communication

Abstract

The present invention provides a sending and receiving method and device for three-dimensional point cloud-assisted video semantic communication. The sending method comprises: a transmitting end acquires a video, generates a mask image set for the first few frames, and transmits it to a receiving end; estimates the camera pose corresponding to the mask image set for the first few frames, and generates a three-dimensional point cloud. A mask image set is generated for the remaining video frames, and the projection of the corresponding three-dimensional point cloud at the corresponding camera pose is calculated. This projection, the mask image set for the remaining frames, and the semantic priority are input into an image compression model to obtain a residual semantic vector; and the residual semantic vector and other information are transmitted to a receiving end. The receiving method comprises: receiving the mask image set for the first few frames to generate a three-dimensional point cloud. The residual compressed semantic vector and other information are received, and the projection of the three-dimensional point cloud corresponding to the remaining frames at the corresponding camera pose is calculated. The projection, the residual compressed semantic vector, and the semantic priority are input into an image decompression model to restore the mask image set for the remaining frames. All mask image sets are synthesized to obtain a complete video. The present invention achieves efficient and high-quality transmission of video data through three-dimensional point cloud-assisted video semantic communication and joint coding of communication source and channel.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and communication technology, and in particular to a method and device for sending and receiving three-dimensional point cloud-assisted video semantic communication. Background Art

[0002] Semantic communication is a new type of communication method that emphasizes matching the semantic information recovered by the receiver with the semantic information extracted by the transmitter. This relaxes the error-free transmission requirement, requiring only that the semantic information recovered by the receiver match the semantic information extracted by the transmitter. This communication method differs from traditional communication methods by removing the bit-level error-free transmission requirement. This eliminates redundant data, reduces the amount of data transmitted, and improves transmission efficiency. Semantic communication is increasingly being applied to various fields, and with the development of related technologies, its application in communications, image processing, wireless communications, and other fields will become even more extensive.

[0003] In some specific application scenarios, such as power monitoring or inspection video transmission, channel conditions and bandwidth limitations often necessitate improving the compression ratio of the transmitted video. Furthermore, given the safety and stability implications of power monitoring, the quality and stability of the surveillance video must be high. Given these requirements and characteristics, improving the compression ratio and transmission efficiency of semantic communication is a pressing issue. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a method and apparatus for sending and receiving three-dimensional point cloud-assisted video semantic communication to solve the problem of low transmission efficiency in existing communication technologies.

[0005] One aspect of the present invention provides a method for transmitting three-dimensional point cloud-assisted video semantic communication, which is performed at a transmitting end and includes the following steps:

[0006] Acquire video data, extract a front video frame of the first number of frames from the front of the video data, perform semantic segmentation on the front video frame using a semantic segmentation model, generate masks for respective objects in the front video frame to form a first mask image set, and transmit the first mask image set to a receiving end;

[0007] Calculating corresponding first camera pose data based on the first mask image set using a camera pose estimation algorithm, and inputting the first mask image set and the first camera pose data into a neural radiance field model to extract previous frame three-dimensional point cloud data corresponding to the first mask image set;

[0008] Perform semantic segmentation on the remaining video frames in the video data to generate a second mask image set; use a feature point extraction and matching algorithm to match the second mask image set with the three-dimensional point cloud data of the previous frame, and generate a three-dimensional point cloud number for labeling; based on the three-dimensional point cloud data of the previous frame, the second mask image set, and the three-dimensional point cloud number corresponding to the second mask image set, use the camera pose estimation algorithm to calculate second camera pose data corresponding to the second mask image set, and calculate a back-frame projection of the three-dimensional point cloud data indicated by the three-dimensional point cloud number corresponding to the second mask image set relative to the second camera pose data;

[0009] Inputting the rear frame projection, the second mask image set, and the preset mask image semantic priority into a pre-trained image compression model to perform source-channel joint coding to obtain a residual compression semantic vector;

[0010] The residual compression semantic vector, the three-dimensional point cloud number corresponding to the second mask image set, the second camera pose data, the mask image semantic priority and the mask center position information of the second mask image set are transmitted to the receiving end through a channel.

[0011] In some embodiments of the present invention, the pre-trained image compression model uses a convolutional neural network; the training steps of the image compression model include:

[0012] Obtaining a training data set, the training data set including a sample mask image set, a sample projection, and a semantic priority of the sample mask image of a sample video; the sample projection is a projection of three-dimensional point cloud data corresponding to the sample mask image set on a camera pose corresponding to the sample mask image set;

[0013] Inputting the training data set into an initial image compression model and outputting a sample residual compression semantic vector;

[0014] Minimizing the deviation between the sample mask image and the output of the corresponding initial image decompression model to update the model parameters and obtain the pre-trained image compression model;

[0015] The output of the corresponding initial image decompression model is composed of the sample projection, the sample residual compression semantic vector and the sample mask image semantic priority, which are simulated by a channel model and then input into the initial image decompression model to obtain the restored sample mask image set.

[0016] In some embodiments of the present invention, the convolutional neural network performs adaptive downsampling according to the semantic priority of the mask image, and the higher the level of the semantic priority of the mask image, the fewer downsampling layers the corresponding mask image passes through; in the convolutional neural network, the number of jump connection layers of the corresponding downsampling layer is set according to the semantic priority of the mask image.

[0017] In some embodiments of the present invention, the camera pose estimation algorithm includes: a visual SLAM algorithm based on feature matching, or using a convolutional neural network to extract features and estimate the camera pose.

[0018] Another aspect of the present invention provides a method for receiving three-dimensional point cloud-assisted video semantic communication, characterized in that the method is performed at a receiving end and includes the following steps:

[0019] Receiving a first mask image set transmitted by a transmitting end; calculating corresponding first camera pose data based on the first mask image set using a camera pose estimation algorithm, and inputting the first mask image set and the first camera pose data into a neural radiance field model to extract previous frame three-dimensional point cloud data corresponding to the first mask image set;

[0020] Receiving the residual compression semantic vector, the three-dimensional point cloud number corresponding to the second mask image set, the second camera pose data, the mask image semantic priority, and the mask center position information of the second mask image set sent by the sending end; calculating the back frame projection using the three-dimensional point cloud data of the previous frame, the three-dimensional point cloud number corresponding to the second mask image set, and the second camera pose data;

[0021] Inputting the post-frame projection, the residual compression semantic vector, and the mask image semantic priority into a pre-trained image decompression model to perform source-channel joint decoding to obtain the second mask image set;

[0022] The second mask image set, the mask center position information of the second mask image set, and the first mask image set are synthesized to obtain the complete video data.

[0023] In some embodiments of the present invention, the pre-trained image decompression model adopts a deconvolutional neural network; the training steps of the image decompression model include:

[0024] Acquire a training data set, the training data set including sample projections transmitted through a simulation channel model, sample residual compression semantic vectors, and sample mask image semantic priorities; the sample projections are projections of three-dimensional point cloud data corresponding to the sample mask image set on camera poses corresponding to the sample mask image set; the sample residual compression semantic vectors are outputs of the corresponding initial image decompression model;

[0025] Inputting the training data set into an initial image decompression model and outputting a sample mask image set;

[0026] Minimize the deviation between the sample mask image output by the initial image decompression model and the sample mask image input to the initial image compression model to update the model parameters and obtain the pre-trained image decompression model.

[0027] In some embodiments of the present invention, the deconvolutional neural network performs adaptive upsampling according to the semantic priority of the mask image. The higher the level of the semantic priority of the mask image, the fewer upsampling layers the corresponding mask image passes through. In the deconvolutional network, the number of jump connection layers of the corresponding upsampling layer is set according to the semantic priority of the mask image.

[0028] In some embodiments of the present invention, the deconvolutional neural network performs adaptive upsampling according to the semantic priority of the mask image, and the method further includes:

[0029] Adjusting the weight of the convolution kernel according to the semantic priority of the mask image to achieve adaptive upsampling of different images, where the higher the priority image, the higher the weight of the corresponding convolution kernel;

[0030] Alternatively, an attention mechanism is introduced to dynamically adjust the weight of the feature map according to the semantic priority of the mask image to reduce downsampling of important semantic areas. The higher the priority of the image, the higher the weight of the corresponding feature map.

[0031] Another aspect of the present invention provides a device for three-dimensional point cloud-assisted video semantic communication, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0032] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0033] The beneficial effects of the present invention are at least:

[0034] The present invention provides a sending and receiving method and device for three-dimensional point cloud-assisted video semantic communication. The sending method comprises: a transmitting end acquires a video, generates a mask image set for the first few frames, and transmits it to a receiving end; estimates the camera pose corresponding to the mask image set for the first few frames, and generates a three-dimensional point cloud. A mask image set is generated for the remaining video frames, and the projection of the corresponding three-dimensional point cloud at the corresponding camera pose is calculated. This projection, the mask image set for the remaining frames, and the semantic priority are input into an image compression model to obtain a residual semantic vector; and the residual semantic vector and other information are transmitted to a receiving end. The receiving method comprises: receiving the mask image set for the first few frames to generate a three-dimensional point cloud. The residual compressed semantic vector and other information are received, and the projection of the three-dimensional point cloud corresponding to the remaining frames at the corresponding camera pose is calculated. The projection, the residual compressed semantic vector, and the semantic priority are input into an image decompression model to restore the mask image set for the remaining frames. All mask image sets are synthesized to obtain a complete video. The present invention achieves efficient and high-quality transmission of video data through three-dimensional point cloud-assisted video semantic communication and joint coding of communication source and channel. By combining 3D point clouds with video semantic information, this invention achieves efficient transmission and reconstruction of video data, improving both the compression rate and transmission efficiency of video transmission. It can also adapt to poor channel conditions, enabling the normal transmission of monitoring or inspection videos in power communication scenarios with high bandwidth limitations.

[0035] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0036] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0038] Figure 1 This is a flow chart of a method for sending three-dimensional point cloud-assisted video semantic communication according to an embodiment of the present invention.

[0039] Figure 2 This is a flow chart of a receiving method for three-dimensional point cloud-assisted video semantic communication according to an embodiment of the present invention.

[0040] Figure 3 This is a structural diagram of a three-dimensional point cloud-assisted video semantic communication system according to another embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0042] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0043] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0044] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0045] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0046] Semantic communication refers to a communication method that extracts semantic information from a signal source, encodes it, transmits it, and ultimately decodes it at the receiving end. Unlike traditional communication methods, which require the decoded information at the receiving end to be strictly consistent with the encoded information at the sending end, effectively achieving bit-level error-free transmission, semantic communication only requires that the semantic information recovered by the receiving end match the semantic information extracted by the sending end. Because semantic communication relaxes the error-free transmission requirement, it can eliminate redundant data, reduce the amount of transmitted data, and improve transmission efficiency.

[0047] An embodiment of the present invention provides a method for transmitting 3D point cloud-assisted video semantic communication, such as Figure 1 As shown, the method is executed at the sending end and includes the following steps S101 to S105:

[0048] Step S101: Obtain video data, extract the front video frame of the first frame number at the front end of the video data, perform semantic segmentation on the front video frame using a semantic segmentation model, generate masks for each object in the front video frame, form a first mask image set, and transmit the first mask image set to the receiving end.

[0049] Step S102: Calculate the corresponding first camera pose data based on the first mask image set using a camera pose estimation algorithm, and input the first mask image set and the first camera pose data into a neural radiation field model to extract the previous frame three-dimensional point cloud data corresponding to the first mask image set.

[0050] Step S103: Perform semantic segmentation on the remaining video frames in the video data to generate a second mask image set. Using a feature point extraction and matching algorithm, the second mask image set is matched with the 3D point cloud data of the previous frame, and a 3D point cloud number is generated for labeling. Based on the 3D point cloud data of the previous frame, the second mask image set, and the 3D point cloud number corresponding to the second mask image set, a camera pose estimation algorithm is used to calculate the second camera pose data corresponding to the second mask image set. Furthermore, the subsequent frame projection of the 3D point cloud data indicated by the 3D point cloud number corresponding to the second mask image set relative to the second camera pose data is calculated.

[0051] Step S104: inputting the subsequent frame projection, the second mask image set, and the preset mask image semantic priority into a pre-trained image compression model to perform source-channel joint coding to obtain a residual compression semantic vector.

[0052] Step S105: Transmitting the residual compressed semantic vector, the three-dimensional point cloud number corresponding to the second mask image set, the second camera pose data, the mask image semantic priority, and the mask center position information of the second mask image set to the receiving end through the channel.

[0053] In step S101, a semantic segmentation model is a deep learning model used to assign each pixel in an image to a specific semantic category. This model is typically based on a convolutional neural network (CNN) structure, enabling fine-grained image segmentation. It is trained using pixel-level annotated data to support image and semantic understanding tasks. Common semantic segmentation models include FCN (Fully Convolutional Network), U-Net, and SegNet.

[0054] In step S102, a camera pose estimation algorithm is used to determine the position and orientation of the camera in three-dimensional space. Camera pose estimation infers the position and attitude of the camera by analyzing feature points, edges or other visual information in the image, combined with computational geometry and optimization methods. Such estimation results are crucial for applications such as target tracking, three-dimensional reconstruction and virtual reality. Camera pose estimation algorithms usually improve the accuracy and efficiency of estimation by using sensor data (such as data obtained by inertial measurement units and depth sensors) and efficient algorithms (such as iterative closest point and RANSAC algorithms).

[0055] In step S102, the Neural Radiance Field Model (NeRF) is a three-dimensional reconstruction method that uses deep learning technology to achieve reconstruction from two-dimensional images to three-dimensional scenes. The Neural Radiance Field Model (NeRF) uses a neural network to model the radiation field properties of each spatial point in the scene, treating it as a function of density and color. The method uses a pair of light directions and origins to parameterize the light path in the scene, and achieves high-quality three-dimensional reconstruction by rendering the observed image and minimizing the loss function. The Neural Radiance Field Model (NeRF) does not require explicit geometric models or depth sensor data, and can accurately reconstruct complex and texture-rich scenes. The Neural Radiance Field Model (NeRF) includes: FastNeRF model, NeRF-W model, D-NeRF model or AtlasNeRF model.

[0056] In step S103, the feature point extraction and matching algorithm refers to detecting the feature points of the mask image corresponding to each object in the second mask image set, and matching them with the three-dimensional point cloud data of the previous frame to generate a three-dimensional point cloud code corresponding to the second mask image set. Feature point extraction algorithms usually use local feature descriptors. These descriptors can describe the local area around the feature points so that these feature points have a unique representation in the feature space. The matching algorithm is used to match these feature points between different image frames to find corresponding feature points in the same scene. The matching algorithm usually uses a similarity measure between feature descriptors, such as a distance calculation based on feature vectors to find the best feature point match.

[0057] In step S103, projection refers to a two-dimensional image of the three-dimensional point cloud data on the camera imaging plane. Projection can provide visual information such as the position, shape and depth of the object from the camera's perspective. In this embodiment, the rear frame projection is used to assist in abstracting and compressing the three-dimensional scene information.

[0058] In step S104 , the semantic priority of the mask image can be changed arbitrarily according to task requirements.

[0059] In step S104, joint source-channel coding is a communication coding method that optimizes source coding and channel coding. Source coding is a data compression process that aims to minimize redundancy in the source to reduce the amount of data transmitted. Channel coding, on the other hand, is a process that adds redundancy to mitigate channel noise and protect the transmitted data. Joint source-channel coding optimizes source coding and channel coding as a whole to maximize transmission reliability and efficiency.

[0060] In step S104 , the residual compressed semantic vector refers to a residual semantic vector obtained by extracting the semantic vector of the image and removing the semantic information including the projection.

[0061] In some embodiments of the present invention, the pre-trained image compression model uses a convolutional neural network. The training steps of the image compression model include:

[0062] Obtain a training dataset, which includes a sample mask image set, sample projections, and semantic priorities of the sample mask images of the sample video. The sample projection is the projection of the 3D point cloud data corresponding to the sample mask image set onto the camera pose corresponding to the sample mask image set.

[0063] The training dataset is input into the initial image compression model, and the sample residual compression semantic vector is output.

[0064] Minimize the deviation between the sample mask image and the output of the corresponding initial image decompression model to update the model parameters and obtain a pre-trained image compression model.

[0065] The output of the corresponding initial image decompression model is composed of sample projection, sample residual compression semantic vector and sample mask image semantic priority, which are simulated through the channel model and then input into the initial image decompression model to obtain the restored sample mask image set.

[0066] The simulated channel model can simulate the real channel environment, including noise and attenuation during signal transmission. Adding the simulated channel model makes the trained image compression model more resistant to noise.

[0067] In some embodiments of the present invention, a convolutional neural network performs adaptive downsampling based on the semantic priority of the mask image. The higher the level of the mask image semantic priority, the fewer downsampling layers the corresponding mask image passes through. In the convolutional neural network, the number of skip connections in the corresponding downsampling layer is set based on the semantic priority of the mask image.

[0068] In some embodiments of the present invention, the camera pose estimation algorithm includes: a visual SLAM algorithm based on feature matching, or using a convolutional neural network to extract features and estimate the camera pose.

[0069] An embodiment of the present invention provides a receiving method for 3D point cloud-assisted video semantic communication, which is executed at a receiving end, such as Figure 2 As shown, the method includes the following steps S201 to S204:

[0070] Step S201: Receive a first mask image set transmitted by a transmitter. Utilize a camera pose estimation algorithm to calculate corresponding first camera pose data based on the first mask image set. Input the first mask image set and the first camera pose data into a neural radiance field model to extract the previous frame 3D point cloud data corresponding to the first mask image set.

[0071] Step S202: Receive the residual compression semantic vector, the 3D point cloud number corresponding to the second mask image set, the second camera pose data, the mask image semantic priority, and the mask center position information of the second mask image set from the transmitter. Calculate the projection of the subsequent frame using the 3D point cloud data of the previous frame, the 3D point cloud number corresponding to the second mask image set, and the second camera pose data.

[0072] Step S203: input the post-frame projection, residual compression semantic vector and mask image semantic priority into a pre-trained image decompression model to perform source-channel joint decoding to obtain a second mask image set.

[0073] Step S204: synthesizing the second mask image set, the mask center position information of the second mask image set, and the first mask image set to obtain complete video data.

[0074] In step S202, the mask center position information of the second mask image set is used to tile the restored images according to the center position information corresponding to the images, so as to restore the complete video frame containing the images of each object.

[0075] In some embodiments of the present invention, the pre-trained image decompression model uses a deconvolutional neural network. The training steps of the image decompression model include:

[0076] Obtain a training dataset, which includes sample projections transmitted through the simulation channel model, sample residual compressed semantic vectors, and sample mask image semantic priorities. The sample projections are projections of the 3D point cloud data corresponding to the sample mask image set onto the camera pose corresponding to the sample mask image set. The sample residual compressed semantic vectors are the output of the corresponding initial image decompression model.

[0077] The training dataset is input into the initial image decompression model and a set of sample mask images is output.

[0078] Minimize the deviation between the sample mask image output by the initial image decompression model and the sample mask image input to the initial image compression model to update the model parameters and obtain a pre-trained image decompression model.

[0079] Specifically, in the above embodiment, the convolutional neural network and the deconvolutional neural network are used in conjunction with each other. The convolutional neural network is used to extract image features and output a low-resolution semantic vector. The deconvolutional neural network is used to upsample the low-resolution semantic vector to the resolution of the original input image. For important semantic areas in the semantic priority of the mask image, the neural network will retain more detailed information to avoid excessive blurring or distortion.

[0080] Furthermore, the image compression and decompression models are trained end-to-end using joint source-channel coding. This technique involves adding noise to the semantically-based source encoding and decoding training of the image compression and decompression models at both the receiving and transmitting ends. This ensures that the source codes obtained by the transmitting end are also highly resilient to noise. This means that the compression and decompression models learn together to maximize the performance of the entire compression and decompression system.

[0081] In some embodiments of the present invention, a deconvolutional neural network performs adaptive upsampling based on the semantic priority of the mask image. The higher the level of the mask image semantic priority, the fewer upsampling layers the corresponding mask image passes through. In the deconvolutional network, the number of skip connections in the corresponding upsampling layer is set based on the semantic priority of the mask image.

[0082] In some embodiments of the present invention, the deconvolutional neural network performs adaptive upsampling according to the semantic priority of the mask image, and the method further includes:

[0083] The weight of the convolution kernel is adjusted according to the semantic priority of the mask image to achieve adaptive upsampling of different images. The higher the priority of the image, the higher the weight of the convolution kernel.

[0084] Alternatively, an attention mechanism can be introduced to dynamically adjust the weight of the feature map according to the semantic priority of the mask image, reducing the downsampling of important semantic areas. The higher the priority of the image, the higher the weight of the corresponding feature map.

[0085] Feature maps are image feature representations obtained through convolution operations in convolutional and deconvolutional neural networks. At each layer of the neural network, a series of feature maps are generated, each corresponding to a feature extracted by a convolution kernel in the network. These feature maps capture characteristic information of the input image at different levels of abstraction, such as edges, texture, and shape.

[0086] Another embodiment of the present invention provides a method for 3D point cloud data-assisted video semantic communication, the method comprising the following steps S301 to S308:

[0087] Step S301: Use the semantic segmentation model to perform semantic segmentation on the image to be transmitted on the robot side, mask out different objects in the image separately to form a mask image set, and process different objects in the image set separately.

[0088] Step S302: The obtained mask image sets are processed separately for the first few frames of different objects at both the transmitting and receiving ends. The feature points of the objects are marked using a camera pose estimation algorithm, and feature point matching is performed to infer the position and pose of the camera.

[0089] Step S303: The mask image set of the first few frames of the object and the corresponding camera pose are used as training data, and high-quality three-dimensional reconstruction is achieved through NeRF neural radiation field technology. Subsequently, the three-dimensional point cloud data of the object is extracted from the NeRF model at both the transmitting and receiving ends.

[0090] Step S304: Based on the 3D point cloud data of the objects obtained in step S303, subsequent video frames are transmitted. First, feature point extraction and matching techniques are used to obtain the 3D point cloud numbers corresponding to the masked objects in the subsequent frames relative to the 3D point cloud data of the previous frames. Based on the current object mask image and the 3D point cloud data corresponding to the number, the camera pose of the object in the frame corresponding to the 3D point cloud data is estimated, and the projection of the 3D point cloud data at this camera pose is obtained.

[0091] Step S305: The projection image obtained in step S304, the mask image of the object, and the semantic priority of the object are input into the image compression model to obtain a residual compressed semantic vector of the mask image relative to the projection image, and the vector is jointly encoded with the source and channel.

[0092] Step S306: The residual compressed semantic vector obtained in step S305, the three-dimensional point cloud number obtained in step S304, the camera pose data, the semantic priority of the object and the center position of the mask are transmitted to the receiving end through the channel.

[0093] Step S307: At the receiving end, the three-dimensional point cloud data obtained in step S303 and the three-dimensional point cloud number obtained in step S306 are used to project the three-dimensional point cloud data corresponding to this camera pose. The projection, the residual compressed semantic vector obtained in step S305 and the semantic priority of the object are input into the image decompression model to obtain a restored object mask map.

[0094] Step S308: synthesize the mask images of different objects obtained in step S307 to obtain a complete video frame, and finally complete the video transmission.

[0095] In some embodiments of the present invention, the camera pose estimation algorithm may use traditional feature point extraction and matching, or may use a deep learning network to predict the camera pose.

[0096] In some embodiments of the present invention, considering the delay requirement of video transmission, the NeRF model used may be FastNeRF or a similar neural radiation field variant for simplifying calculations.

[0097] In some embodiments of the present invention, the semantic priority of each object may be specified in advance according to a monitoring or inspection task, and may be changed arbitrarily according to task requirements later.

[0098] In some embodiments of the present invention, objects with different priorities are compressed with different downsampling times in the compression model. Objects with lower priorities will pass through more downsampling layers to obtain semantic compression vectors with higher compression rates.

[0099] Another embodiment of the present invention provides a system for 3D point cloud data-assisted video semantic communication. The execution process of the semantic communication system is mainly divided into three parts:

[0100] 1) The semantic segmentation model masks out different objects in the inspection image separately.

[0101] 2) Using the mask images of different objects in the first few frames of the video, we estimate the camera pose, reconstruct the NeRF neural radiation field 3D, and extract 3D point cloud data from the NeRF model. 3D point cloud data of different objects in the scene are obtained at both the transmitter and receiver.

[0102] 3) Use 3D point cloud information to assist in object mask image compression in video frames.

[0103] Furthermore, if Figure 3 As shown, the execution process of the large semantic segmentation model includes: On the robot side, the large semantic segmentation model performs semantic segmentation on the inspection images to be transmitted, individually masking different objects in the image to form a set of masked images. The significance of this step is: first, in subsequent image compression, different levels of compression can be achieved based on the semantic priority of different objects, thereby improving the compression rate. Second, in the subsequent 3D reconstruction, the reconstruction effect of single objects will be significantly improved compared to the 3D reconstruction of complex scenes, which can improve the overall transmission quality.

[0104] Furthermore, in the process of object 3D reconstruction and 3D point cloud data acquisition based on camera pose estimation, Neural Radiance Field (NeRF), for each object, sparse visual reconstruction of the object can be achieved based on its mask images at different viewing angles in the previous frames. The process includes the following steps S401 to S403:

[0105] Step S401: Estimate the camera pose of the mask atlas using a traditional camera pose estimation algorithm based on key point extraction and matching technology.

[0106] Step S402: The NeRF neural radiance field technique takes several scene images and their camera poses as input and uses a neural network to model the radiance field properties of each spatial point in the scene, treating them as a function of density and color. This method parameterizes the ray paths in the scene using a pair of ray directions and an origin, and achieves high-quality 3D reconstruction by rendering the observed images and minimizing a loss function. Therefore, using the results obtained in step S301 as training data, high-quality 3D reconstruction of objects using NeRF neural radiance fields can be achieved.

[0107] Step S403: Extract the 3D point cloud data of the object from the trained NeRF model to complete the 3D reconstruction of the object. The 3D point cloud data will be used as an auxiliary compression tool for subsequent video frame transmission.

[0108] Furthermore, if Figure 2 As shown, the process of using 3D point cloud information to assist in object mask image compression in a video frame includes the following steps S501 to S504:

[0109] Step S501: extracting key points of an object and matching the key points to obtain a corresponding three-dimensional point cloud model of the object from an existing three-dimensional point cloud dataset.

[0110] Step S502: Input the mask image of the object and its corresponding three-dimensional point cloud model into the camera pose estimation network (a traditional camera pose estimation algorithm based on key point matching technology can be used, or a deep network can be used for estimation) to estimate the camera pose of the mask image relative to the three-dimensional point cloud data, and obtain the projection image of the three-dimensional point cloud data under the camera pose.

[0111] Step S503: Since the projection image is available at both the transmitting and receiving ends, the receiving end can obtain it using the 3D point cloud data from the receiving end and the camera pose transmitted from the transmitting end. Therefore, during image compression at the transmitting end, the projection image is introduced to achieve high compression of the object mask image.

[0112] Step S504: The compression model is composed of a convolutional neural network. Through end-to-end joint training, projection maps are introduced in the heads of both the encoder and the decoder to control the output of the encoder to be a residual semantic compression vector that does not contain projection map features.

[0113] The number of downsampling layers in the convolutional neural network varies depending on the semantic priority of the transmitted object. Objects with higher semantic priorities undergo fewer downsampling layers during encoding, preserving more semantic information. Therefore, the semantic encoder and decoder also have a stackable slice structure in addition to the main body.

[0114] In summary, the present invention provides a sending and receiving method and apparatus for 3D point cloud-assisted video semantic communication. The sending method comprises: a transmitting end acquires a video, generates a mask image set for the first few frames, and transmits it to a receiving end; estimates the camera pose corresponding to the mask image set for the first few frames, and generates a 3D point cloud. Then, a mask image set for the remaining video frames is generated, and the projection of the corresponding 3D point cloud at the corresponding camera pose is calculated. This projection, the mask image set for the remaining frames, and the semantic priority are input into an image compression model to obtain a residual semantic vector; and this residual semantic vector and other information are transmitted to a receiving end. The receiving method comprises: receiving the mask image set for the first few frames to generate a 3D point cloud; receiving the residual compressed semantic vector and other information; calculating the projection of the 3D point cloud corresponding to the remaining frames at the corresponding camera pose; inputting the projection of the subsequent frames, the residual compressed semantic vector, and the semantic priority into an image decompression model to restore the mask image set for the remaining frames; and synthesizing all the mask image sets to obtain a complete video. The present invention achieves efficient and high-quality transmission of video data through 3D point cloud-assisted video semantic communication and joint coding of communication source and channel.

[0115] Corresponding to the above method, the present invention also provides a system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0116] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0117] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0118] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0119] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0120] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for transmitting three-dimensional point cloud-assisted video semantic communication, characterized in that: The method is executed at the sending end and includes the following steps: Acquire video data, extract a front video frame of the first number of frames from the front of the video data, perform semantic segmentation on the front video frame using a semantic segmentation model, generate masks for respective objects in the front video frame to form a first mask image set, and transmit the first mask image set to a receiving end; Calculating corresponding first camera pose data based on the first mask image set using a camera pose estimation algorithm, and inputting the first mask image set and the first camera pose data into a neural radiance field model to extract previous frame three-dimensional point cloud data corresponding to the first mask image set; Perform semantic segmentation on the remaining video frames in the video data to generate a second mask image set; use a feature point extraction and matching algorithm to match the second mask image set with the three-dimensional point cloud data of the previous frame, and generate a three-dimensional point cloud number for labeling; based on the three-dimensional point cloud data of the previous frame, the second mask image set, and the three-dimensional point cloud number corresponding to the second mask image set, use the camera pose estimation algorithm to calculate second camera pose data corresponding to the second mask image set, and calculate a back-frame projection of the three-dimensional point cloud data indicated by the three-dimensional point cloud number corresponding to the second mask image set relative to the second camera pose data; Inputting the rear frame projection, the second mask image set, and the preset mask image semantic priority into a pre-trained image compression model to perform source-channel joint coding to obtain a residual compression semantic vector; The residual compression semantic vector, the three-dimensional point cloud number corresponding to the second mask image set, the second camera pose data, the mask image semantic priority and the mask center position information of the second mask image set are transmitted to the receiving end through a channel.

2. The method according to claim 1, characterized in that The pre-trained image compression model adopts a convolutional neural network; the training steps of the image compression model include: Obtaining a training data set, the training data set including a sample mask image set, a sample projection, and a semantic priority of the sample mask image of a sample video; the sample projection is a projection of three-dimensional point cloud data corresponding to the sample mask image set on a camera pose corresponding to the sample mask image set; Inputting the training data set into an initial image compression model and outputting a sample residual compression semantic vector; Minimizing the deviation between the sample mask image and the output of the corresponding initial image decompression model to update the model parameters and obtain the pre-trained image compression model; The output of the corresponding initial image decompression model is composed of the sample projection, the sample residual compression semantic vector and the sample mask image semantic priority, which are simulated by a channel model and then input into the initial image decompression model to obtain the restored sample mask image set.

3. The method according to claim 2, characterized in that The convolutional neural network performs adaptive downsampling according to the semantic priority of the mask image. The higher the level of the semantic priority of the mask image, the fewer downsampling layers the corresponding mask image passes through. In the convolutional neural network, the number of skip connection layers of the corresponding downsampling layer is set according to the semantic priority of the mask image.

4. The method according to claim 1, wherein The camera pose estimation algorithm includes: a visual SLAM algorithm based on feature matching, or a convolutional neural network to extract features and estimate the camera pose.

5. A receiving method for three-dimensional point cloud-assisted video semantic communication, characterized in that: The method is executed at the receiving end and includes the following steps: Receiving a first mask image set transmitted by a transmitting end; calculating corresponding first camera pose data based on the first mask image set using a camera pose estimation algorithm, and inputting the first mask image set and the first camera pose data into a neural radiance field model to extract previous frame three-dimensional point cloud data corresponding to the first mask image set; Receiving the residual compression semantic vector, the three-dimensional point cloud number corresponding to the second mask image set, the second camera pose data, the mask image semantic priority, and the mask center position information of the second mask image set sent by the sending end; calculating the back frame projection using the three-dimensional point cloud data of the previous frame, the three-dimensional point cloud number corresponding to the second mask image set, and the second camera pose data; Inputting the post-frame projection, the residual compression semantic vector, and the mask image semantic priority into a pre-trained image decompression model to perform source-channel joint decoding to obtain the second mask image set; The second mask image set, the mask center position information of the second mask image set, and the first mask image set are synthesized to obtain the complete video data.

6. The method according to claim 5, characterized in that The pre-trained image decompression model adopts a deconvolutional neural network; The training steps of the image decompression model include: Acquire a training data set, the training data set including sample projections transmitted through a simulation channel model, sample residual compression semantic vectors, and sample mask image semantic priorities; the sample projections are projections of three-dimensional point cloud data corresponding to the sample mask image set on camera poses corresponding to the sample mask image set; the sample residual compression semantic vectors are outputs of the corresponding initial image decompression model; Inputting the training data set into an initial image decompression model and outputting a sample mask image set; Minimize the deviation between the sample mask image output by the initial image decompression model and the sample mask image input to the initial image compression model to update the model parameters and obtain the pre-trained image decompression model.

7. The method according to claim 6, characterized in that The deconvolution neural network performs adaptive upsampling according to the semantic priority of the mask image. The higher the level of the semantic priority of the mask image, the fewer upsampling layers the corresponding mask image passes through. In the deconvolution network, the number of skip connection layers of the corresponding upsampling layer is set according to the semantic priority of the mask image.

8. The method according to claim 6, characterized in that The deconvolutional neural network performs adaptive upsampling according to the semantic priority of the mask image, and the method further includes: Adjusting the weight of the convolution kernel according to the semantic priority of the mask image to achieve adaptive upsampling of different images, where the higher the priority image, the higher the weight of the corresponding convolution kernel; Alternatively, an attention mechanism is introduced to dynamically adjust the weight of the feature map according to the semantic priority of the mask image to reduce downsampling of important semantic areas. The higher the priority of the image, the higher the weight of the corresponding feature map.

9. A device for three-dimensional point cloud-assisted video semantic communication, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as described in any one of claims 1 to 4, or implements the steps of the method as described in any one of claims 5 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images