Semantic communication method, device and equipment and readable storage medium

Abstract

The invention provides a semantic communication method, device and equipment and a readable storage medium, and aims to solve the problem of poor semantic communication effect of a communication system for multi-modal data transmission. The method comprises the steps that based on a target video and a semantic knowledge base, space-time sketch information of the target video is generated, and the space-time sketch information comprises at least one of a target object in the target video, attribute information of the target object and relation information of the target object; performing semantic flow transmission on the space-time sketch information; and generating a reconstructed video based on the space-time sketch information, and performing optimization processing based on the reconstructed video. According to the method, bandwidth resources can be greatly saved, and the semantic communication effect for the multi-modal data is improved.

Description

Technical Field

[0001] The present invention relates to the field of video transmission technology, and in particular to a semantic communication method, device, equipment and readable storage medium. Background Art

[0002] Due to the increase in demand for mobile communication services and the emergence of new application scenarios, higher-level requirements for mobile communication capabilities have been derived, promoting the development of the sixth generation of mobile communication technology (6-Generation, 6G). The application of 6G involves high-definition video, virtual reality (VR), augmented reality (AR), holographic projection and other large-capacity multimodal services. These multimodal services have put forward new requirements for intelligence, personalization and collaboration. Semantic communication has great potential in the above multimodal services. Related technologies have explored communication systems for single-mode data transmission such as text. However, the semantic communication effect of communication systems for multimodal data transmission is poor. Summary of the invention

[0003] The embodiments of the present invention provide a semantic communication method, apparatus, device and readable storage medium. In current practical applications, the semantic communication effect of a communication system for multimodal data transmission is relatively poor.

[0004] In order to solve the above problems, the embodiments of the present invention adopt the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a semantic communication method, including:

[0006] Based on the target video and the semantic knowledge base, generating spatiotemporal sketch information of the target video, the spatiotemporal sketch information including at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object;

[0007] Performing semantic streaming transmission on the spatiotemporal sketch information;

[0008] A reconstructed video is generated based on the spatiotemporal sketch information, and optimization processing is performed based on the reconstructed video.

[0009] Optionally, generating spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base includes:

[0010] Cut the target video into multiple sub-videos;

[0011] Determining a base video frame for each sub-video in the plurality of sub-videos;

[0012] Determine sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame;

[0013] Based on the sketch representation information of the base video frame, spatiotemporal sketch information of the target video is generated.

[0014] Optionally, the generating the spatiotemporal sketch information of the target video based on the sketch representation information of the base video frame includes:

[0015] Determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video;

[0016] Predicting sketch key points in the target video frame according to the optical flow map;

[0017] Sketch representation information of the target video frame is generated according to the optical flow map, the mask of the moving target and the sketch key points, and the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

[0018] Optionally, generating the sketch representation information of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points includes:

[0019] Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points;

[0020] Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch;

[0021] Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object;

[0022] Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch;

[0023] The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

[0024] Optionally, the performing optimization processing based on the reconstructed video includes:

[0025] According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, optimization processing is performed with the goal of improving the evaluation quality and the semantic fidelity.

[0026] Optionally, the performing optimization processing based on the reconstructed video includes:

[0027] Obtaining a first category of the reconstructed video and a second category of the target video;

[0028] An optimization process is performed according to the difference between the first category and the second category.

[0029] Optionally, the performing semantic stream transmission on the spatiotemporal sketch information includes:

[0030] The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

[0031] In a second aspect, an embodiment of the present invention provides a semantic communication device, the semantic communication device comprising a sending module, a network transmission module and a receiving module;

[0032] The sending module is used to generate spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base, wherein the spatiotemporal sketch information includes at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object;

[0033] The network transmission module is used to perform semantic stream transmission on the spatiotemporal sketch information;

[0034] The receiving module is used to generate a reconstructed video based on the spatiotemporal sketch information, and optimize the sending module, the network transmission module and the receiving module based on the reconstructed video.

[0035] Optionally, the sending module includes:

[0036] A cutting unit, used for cutting a target video into multiple sub-videos;

[0037] A first determining unit, configured to determine a base video frame for each of the plurality of sub-videos;

[0038] A second determining unit, configured to determine the sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame;

[0039] A generating unit is used to generate spatiotemporal sketch information of the target video based on the sketch representation information of the base video frame.

[0040] Optionally, the generating unit includes:

[0041] A determination subunit, configured to determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video;

[0042] A prediction subunit, configured to predict sketch key points in the target video frame according to the optical flow map;

[0043] A generating subunit is used to generate the sketch representation information of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points, wherein the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

[0044] Optionally, the generating subunit is specifically used for:

[0045] Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points;

[0046] Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch;

[0047] Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object;

[0048] Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch;

[0049] The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

[0050] Optionally, the receiving module is specifically used for:

[0051] generating a reconstructed video based on the spatiotemporal sketch information;

[0052] According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, the sending module, the network transmission module and the receiving module are optimized with the goal of improving the evaluation quality and the semantic fidelity.

[0053] Optionally, the receiving module is specifically used for:

[0054] generating a reconstructed video based on the spatiotemporal sketch information;

[0055] Obtaining a first category of the reconstructed video and a second category of the target video;

[0056] The sending module, the network transmission module and the receiving module are optimized according to the difference between the first category and the second category.

[0057] Optionally, the network transmission module is specifically used to:

[0058] The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

[0059] In a third aspect, an embodiment of the present invention provides a communication device, the communication device comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;

[0060] The processor is used to read the program in the memory to implement the steps in the method described in the first aspect.

[0061] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, wherein the program or instruction, when executed by a processor, implements the steps in the method described in the first aspect.

[0062] In an embodiment of the present invention, the semantic communication method can generate spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base, and perform semantic stream transmission on the spatiotemporal sketch information. The semantic stream transmission can focus on the semantic features within the transmitted information and transmit semantic information, thereby greatly saving bandwidth resources and taking into account the spatiotemporal relationship in the video, so that the effect of semantic communication for multimodal data is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0064] Figure 1 is a flow chart of a semantic communication method provided by an embodiment of the present invention;

[0065] Figure 2 is a flow chart of generating a reconstructed video frame provided by an embodiment of the present invention;

[0066] Figure 3 This is one of the structural schematic diagrams of a semantic communication device provided by an embodiment of the present invention;

[0067] Figure 4 This is a second structural diagram of a semantic communication device provided by an embodiment of the present invention;

[0068] Figure 5 It is a structural diagram of a communication device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0071] See also Figure 1 , Figure 1 is a flowchart of a semantic communication method provided by an embodiment of the present invention, the semantic communication method comprising:

[0072] Step 101, based on the target video and the semantic knowledge base, generate spatiotemporal sketch information of the target video, the spatiotemporal sketch information includes at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object.

[0073] Specifically, the target video can be any video, and the semantic knowledge base can include large-scale prior rules, methods and training models to provide guidance for semantic information processing and perceive semantic features. The semantic knowledge base can match natural language understanding and keywords by machines, computers, etc.; based on the semantic knowledge base, the semantic features of each of the multiple video frames of the target video are extracted, and the semantic features of each of the multiple video frames are encoded based on the encoder to achieve semantic extraction and obtain the spatiotemporal sketch information of the target video.

[0074] The spatiotemporal sketch information may include the target object in the target video, the attribute information of the target object, and the relationship information of the target object. The target object may be an object that needs to be paid attention to in the target video. For example, in a video of a football game, the target object may be a football, a player, etc. A video may include multiple target objects. When the target object is a player, the attribute information of the target object may be the color of the jersey worn by the player, the skin color of the player, the height of the player, etc. When the two target objects are a player and a football, respectively, the relationship between the target objects may be a player kicking a football. The spatiotemporal sketch information may also include spatiotemporal information. In a video, the behavior and movement trajectory of the target object are usually continuous, and there is spatiotemporal information between the multiple video frames included in the target video.

[0075] Step 102: Perform semantic stream transmission on the spatiotemporal sketch information.

[0076] Specifically, the semantic stream transmission may be the transmission of semantic information, may adopt a semantic entropy compression algorithm for semantic stream transmission, and may be transmitted with the aid of multiple access technology, such as multiple access transmission based on non-orthogonal multiple access (NOMA).

[0077] Step 103: Generate a reconstructed video based on the spatiotemporal sketch information, and perform optimization processing based on the reconstructed video.

[0078] Specifically, the generating of the reconstructed video may be based on decoding and reconstructing the spatiotemporal sketch information by a decoder and the semantic knowledge base, generating a plurality of reconstructed images, the reconstructed images may be video frames, and combining a plurality of reconstructed video frames to obtain the reconstructed video. Figure 2 A flowchart for generating a reconstructed video frame is provided in an embodiment of the present invention, such as Figure 2 As shown, when generating images based on spatiotemporal sketch information, semantic segmentation and color masks can also be used to randomly select points in the video frame to extract the color information of the video frame. Colors that have an impact on vision can be extracted to as many points as possible at the same time for the receiving module to generate better images. Due to the uneven RGB color space, the color difference obtained based on the spatial distance of the three RGB colors does not conform to human vision. The color model (Label) can be used to describe the color space of the color difference through brightness, red-green difference, and blue-yellow difference. In addition, the number of sampling points cannot be too large. In the experiment, 200 pixels of color information can be selected to be transmitted.

[0079] In addition to color information, semantic segmentation results can also improve the quality of generated images. Each pixel in the semantic segmentation map represents the real-world category of the pixel, such as pedestrians and vehicles. By combining the semantic segmentation results with the sketch, each contour in the sketch is given semantic information, while the original sketch only represents whether the pixel on the contour is at the edge of the object. The contour of the object contains more object category information. In the process of reconstructing the image by the generative adversarial network, the characteristics of the original image can be grasped more accurately and the image can be better restored.

[0080] There are many quantitative evaluation indicators for image quality. The image coding result can be evaluated by bits per pixel (BPP), which indicates how many bits are needed on average to store each pixel of the original image. The less memory or bandwidth required for transmission, the better the image compression. The decoding result can be achieved by the following three commonly used image evaluation indicators: mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM), which evaluate the effect of reconstructed images. It can also be evaluated based on quality of experience (QoE) and semantic similarity.

[0081] In an embodiment of the present invention, the semantic communication method can generate spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base, and perform semantic stream transmission on the spatiotemporal sketch information. The semantic stream transmission can focus on the semantic features within the transmitted information and transmit semantic information, thereby greatly saving bandwidth resources and taking into account the spatiotemporal relationship in the video, so that the effect of semantic communication for multimodal data is better.

[0082] Optionally, generating spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base includes:

[0083] Cut the target video into multiple sub-videos;

[0084] Determining a base video frame for each sub-video in the plurality of sub-videos;

[0085] Determine sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame;

[0086] Based on the sketch representation information of the base video frame, spatiotemporal sketch information of the target video is generated.

[0087] Specifically, each sub-video includes multiple video frames, and a basic video frame can be determined from the multiple video frames included in each sub-video. For example, the basic video frame can be the first video frame of the sub-video; the sketch representation information of the basic video frame of each sub-video can be determined by performing edge detection on the basic video frame to obtain an edge map of the basic video frame, performing single-pixel maximum suppression on the edge map of the basic video frame, and judging the category of each pixel point in the edge map of the basic video frame to determine whether each pixel point is a pixel point in the sketch and whether it needs to be connected to other pixels, thereby obtaining the sketch representation information of the basic video frame.

[0088] In an embodiment of the present invention, the semantic communication method can cut the target video into multiple sub-videos, extract spatiotemporal sketch information for each sub-video, and then obtain the spatiotemporal sketch information of the target video, so that the spatiotemporal sketch information of the target video is more accurate.

[0089] Optionally, the generating the spatiotemporal sketch information of the target video based on the sketch representation information of the base video frame includes:

[0090] Determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video;

[0091] Predicting sketch key points in the target video frame according to the optical flow map;

[0092] Sketch representation information of the target video frame is generated according to the optical flow map, the mask of the moving target and the sketch key points, and the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

[0093] Specifically, the optical flow map can characterize the changes between video frames and be used to predict the sketch key points in the target video frame. The mask of the moving target can be determined based on the basic video frame and the target video frame, and the mask of the moving target is extracted.

[0094] In an embodiment of the present invention, the semantic communication method can obtain the sketch representation information of the target video frame of each sub-video based on the sketch representation information of the basic video frame using the optical flow map, and then generate the spatiotemporal sketch information of the target video.

[0095] Optionally, generating the sketch representation information of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points includes:

[0096] Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points;

[0097] Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch;

[0098] Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object;

[0099] Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch;

[0100] The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

[0101] Specifically, the optical flow-based method generates sketch representations of other video frames, which mainly includes: sketch key point prediction, sketch line cutting, mask extraction of moving targets, edge optical flow correction of moving targets, noise suppression and optical flow information compression.

[0102] Taking a basic video frame and a subsequent video frame as an example: feed the basic video frame and the subsequent video frame to a neural network to obtain an optical flow map between the basic video frame and the subsequent video frame, and a subset of the optical flow map can be used to predict key points in a sketch of the subsequent video frame; use the optical flow information to predict a sketch representation of the subsequent video frame; crop and perform optical flow correction on the sketch line segments between the background and the foreground, and the correction process is described as follows: assume that the moving target is a rigid body in a short period of time, calculate the average motion vector of each point in the moving target mask, and assign it to the point on the edge; perform optical flow noise suppression, and the suppression method is described as follows: suppose the complete optical flow distribution is F, the edge set of the mask is E, perform Gaussian smoothing on the regions of F and E, maintain the gradient of the optical flow at the edge of the mask, and then perform entropy coding, such as Huffman coding, to compress the optical flow information.

[0103] After obtaining the fourth sketch, the quality of the fourth sketch can be calculated, and then the fourth sketch can be optimized according to the quality of the sketch representation. The quality of the sketch representation can be determined according to the structural index. For the video frame I, it is assumed that in its ideal sketch S0, each pixel point p corresponds to the local maximum point of the gradient in I. Therefore, the structural similarity α between the sketch and the video frame I is determined along the average value of the gradient amplitude of each pixel point of the video frame I. For the video frame I, the structural similarity of the ideal sketch S0 is α0, and the similarity of the generated sketch Si is αi. The quality of the generated sketch Qi=αi / α0. The quality of the sketch can be used to evaluate the degree of fit between the generated sketch Si and the edge in the video frame I. The higher the degree of fit, the better the quality of the generated sketch Si. In addition, the image can be reconstructed according to the generated sketch, and the peak signal-to-noise ratio and structural similarity between the reconstructed image and the video frame are calculated to evaluate the quality of the sketch.

[0104] In an embodiment of the present invention, the semantic communication method can generate a sketch representation of the target video frame based on the optical flow method, and solve the problem of severe distortion of the optical flow at the edge of the moving target caused by the smoothing assumption in the optical flow method through cropping processing. By extracting the mask of the moving target, the original shape of the sketch line segment of the rigid target is better maintained, and denoising is performed through noise suppression processing of the optical flow, thereby improving the quality of the fourth sketch.

[0105] Optionally, the performing optimization processing based on the reconstructed video includes:

[0106] According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, optimization processing is performed with the goal of improving the evaluation quality and the semantic fidelity.

[0107] Specifically, the evaluation quality may be a user's subjective evaluation of the reconstructed video, which may be collected directly from the user, and the semantic fidelity may be obtained by obtaining each video frame in the reconstructed video and the corresponding video frame in the target video to calculate the semantic fidelity.

[0108] The semantic fidelity can be the semantic fidelity of the reconstructed video relative to the target video, which can be measured by semantic information, semantic mutual information, etc., and uncertainty mathematical theories such as fuzzy sets are applied. For example, the semantic features of the video frames of the reconstructed video and the semantic features of the video frames of the target video are mapped to fuzzy sets, and appropriate membership functions are selected to derive a general expression of semantic entropy. The semantic information measurement is achieved based on the semantic conditional entropy of semantic entropy and semantic mutual information. In addition, cognitive mechanisms or intelligent algorithms can be used to complete the mapping from local knowledge bases and shared knowledge bases to semantic space, and the semantic mutual information of the source and output of each local knowledge base is calculated to calculate the semantic fidelity of the reconstructed video relative to the target video.

[0109] In an embodiment of the present invention, the semantic communication method can improve the quality of the generated reconstructed video by optimizing the quality of the evaluation and the semantic fidelity of the reconstructed video based on the user's evaluation of the reconstructed video and the semantic fidelity of the reconstructed video, with the goal of improving the evaluation quality and the semantic fidelity.

[0110] Optionally, the performing optimization processing based on the reconstructed video includes:

[0111] Obtaining a first category of the reconstructed video and a second category of the target video;

[0112] An optimization process is performed according to the difference between the first category and the second category.

[0113] Specifically, the first category may be a predicted category of the reconstructed video, the second category may be a real category of the target video, and the optimization processing based on the difference between the first category and the second category may be optimized with the goal of reducing the difference between the first category and the second category. When the actual tasks to be processed are different, a recommendation result for the reconstructed video may also be obtained, and the optimization processing may be performed based on the recommendation result.

[0114] In an embodiment of the present invention, the semantic communication method can be optimized with the goal of reducing the difference between the first category and the second category, thereby improving the accuracy of the predicted category of the reconstructed video and being more suitable for downstream classification tasks.

[0115] Optionally, the performing semantic stream transmission on the spatiotemporal sketch information includes:

[0116] The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

[0117] Specifically, the spatiotemporal sketch information can be transmitted as a bit sequence using traditional methods, or a semantic transmission system can be used to effectively transmit semantic features. The bit stream transmission of the spatiotemporal sketch information can use an information entropy compression algorithm, and the semantic stream and the bit stream can be sent simultaneously. The semantic stream and the bit stream are transmitted through a completely shared frequency band. During the transmission, the semantic stream can be transmitted in an interference-free manner with the help of continuous interference elimination.

[0118] In the embodiment of the present invention, the semantic communication method can transmit semantic streams and bit streams through a completely shared frequency band to achieve higher spectrum efficiency, and because semantic stream transmission and bit transmission coexist, compatibility with traditional communications is improved.

[0119] See also Figure 3 and Figure 4 , Figure 3 and Figure 4is a schematic diagram of the structure of a semantic communication device provided by an embodiment of the present invention, such as Figure 3 As shown, the semantic communication device 300 includes a sending module 301, a network transmission module 302 and a receiving module 303;

[0120] The sending module 301 is used to generate spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base, wherein the spatiotemporal sketch information includes at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object;

[0121] The network transmission module 302 is used to perform semantic stream transmission on the spatiotemporal sketch information;

[0122] The receiving module 303 is used to generate a reconstructed video based on the spatiotemporal sketch information, and optimize the sending module, the network transmission module and the receiving module based on the reconstructed video.

[0123] Specifically, Figure 4 As shown, the network transmission module may include a cloud server and an edge server. A semantic knowledge base is deployed in the cloud server and maintained in the edge server to provide a semantic knowledge base for the sending module and the receiving module. The sending module and the receiving module can obtain the required personalized semantic knowledge base from the semantic knowledge base maintained by the network transmission module according to actual needs. The personalized semantic knowledge base is a part of the huge semantic knowledge base maintained by the network transmission module, and can also be called a semantic knowledge base. The personalized semantic knowledge base of the sending module is a semantic knowledge base related to the task performed by the sending module, and the personalized semantic knowledge base of the receiving module is a semantic knowledge base related to the task performed by the receiving module.

[0124] Optionally, the sending module 301 includes:

[0125] A cutting unit, used for cutting a target video into multiple sub-videos;

[0126] A first determining unit, configured to determine a base video frame for each of the plurality of sub-videos;

[0127] A second determining unit, configured to determine the sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame;

[0128] A generating unit is used to generate spatiotemporal sketch information of the target video based on the sketch representation information of the base video frame.

[0129] Optionally, the generating unit includes:

[0130] A determination subunit, configured to determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video;

[0131] A prediction subunit, configured to predict sketch key points in the target video frame according to the optical flow map;

[0132] A generating subunit is used to generate the sketch representation information of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points, wherein the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

[0133] Optionally, the generating subunit is specifically used for:

[0134] Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points;

[0135] Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch;

[0136] Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object;

[0137] Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch;

[0138] The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

[0139] Optionally, the receiving module 303 is specifically used for:

[0140] generating a reconstructed video based on the spatiotemporal sketch information;

[0141] According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, the sending module, the network transmission module and the receiving module are optimized with the goal of improving the evaluation quality and the semantic fidelity.

[0142] Optionally, the receiving module 303 is specifically used for:

[0143] generating a reconstructed video based on the spatiotemporal sketch information;

[0144] Obtaining a first category of the reconstructed video and a second category of the target video;

[0145] The sending module, the network transmission module and the receiving module are optimized according to the difference between the first category and the second category.

[0146] Optionally, the network transmission module 302 is specifically used for:

[0147] The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

[0148] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a communication device provided by an embodiment of the present invention, such as Figure 5 As shown, the sketch image generating device includes a bus 501, a transceiver 502, an antenna 503, a bus interface 504, a processor 505 and a memory 506:

[0149] The processor 505 is used to: generate spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base, wherein the spatiotemporal sketch information includes at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object;

[0150] Performing semantic streaming transmission on the spatiotemporal sketch information;

[0151] A reconstructed video is generated based on the spatiotemporal sketch information, and optimization processing is performed based on the reconstructed video.

[0152] Optionally, the processor 505 is specifically configured to:

[0153] Cut the target video into multiple sub-videos;

[0154] Determining a base video frame for each sub-video in the plurality of sub-videos;

[0155] Determine sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame;

[0156] Based on the sketch representation information of the base video frame, spatiotemporal sketch information of the target video is generated.

[0157] Optionally, the processor 505 is specifically configured to:

[0158] Determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video;

[0159] Predicting sketch key points in the target video frame according to the optical flow map;

[0160] Sketch representation information of the target video frame is generated according to the optical flow map, the mask of the moving target and the sketch key points, and the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

[0161] Optionally, the processor 505 is specifically configured to:

[0162] Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points;

[0163] Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch;

[0164] Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object;

[0165] Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch;

[0166] The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

[0167] Optionally, the processor 505 is specifically configured to:

[0168] According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, optimization processing is performed with the goal of improving the evaluation quality and the semantic fidelity.

[0169] Optionally, the processor 505 is specifically configured to:

[0170] Obtaining a first category of the reconstructed video and a second category of the target video;

[0171] An optimization process is performed according to the difference between the first category and the second category.

[0172] Optionally, the processor 505 is specifically configured to:

[0173] The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

[0174] exist Figure 5In the embodiment, the bus architecture (represented by bus 501) is shown, and bus 501 may include any number of interconnected buses and bridges, and bus 501 links various circuits including one or more processors represented by processor 505 and memory represented by memory 506. Bus 501 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 504 provides an interface between bus 501 and transceiver 502. Transceiver 502 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 505 is transmitted on a wireless medium via antenna 503, and further, antenna 503 also receives data and transmits the data to processor 505.

[0175] The processor 505 is responsible for managing the bus 501 and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 506 can be used to store data used by the processor 505 when performing operations.

[0176] Optionally, the processor 505 may be a CPU, an ASIC, an FPGA or a CPLD.

[0177] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the auxiliary device state control method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0178] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0180] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A semantic communication method, characterized in that: include: Based on the target video and the semantic knowledge base, generating spatiotemporal sketch information of the target video, the spatiotemporal sketch information including at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object; Performing semantic streaming transmission on the spatiotemporal sketch information; A reconstructed video is generated based on the spatiotemporal sketch information, and optimization processing is performed based on the reconstructed video.

2. The method according to claim 1, characterized in that: The step of generating spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base includes: Cut the target video into multiple sub-videos; Determining a base video frame for each sub-video in the plurality of sub-videos; Determine sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame; Based on the sketch representation information of the base video frame, spatiotemporal sketch information of the target video is generated.

3. The method according to claim 2, characterized in that The step of generating the spatiotemporal sketch information of the target video based on the sketch representation information of the base video frame comprises: Determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video; Predicting sketch key points in the target video frame according to the optical flow map; Sketch representation information of the target video frame is generated according to the optical flow map, the mask of the moving target and the sketch key points, and the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

4. The method according to claim 3, characterized in that Generating the sketch representation information of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points includes: Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points; Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch; Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object; Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch; The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

5. The method according to claim 1, characterized in that The optimizing process based on the reconstructed video includes: According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, optimization processing is performed with the goal of improving the evaluation quality and the semantic fidelity.

6. The method according to claim 1, characterized in that The optimizing process based on the reconstructed video includes: Obtaining a first category of the reconstructed video and a second category of the target video; An optimization process is performed according to the difference between the first category and the second category.

7. The method according to claim 1, characterized in that The performing semantic stream transmission on the spatiotemporal sketch information comprises: The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

8. A semantic communication device, characterized in that: The semantic communication device includes a sending module, a network transmission module and a receiving module; The sending module is used to generate spatiotemporal sketch information of the target video based on the target video and the semantic knowledge base, wherein the spatiotemporal sketch information includes at least one of the following: a target object in the target video, attribute information of the target object, and relationship information of the target object; The network transmission module is used to perform semantic stream transmission on the spatiotemporal sketch information; The receiving module is used to generate a reconstructed video based on the spatiotemporal sketch information, and optimize the sending module, the network transmission module and the receiving module based on the reconstructed video.

9. The device according to claim 8, characterized in that The sending module comprises: A cutting unit, used for cutting a target video into multiple sub-videos; A first determining unit, configured to determine a base video frame for each of the plurality of sub-videos; A second determining unit, configured to determine the sketch representation information of the basic video frame of each sub-video according to the semantic knowledge base and the basic video frame; A generating unit is used to generate spatiotemporal sketch information of the target video based on the sketch representation information of the base video frame.

10. The device according to claim 9, characterized in that The generating unit comprises: A determination subunit, configured to determine an optical flow map and a mask of a moving target according to the sketch representation information of the basic video frame and a target video frame, wherein the target video frame is a video frame other than the basic video frame in each sub-video; A prediction subunit, configured to predict sketch key points in the target video frame according to the optical flow map; A generating subunit is used to generate the sketch representation information of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points, wherein the spatiotemporal sketch information of the target video includes the sketch representation information of the target video frame.

11. The device according to claim 10, characterized in that The generating subunit is specifically used for: Generate a first sketch of the target video frame according to the optical flow map, the mask of the moving target and the sketch key points; Cutting the line segments of the first sketch according to the optical flow map to obtain a second sketch; Calculating an average motion vector of each point in the mask of the moving object according to the second sketch and the mask of the moving object; Assigning the average motion vector to points on the edge of the moving object to obtain a third sketch; The third sketch is subjected to optical flow noise suppression processing according to the optical flow map to obtain a fourth sketch, where the fourth sketch is sketch representation information of the target video frame.

12. The device according to claim 8, characterized in that The receiving module is specifically used for: generating a reconstructed video based on the spatiotemporal sketch information; According to the user's evaluation quality of the reconstructed video and the semantic fidelity of the reconstructed video, the sending module, the network transmission module and the receiving module are optimized with the goal of improving the evaluation quality and the semantic fidelity.

13. The device according to claim 8, characterized in that The receiving module is specifically used for: generating a reconstructed video based on the spatiotemporal sketch information; Obtaining a first category of the reconstructed video and a second category of the target video; The sending module, the network transmission module and the receiving module are optimized according to the difference between the first category and the second category.

14. The device according to claim 8, characterized in that The network transmission module is specifically used for: The spatiotemporal sketch information is subjected to semantic stream transmission and bit stream transmission.

15. A communication device, comprising: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: The processor is used to read the program in the memory to implement the steps in the method according to any one of claims 1 to 7.

16. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images