A cross-domain emergency command and dispatch method and system

Through cross-domain emergency command and dispatching methods and systems, the neural network model is used to compress the displacement data to video, which solves the problems of waste of resources and poor controllability in traditional technology, and realizes efficient interconnection and information resources of audio and video signals in multiple places shared.

CN118764649BActive Publication Date: 2024-11-15BEIJING LOIT TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411204640.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-15
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The interconnection and interoperability of audio and video signals in traditional remote command sites has problems of waste of resources and poor controllability, and it is impossible to share and arbitrary retrieval of information resources in multiple places.

Method used

Using cross-domain emergency command and dispatching methods and systems, through the combination of encoding equipment, decoding equipment and switches, the neural network model is used to compress the displacement data to generate displacement code streams, and cross-domain transmission is carried out through switches to realize multi-territorial sound. Interconnection of video signals.

Benefits of technology

It improves the flexibility, real-time and controllability of video encoding, reduces the complexity of video compression processing, avoids resource waste, and realizes information resource sharing in command sites in multiple places and the interconnection of audio and video signals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118764649B_ABST
    Figure CN118764649B_ABST
Patent Text Reader

Abstract

An embodiment of the present application discloses a cross-domain emergency command and dispatch method and system, which includes a switch, multiple encoding devices, multiple decoding devices and multiple display devices, the output port of each decoding device is connected to the corresponding display device, different decoding devices are deployed in different command locations, and each decoding device and the corresponding display device are deployed in the same command location; the encoding device processes multiple frames of displacement data to obtain multiple frames of first displacement data, and determines the size of the displacement video corresponding to the multiple frames of displacement data; the encoding device converts the multiple frames of first displacement data into a displacement video according to the size of the displacement video; the encoding device inputs the displacement video into a first neural network model, performs a first processing on the displacement video based on the first neural network model, and obtains a compressed video; the encoding device encodes the compressed video to obtain a displacement code stream, and sends the displacement code stream to the switch.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to a cross-domain emergency command and dispatch method and system. Background Art

[0002] With the continuous development and progress of society, various emergencies and disasters occur frequently, posing a serious threat to the safety of life and property of the people. Therefore, it is extremely important to establish and improve the emergency command system. As an important component, the emergency command site undertakes the tasks of emergency command, coordination, information release, control and early warning of emergencies. The application of information technology means such as information resource sharing and audio and video signal interconnection between multiple command sites in different locations can greatly enhance the functions of resource sharing, force coordination, site linkage, auxiliary support and off-site backup between multi-network, multi-level, multi-domain and multi-location command sites, improve the efficiency of emergency command sites in responding to emergency command and dispatch and disaster recovery and backup capabilities, and achieve better emergency management.

[0003] The traditional interconnection of audio and video signals in remote command sites uses point-to-point optical fiber transmission and point-to-point codec control. This method requires the use of multiple input and output ports of the core matrix of a command site to achieve lossless transmission of each signal, and can only achieve point-to-point transmission, and cannot achieve information resource sharing and arbitrary retrieval in multiple command sites. Therefore, the existing technology has the disadvantages of waste of resources and poor controllability. Summary of the invention

[0004] The embodiments of the present application provide a cross-domain emergency command and dispatch method and system, which can solve the defects of resource waste and poor controllability in the prior art.

[0005] In a first aspect, a cross-domain emergency command and dispatch method is provided, which is applied to a cross-domain emergency command and dispatch system including a switch, multiple encoding devices, multiple decoding devices, and multiple display devices, wherein the output port of each encoding device is connected to the input port of the switch, the input port of each decoding device is connected to the output port of the switch, and the output port of each decoding device is connected to the corresponding display device. Different decoding devices are deployed in different command locations, and each decoding device and the corresponding display device are deployed in the same command location. The method comprises the following steps:

[0006] The encoding device processes the multiple frames of displacement data to obtain multiple frames of first displacement data, and determines the size of the displacement video corresponding to the multiple frames of displacement data; the displacement data is used to store the distance information from the subdivided basic network vertex to the nearest neighbor point of the original input network in the dynamic mesh encoding process;

[0007] The encoding device converts the plurality of frames of first displacement data into a displacement video according to the size of the displacement video;

[0008] The encoding device inputs the displacement video into a first neural network model, and performs a first process on the displacement video based on the first neural network model to obtain a compressed video;

[0009] The encoding device encodes the compressed video to obtain a displacement code stream, and sends the displacement code stream to the switch;

[0010] Among them, the first neural network model is a neural network model with N video compression processing functions, N is an integer greater than 1; the video compression processing method corresponding to the first processing is determined by the first neural network model.

[0011] In a second aspect, a cross-domain emergency command and dispatch system is provided, comprising a switch, a plurality of encoding devices, a plurality of decoding devices, and a plurality of display devices, wherein the output port of each encoding device is connected to the input port of the switch, the input port of each decoding device is connected to the output port of the switch, the output port of each decoding device is connected to the corresponding display device, different decoding devices are deployed at different command locations, and each decoding device and the corresponding display device are deployed at the same command location;

[0012] The encoding device is used to process multiple frames of displacement data to obtain multiple frames of first displacement data and determine the size of the displacement video corresponding to the multiple frames of displacement data; the displacement data is used to store the distance information from the subdivided basic network vertex to the nearest neighbor point of the original input network during the dynamic mesh encoding process;

[0013] And, according to the size of the displacement video, converting multiple frames of first displacement data into a displacement video; inputting the displacement video into a first neural network model, performing a first processing on the displacement video based on the first neural network model to obtain a compressed video; encoding the compressed video to obtain a displacement code stream, and sending the displacement code stream to the switch;

[0014] Among them, the first neural network model is a neural network model with N video compression processing functions, N is an integer greater than 1; the video compression processing method corresponding to the first processing is determined by the first neural network model.

[0015] In an embodiment of the present application, multiple frames of displacement data are converted into displacement video, and the size of the displacement video is determined before generating the displacement video, thereby improving the flexibility, real-time and controllability of subsequent video encoding; a variety of video compression processing functions are realized through a unified neural network model, which is conducive to reducing the complexity of video compression processing and avoiding waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart showing the steps of the cross-domain emergency command and dispatch method provided in an embodiment of the present application;

[0017] Figure 2 A flowchart showing the steps of the method for generating a displacement information video provided by an embodiment of the present application;

[0018] Figure 3 A schematic diagram showing the principle of a method for generating a displacement information video provided in an embodiment of the present application;

[0019] Figure 4 A flowchart showing the steps of the video compression processing method provided by an embodiment of the present application;

[0020] Figure 5 A flowchart showing the steps of the decoding method provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram showing the principle of a decoding method provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram showing the application of the displacement information video generation method provided in an embodiment of the present application in a VDMC framework;

[0023] Figure 8 It represents an example of mesh simplification in the VDMC framework provided in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram showing the principle of a subgrid compression module in a VDMC framework provided in an embodiment of the present application;

[0025] Fig.10 A schematic diagram showing an application of the decoding method provided in an embodiment of the present application in a VDMC framework;

[0026] Fig.11 A schematic diagram showing the structure of a cross-domain emergency command and dispatch system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results, etc. in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result, etc. according to the judgment result.

[0030] With the rapid development of multimedia technology, related research results have been rapidly industrialized and have become an indispensable part of people's lives. Three-dimensional models have become a new generation of digital media after audio, images, and videos. Three-dimensional grids are a commonly used way to represent three-dimensional models. Compared with traditional multimedia such as images and videos, three-dimensional network models have stronger interactivity and realism, making them more and more widely used in various fields such as commerce, manufacturing, construction, education, medicine, entertainment, art, and military.

[0031] The Visual Volumetric Video-based Coding (V3C) standard provides a method for encoding and decoding various three-dimensional media through video or image coding technology. Specifically, it converts the three-dimensional media content from a three-dimensional representation to multiple two-dimensional representations (called V3C components) through projection and other methods before encoding, and then encodes the two-dimensional representation using existing video or image coding technology. V3C components mainly include occupancy components, geometric components, and attribute components. The occupancy component can indicate which areas in the two-dimensional representation are associated with the data of the three-dimensional representation; the geometric component represents information related to the position of the three-dimensional data in space, and the attribute component can provide attribute information corresponding to the vertex, such as material, texture, etc. In addition, the components also contain information on how to reconstruct the three-dimensional model through these components, which is called atlas information.

[0032] The atlas information is used to associate all components, and additional information for reconstructing from 2D back to 3D is also included in the atlas components. The atlas consists of multiple basic units, called patches. Each patch represents an area in the available 2D components and contains the information needed to project the area back to 3D space.

[0033] Video-based Dynamic Mesh Coding (VDMC) is a standard for compressing three-dimensional meshes. Its main idea is to compress three-dimensional meshes by using the existing V3C standard. Since three-dimensional meshes have connection information that needs to be encoded, its specific encoding process is slightly different from V3C. It is necessary to expand the syntax and semantics of the V3C standard decoding end and the decoding operation to support the decoding and reconstruction of three-dimensional meshes.

[0034] Triangular mesh is the most common representation of 3D networks. A 3D mesh can be considered as composed of three basic elements: vertices, edges, and faces. Vertices are the most basic elements in a mesh, and they define a position in a 3D space. Edges are line segments connecting two vertices in a mesh. Faces can be considered as polygons formed by closed paths of edges. For a triangular mesh, each face is a triangle.

[0035] The information contained in the mesh is usually divided into three categories: geometric information, connection information, and attribute information. Geometric information refers to the position of each vertex of the mesh in three-dimensional space. Connection information describes the association between elements in the mesh, that is, the connection relationship between vertices. Attribute information is optional, and it can associate attributes to corresponding mesh elements (such as vertex color, normal vector, etc. can be associated with mesh vertices). Mesh parameterization can also be used to map the mesh from three-dimensional space to a two-dimensional plane area. This mapping relationship is usually described by a set of parameter coordinates, called UV coordinates or texture coordinates, which are associated with mesh vertices. This two-dimensional mapping can be used to represent high-resolution attribute information, such as textures, normal vectors, etc.

[0036] In almost all application fields that use 3D meshes (such as computational simulation, entertainment, medical imaging, digital cultural relics, computer design, e-commerce, etc.), as people's demand for higher visual effects of 3D mesh models increases, the models are becoming more and more complex, and the accuracy of the models is also increasing. Therefore, the amount of data required to represent the 3D mesh is also increasing accordingly. The above problems have led to the increasing complexity of the processing, visualization, transmission and storage of 3D meshes. 3D mesh compression can be regarded as a way to solve the above problems. It reduces the size of model data and is conducive to the processing, storage and transmission of 3D meshes. Therefore, it is necessary to propose an efficient and general 3D mesh compression algorithm.

[0037] Recently, the International Organization for Standardization (MPEG) in the field of audio and video coding compression has begun to develop a compression standard VDMC for 3D meshes. This standard is specified based on the existing V3C standard. The V3C standard provides a general method for compressing 3D models, which can be presented in the form of point clouds, meshes or panoramic videos. Making the compression method of 3D mesh models compatible with this standard is conducive to the promotion and applicability of this method. Therefore, it is of great significance to optimize the 3D mesh encoding and decoding method in VDMC and combine the optimization method with the V3C standard. One possible optimization method is to optimize the encoding and decoding of the basic mesh in the frame. In the existing framework, the information of the basic mesh is divided into connection relationship, geometric information and attribute information for processing. For the geometric information encoding and decoding of the basic mesh of the 3D mesh encoding and decoding, a geometric information encoding and decoding that only depends on part of the connection relationship is proposed, which can support the parallel scheme of connection relationship and geometric information encoding and decoding. The encoding and decoding of connection relationship and geometric information can be completed in only one traversal, which can reduce the time complexity.

[0038] This embodiment provides an audio and video signal interconnection system for command sites in multiple locations based on a networked distributed system architecture. It adopts a networked basic architecture and a distributed system architecture, which exceeds the limitations of traditional video cable transmission distance and excessive port occupancy, forms a unified standard network interface, and forms a larger system network through the superposition of switches. Digitize local videos, standardize streaming media, and uniformly plan them all to access a networked distributed system, establish an information "resource pool" for multi-level and multi-domain command sites, and realize the interconnection, intercommunication, and mutual control of upper-layer applications and audio and video signals of different interfaces, different platforms, and different networks in command sites in various places. It has the excellent characteristics of easy deployment, easy expansion, and easy intercommunication. It has promoted the modernization of emergency management systems and capabilities, and improved the capabilities of emergency command, force coordination, multi-location linkage, resource sharing, and urban safety risk monitoring and early warning.

[0039] The cross-domain emergency command and dispatch method provided by the embodiment of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0040] like Figure 1 As shown, an embodiment of the present application provides a cross-domain emergency command and dispatch method, which is applied to a cross-domain emergency command and dispatch system including a switch, multiple encoding devices, multiple decoding devices and multiple display devices, wherein the output port of each encoding device is connected to the input port of the switch, the input port of each decoding device is connected to the output port of the switch, the output port of each decoding device is connected to the corresponding display device, different decoding devices are deployed in different command locations, and each decoding device and the corresponding display device are deployed in the same command location, and the method comprises the following steps:

[0041] Step 101: an encoding device processes multiple frames of displacement data to obtain multiple frames of first displacement data, and determines the size of the displacement video corresponding to the multiple frames of displacement data.

[0042] Among them, the displacement data is used to store the distance information from the subdivided basic network vertices to the nearest neighbor points of the original input network during the dynamic mesh encoding process.

[0043] Step 102: The encoding device converts a plurality of frames of first displacement data into a displacement video according to the size of the displacement video.

[0044] In step 103, the encoding device inputs the displacement video into a first neural network model, performs a first processing on the displacement video based on the first neural network model, and obtains a compressed video.

[0045] Step 104: The encoding device encodes the compressed video to obtain a displacement code stream, and sends the displacement code stream to the switch.

[0046] Among them, the first neural network model is a neural network model with N video compression processing functions, N is an integer greater than 1; the video compression processing method corresponding to the first processing is determined by the first neural network model.

[0047] In an embodiment of the present application, multiple frames of displacement data are converted into displacement video, and the size of the displacement video is determined before generating the displacement video, thereby improving the flexibility, real-time and controllability of subsequent video encoding; a variety of video compression processing functions are realized through a unified neural network model, which is conducive to reducing the complexity of video compression processing and avoiding waste of resources.

[0048] The embodiments of the present application adopt a networked basic architecture and a distributed system architecture, which goes beyond the limitations of traditional video cable transmission distances, forms a unified standard network interface, and forms a larger system network through the superposition of switches; digitizes local videos, standardizes streaming media, and uniformly plans and connects all of them to a networked distributed system, which can establish an information resource pool for multi-level and multi-domain command sites, and realize the interconnection, intercommunication, and mutual control of upper-layer applications and audio and video signals of different interfaces and platforms of command sites in various places, and has the excellent characteristics of easy deployment, easy expansion, and easy intercommunication; promotes the modernization of emergency management systems and capabilities, and improves the capabilities of emergency command, force coordination, multi-location linkage, resource sharing, and urban safety risk monitoring and early warning.

[0049] like Figure 2 As shown, the embodiment of the present application provides a method for generating a displacement information video, including:

[0050] Step 201, the encoding device processes multiple frames of displacement data to obtain multiple frames of first displacement data;

[0051] Step 202, the encoding device determines the size of the displacement video corresponding to the multiple frames of displacement data;

[0052] Step 203: The encoding device converts the multiple frames of first displacement data into a displacement video according to the size of the displacement video.

[0053] Among them, displacement data is an important part of the dynamic mesh encoding process, storing the distance information from the subdivided basic network vertices to the nearest neighbor points of the original input network.

[0054] In at least one embodiment of the present application, the method further includes:

[0055] The encoding device inputs the displacement video into a first neural network model, and performs a first process on the displacement video based on the first neural network model to obtain a compressed video;

[0056] The encoding device encodes the compressed video to obtain a displacement code stream, and sends the displacement code stream to the switch;

[0057] In one implementation, Figure 3 As shown in the figure, the pre-processing module processes the displacement data of multiple frames, such as prediction, transformation, quantization, etc. After pre-processing, multiple frames of first displacement data are obtained. After pre-processing, each frame of first displacement data needs to be input into the Packing module. Packing is a displacement image, and finally the multiple frames of displacement data are converted into displacement video. Before the Packing operation, the size of the displacement video needs to be determined first and stored in the buffer (Buffer), which is called by the Packing module. Finally, the generated displacement video is sent to the video encoder for encoding to obtain the displacement code stream. There are many options for specific video encoders, such as High Efficiency Video Coding (HEVC), Vertex Chain Code (VCC), etc.

[0058] As an optional embodiment, step 202 includes:

[0059] The encoding device specifies the size of the displacement video; for example, a user of the encoding device directly specifies the size of the displacement video;

[0060] Alternatively, step 202 includes:

[0061] The encoding device determines the size of the displacement video according to the size of the displacement image of the target frame; for example, the size of the displacement image of the target frame is used as the size of the shifted video.

[0062] In at least one embodiment of the present application, in the implementation process of using the size of the target frame displacement image as the size of the displacement video, it is necessary to first determine the size of the target frame displacement image. In one implementation, the method further includes:

[0063] The encoding device determines the size of the target frame displacement image according to the number of data blocks in each row of the displacement video, the size of the data blocks in the displacement video and the number of displacement data in the target frame.

[0064] It should be noted that after calculating the size of the displacement image of the current target frame, it is used as the size of the displacement video composed of all displacement images from the current target frame to the next target frame. After determining the size of the displacement video, it is stored in the Buffer; when encountering the next target frame, the size of the displacement video in the Buffer needs to be updated.

[0065] The encoding device determines the size of the target frame displacement image according to the number of data blocks in each row of the displacement video, the size of the data blocks in the displacement video and the number of displacement data of the target frame, including:

[0066] Determine the width of the target frame displacement image according to the number of each row of data blocks in the displacement video and the size of the data blocks in the displacement video;

[0067] The height of the target frame displacement image is determined according to the width of the target frame displacement image and the amount of target frame displacement data.

[0068] In an optional implementation, the number of data blocks in each line of the displacement video can be specified by a user or an encoding device; in another optional implementation, the size of the data block in the displacement video can be specified by a user or an encoding device.

[0069] Optionally, determining the width of the target frame displacement image according to the number of data blocks in each row of the displacement video and the size of the data blocks in the displacement video includes:

[0070] According to the first formula, the width of the target frame displacement image is determined; wherein the first formula is:

[0071]

[0072] in, Indicates the width of the target frame displacement image; Indicates the number of data blocks per line in the displacement video; Indicates the size of the data block in the displacement video.

[0073] Optionally, determining the height of the target frame displacement image according to the width of the target frame displacement image and the amount of target frame displacement data includes:

[0074] According to the second formula, the height of the target frame displacement image is determined; wherein the second formula is:

[0075]

[0076] in, Indicates the height of the target frame displacement image; Indicates the number of target frame displacement data; Indicates the width of the target frame displacement image; Indicates a round-up operation.

[0077] In at least one embodiment of the present application, step 203 includes:

[0078] The encoding device arranges each frame of first displacement data into a displacement image in a predetermined order according to the size of the displacement video; and multiple frames of displacement images constitute the displacement video.

[0079] In one implementation, Figure 3 As shown in the figure, after determining the size of the displacement video, it is stored in the Buffer (when encountering the next target frame, the size of the displacement video in the Buffer needs to be updated) for the Packing module. The Packing module sorts each frame of displacement data into a displacement image in a certain order, and finally converts it into a displacement video, and outputs possible related identifiers.

[0080] This application does not specify a specific arrangement order, it only needs to ensure that the codec end (i.e., the decoding device) is consistent. For example, the Morton order can be used for arrangement within the block, and the raster scanning order can be used between blocks. No specific limitation is made here.

[0081] In an optional embodiment of the present application, during the arrangement process, the method further includes:

[0082] If the amount of displacement data in the current frame is less than the size of the displacement video, the extra pixels are filled with default values; this application does not specify a specific filling value;

[0083] or,

[0084] If the amount of displacement data in the current frame is greater than the size of the displacement video, the redundant displacement data is removed.

[0085] The size of the displacement video may also be referred to as the number of pixels in one frame of the displacement video, and the number of pixels is equal to the width of the displacement image multiplied by the height of the displacement image.

[0086] In one implementation, before determining the target frame displacement image, it is necessary to first determine whether the current frame displacement data is the target frame displacement data. The method further includes:

[0087] According to a first method, it is determined whether the current frame displacement data is the target frame displacement data; wherein the first method includes at least one of the following:

[0088] Presetting the Mth frame displacement data as the target frame displacement data;

[0089] Divide the multiple frames of displacement data into multiple groups of pictures GOP in advance, and set the Nth frame displacement data of each GOP as the target frame displacement data;

[0090] Wherein, M and N are integers greater than or equal to 1 respectively.

[0091] For example, the first frame displacement data is used as the target frame displacement data; or a plurality of frame displacement data are divided into a plurality of GOPs, and the first frame displacement data of each GOP is used as the target frame displacement data.

[0092] In summary, in the embodiment of the present application, multiple frames of displacement data are converted into a displacement video, and the size of the displacement video is determined before the displacement video is generated, thereby improving the flexibility and real-time performance of subsequent video encoding.

[0093] Furthermore, the embodiment of the present application also provides a video compression processing method, such as Figure 4 As shown, the following steps are included:

[0094] Step 401: The encoding device inputs the displacement video into a first neural network model, where the first neural network model is a neural network model having N types of video compression processing functions, where N is an integer greater than 1;

[0095] Step 402: The encoding device performs a first process on the displacement video based on a first neural network model to obtain a compressed video.

[0096] The video compression processing mode corresponding to the first processing is determined by the first neural network model, and the displacement video is used as the input of the first neural network model. The first neural network model can perform the first processing on the displacement video based on the determined video compression processing mode to obtain a compressed video. The compressed video may include a filtered reconstructed block or a luminance reconstructed block of a first resolution, and the first resolution is higher than the resolution of the reconstructed block corresponding to the displacement video.

[0097] The first neural network model determines the video compression processing method corresponding to the first processing because the first neural network model has multiple (including two) video compression processing functions. The video compression processing function corresponding to the first processing can be any one of the N video compression processing functions.

[0098] Since the first neural network model has multiple video compression processing functions, the first neural network model can be used as a unified neural network model to implement multiple video compression processing functions. In this way, when multiple video compression processing is required for an image, it can be implemented through a unified neural network model, without the need to input the image multiple times into multiple single-function neural network models for processing, which is conducive to reducing the complexity of video compression processing.

[0099] In an embodiment of the present application, the displacement video is input into a first neural network model, and the displacement video is subjected to a first processing based on the first neural network model to obtain a compressed video. The first neural network model is a neural network model having N types of video compression processing functions, and the video compression processing method corresponding to the first processing is determined by the first neural network model. In this way, the functional diversity of the video compression processing neural network model can be improved, so that a variety of video compression processing functions can be realized through a unified neural network model, which is conducive to reducing the complexity of video compression processing.

[0100] It should be noted that the video compression processing method provided in the embodiment of the present application is applicable to both the model application stage (also called the model inference stage) and the model training stage, and the embodiment of the present application does not limit this.

[0101] Optionally, the N video compression processing functions include at least two of a loop filtering function, a super-resolution function and a post-processing filtering function;

[0102] The video compression processing method corresponding to the first processing includes one of a loop filtering processing method, a super-resolution processing method and a post-processing filtering processing method.

[0103] For the loop filtering processing method, the sizes of the input image and the output image (size in the width direction and size in the height direction) are the same.

[0104] Exemplarily, when performing loop filtering processing as a loop filtering model, the sizes (size in the width direction and size in the height direction) of the source image, the input image, and the output image are the same.

[0105] Exemplarily, when loop filtering is performed as a loop filter 2x model, the source image is twice the size of the input image (the size in the width direction and the size in the height direction), and the size of the input image and the output image (the size in the width direction and the size in the height direction) are the same.

[0106] It should be noted that the input image and the output image are respectively the input and output of the video compression processing method.

[0107] The process of loop filtering based on the first neural network model may be: first, the first luminance data and the first chrominance data are subjected to feature extraction through the convolution layer and activation layer of the first neural network model (it should be noted that each convolution layer may correspond to an activation layer, such as the activation layer may be a parametric rectified linear unit (PReLU)) to generate K feature maps, where K is a positive integer. Next, these feature maps are subjected to feature fusion through the convolution layer and the activation layer to generate S fused feature maps, where S is a positive integer. Then, these fused feature maps are subjected to feature enhancement through multiple residual blocks to generate L enhanced feature maps, where L is a positive integer. Finally, these enhanced feature maps are subjected to operations such as convolution layers and activation layers to generate filtered reconstructed image blocks.

[0108] There may be multiple super-resolution processing multiples, and different super-resolution multiples may be used as different super-resolution processing modes.

[0109] Exemplarily, when super-resolution processing is performed as a super-resolution 1.5x model, the source image is 1.5 times the size of the input image (the size in the width direction and the size in the height direction), and the output image is 1.5 times the size of the input image (the size in the width direction and the size in the height direction).

[0110] Exemplarily, when super-resolution processing is performed as a super-resolution 2x model, the source image is twice the size of the input image (the size in the width direction and the size in the height direction), and the output image is twice the size of the input image (the size in the width direction and the size in the height direction).

[0111] Super-resolution processing can also be called super-resolution reconstruction processing. The process of super-resolution reconstruction processing based on the first neural network model can be: first, the first luminance data and the first chrominance data are subjected to feature extraction through the convolution layer and activation layer of the first neural network model to generate T feature maps, where T is a positive integer. Next, these feature maps are subjected to feature fusion through the convolution layer and activation layer to generate M fused feature maps, where M is a positive integer. Then, these fused feature maps are subjected to feature enhancement through multiple residual blocks to generate P enhanced feature maps, where P is a positive integer. Finally, these enhanced feature maps are subjected to operations such as convolution layers, activation layers, and upsampling to generate high-resolution image blocks.

[0112] The post-processing filtering method is roughly the same as the loop filtering method. The process of post-processing filtering based on the first neural network model can refer to the above-mentioned loop filtering process. In order to avoid repetition, this will not be described in detail. It should be noted that the above-mentioned video compression processing functions and various video compression processing methods are only examples. When there are other video compression processing functions, the first neural network model can continue to expand other video compression processing functions, and the embodiments of the present application are not limited to this.

[0113] In an embodiment of the present application, the first neural network model determines the video compression processing method corresponding to the first processing, which may include multiple implementation methods, which are described below respectively.

[0114] In some embodiments, the method further comprises:

[0115] Inputting target information into the first neural network model, the target information being information for indicating a video compression processing method;

[0116] Among them, the video compression processing method corresponding to the first processing is determined by the first neural network model based on the target information.

[0117] In this embodiment, in addition to inputting the displacement video into the first neural network model, the target information is also input into the first neural network model. In other words, the input data of the first neural network model includes the target information and the displacement video.

[0118] The target information can be understood as function indication information, which is used to indicate the video compression processing function. That is, in this embodiment, the first neural network model determines the video compression processing method corresponding to the first processing based on the target information.

[0119] Exemplarily, the target information is a function identifier.

[0120] Optionally, the target information includes at least one of the following:

[0121] A first identifier, wherein an index value of the first identifier is mapped to a corresponding video compression processing method;

[0122] The second identifier includes a multiple of the image width direction and a multiple of the image height direction.

[0123] Exemplarily, the first identifier may be an index value such as 0, 1, 2, 3, etc.

[0124] The meaning of each index value may be, for example:

[0125] Label 0: loop filter model;

[0126] Logo 1: Super-resolution 2x model;

[0127] Logo 2: Super-resolution 1.5x model;

[0128] Label 3: Post-processing filter model.

[0129] If the first identifier is 0, the first neural network model is used as a loop filter model to perform filtering processing.

[0130] If the first flag is 1, the first neural network model is used as a super-resolution 2x model for super-resolution processing.

[0131] If the first identifier is 2, the first neural network model is used as a super-resolution 1.5x model for super-resolution processing.

[0132] If the first identifier is 3, the first neural network model is used as a post-processing filter model.

[0133] The meaning of each index value identifier may also be, for example:

[0134] Label 0: loop filter model;

[0135] Identification 1: Loop filter 2x model;

[0136] Logo 2: Super-resolution 2x model;

[0137] Label 3: Post-processing filter model.

[0138] If the first identifier is 0, the first neural network model is used as a loop filter model to perform filtering processing.

[0139] If the first flag is 1, the first neural network model is used as a loop filter 2x model for filtering.

[0140] If the first identifier is 2, the first neural network model is used as a super-resolution 2x model for super-resolution processing, and the magnification is 2 times.

[0141] If the first identifier is 3, the first neural network model is used as a post-processing filter model.

[0142] The target information is relatively concise using the above-mentioned first identification method, which enables the first neural network model to directly determine the video compression processing method.

[0143] During training, the first neural network model can learn the functions of the model by itself to improve the generalization of the model.

[0144] Exemplarily, the second identifier may be in a data format such as (1, 1), (2, 2), (1.5, 1.5), etc.

[0145] If the second identifier is (1, 1), indicating that the multiples in the image width direction and the image height direction are both 1, the first neural network model can be used as a loop filtering model for filtering processing.

[0146] If the second identifier is (2, 2), indicating that the multiples in the image width direction and the image height direction are both 2, the first neural network model can be used as a super-resolution 2x model for super-resolution processing.

[0147] If the second identifier is (1.5, 1.5), indicating that the multiples in the image width direction and the image height direction are both 1.5, the first neural network model can be used as a super-resolution 1.5x model for super-resolution processing.

[0148] The target information adopts the above-mentioned second identification method which is more flexible and can provide effective input information for the first neural network model.

[0149] In this implementation, target information is used to assist the first neural network model in determining a video compression processing method. This method is simple and efficient and can improve the performance of the first neural network model.

[0150] like Figure 5 As shown, the embodiment of the present application also provides a decoding method, including:

[0151] Step 501: The decoding device receives the displacement code stream through the switch and decodes the displacement code stream into a compressed video.

[0152] Step 502: The decoding device inputs the compressed video into a second neural network model, performs a second process on the compressed video based on the second neural network model, and obtains the displacement video.

[0153] Among them, the second neural network model is a neural network model with N video decompression processing functions, N is an integer greater than 1; the video decompression processing method corresponding to the second processing is determined by the second neural network model.

[0154] Step 503: The decoding device extracts pixel values ​​from each displacement image frame of the displacement video in a target order to obtain target data; the target order is the same as the order in which the displacement images are arranged by the encoding device.

[0155] Step 504: The decoding device processes the extracted target data to obtain multiple frames of displacement data.

[0156] Step 505: The display device displays corresponding information resources and audio and video signals according to the multiple frames of displacement data.

[0157] In one implementation, Figure 6As shown, at the decoding end, the displacement code stream is first decoded into a displacement video by a video decoder corresponding to the video encoder on the encoding device side. The displacement video is then input to the unpacking module, which uses the same arrangement order as the encoding end to extract pixel values ​​from each frame of the displacement video according to possible related identifiers. Finally, the extracted data is input to the post-processing module to restore the displacement data. The operation of the post-processing module is the reverse implementation of the operation of the pre-processing module of the encoding device, such as inverse quantization, inverse transformation, prediction, etc.

[0158] In at least one embodiment of the present application, step 503 includes:

[0159] For a frame of displacement image, determining the number of pixel values ​​that need to be extracted from the displacement image according to the number of vertices of the subdivided reconstructed basic mesh corresponding to the displacement image;

[0160] If the number of vertices of the reconstructed basic mesh after subdivision is less than or equal to the number of pixel values ​​required to be extracted from the displacement image, extracting pixel values ​​equal to the number of vertices of the reconstructed basic mesh after subdivision from the displacement image in the target order to obtain target data;

[0161] Alternatively, if the number of vertices of the basic mesh reconstructed after subdivision is greater than the number of pixel values ​​that need to be extracted from the displacement image, all pixel values ​​are extracted from the displacement image in the target order, and a padding operation is performed at the end of the extracted data until the number of data is equal to the number of vertices of the basic mesh reconstructed after subdivision, thereby obtaining the target data.

[0162] Optionally, the filled value can be determined in a variety of ways, such as being equal to 0, equal to the average of the extracted pixel values, etc., which is not specifically limited here.

[0163] In summary, for the solution of converting multiple frames of displacement data into a displacement video and determining the size of the displacement video before generating the displacement video, the embodiment of the present application provides a corresponding decoding method, thereby improving the decoding accuracy.

[0164] The application of the coding device side solution proposed in this application at the coding end of the VDMC framework is mainly in the process of shifting the "two-dimensional arrangement", such as Figure 7 As shown. Next, the entire application process is described in detail, as follows Figure 7 shown.

[0165] (1) Mesh simplification

[0166] Mesh simplification is to simplify the current input mesh into a basic mesh with relatively few points and faces, and keep the shape of the original mesh as much as possible. The focus of mesh simplification is the simplification operation and the corresponding error metric. A feasible mesh simplification operation is shown in Figure 8, which merges the vertices at both ends of the edge into one vertex and deletes the connection between the two vertices. Repeat this process in the entire mesh according to certain rules to reduce the number of faces and vertices of the mesh to the target value.

[0167] During the simplification process, you can select an error metric to optimize the simplified result. For example, you can select the sum of the coefficients of the equations of all adjacent faces of a vertex as the error metric of the vertex, and the error metric of the corresponding edge is the sum of the error metrics of the two vertices on the edge. In other words, the error generated by merging an edge is the sum of the distances of all adjacent planes from the merged vertex to the original two vertices of the edge.

[0168] After determining the simplification operation and the corresponding error metric, the mesh simplification process begins iteratively. First, the vertex error of the initial mesh is calculated to obtain the error of each edge. Then each edge is arranged from small to large error, and the edge with the smallest error is selected for merging each time. At the same time, the position of the merged vertex is calculated, and the error of all edges related to the merged vertex is updated. That is, the order of edge arrangement is updated to ensure that each iteration is based on the global error metric. Through iteration, the mesh faces are simplified to the number required to meet lossy coding.

[0169] (2) Mesh parameterization

[0170] Texture coordinates may be regenerated based on the reconstructed base mesh of the current frame. This step requires generating texture coordinates for each attribute map of the input mesh. If multiple attribute maps have similar characteristics, they can share the same texture coordinates. There are many ways to generate texture coordinates, including mesh parameterization. There are many algorithms for parameterizing meshes, such as the Isocharts algorithm, which uses spectral analysis to achieve stretch-driven 3D mesh parameterization, UV-unwrapping, slicing, and packing the 3D mesh into a 2D texture domain.

[0171] (3) Subdivision deformation

[0172] The subdivision and deformation modules are optional modules that can improve the quality of the mesh reconstructed at the decoding end. This module can be omitted when the quality of the basic mesh can meet the application requirements.

[0173] The same concepts of the basic idea of ​​the subdivision and deformation modules are applied to the input 3D mesh to generate displacement vector information. The input 2D curve (represented by a 2D polyline), called the "original" curve, is first downsampled to generate the basic curve / polyline, called the "simplified" curve. The subdivision scheme is then applied to the simplified polyline to generate the "subdivided" curve. The subdivided polyline is then deformed to get a better approximation of the original curve. That is, a geometric displacement vector is calculated for each vertex of the subdivided mesh so that the shape of the subdivided curve is as close to the shape of the original curve as possible. These geometric displacement vectors are the geometric displacement vector information (vertex displacement) output by this module. The same deformation process is also applied to the attribute information corresponding to the vertex to obtain the corresponding attribute displacement vector.

[0174] (4) Compression of base mesh

[0175] The basic mesh compression module subdivides the basic mesh information output by the deformation module. There are two main modes of basic mesh compression, namely intra-frame mode and inter-frame mode, as shown in Figure 9. In intra-frame mode, the input of the basic mesh compression module is a three-dimensional mesh, which contains geometric coordinates, connection relationships, and attribute information associated with vertices. In inter-frame mode, the input of the basic mesh compression module is the motion vector and its identifier and possible intra-frame sub-meshes. After encoding, the encoded mesh needs to be reconstructed so that it can be provided to subsequent modules for processing. The following is a brief introduction to the encoding of the two modes:

[0176] ①Intra-frame mode

[0177] The grid coding in intra mode can use the existing grid encoder to encode the input base grid, such as Draco, etc. The grid encoder type is encoded with auxiliary information and passed to the decoding end.

[0178] ②Inter-frame mode

[0179] In inter-frame mode, motion vectors and identifiers need to be encoded. The motion vector identifier is an array of numbers greater than or equal to 0. The array size is the same as the number of vertices of the base grid of the reference frame corresponding to the temporal motion vector identifier. The array can be directly encoded using an existing entropy coding algorithm, such as CABAC (context-adaptive binary arithmetic coding).

[0180] (5) Reconstructing the base mesh

[0181] After the base grid is encoded, the base grid needs to be reconstructed. For a grid encoded in intra-frame mode, the encoded grid can be decoded to obtain a reconstructed grid.

[0182] For the basic grid information encoded in the inter-frame mode, it is necessary to reconstruct the inter-frame sub-grid based on the motion vector and the identifier. If there is an intra-frame sub-grid in the inter-frame mode, it is also necessary to merge the reconstructed inter-frame sub-grid with the intra-frame sub-grid to obtain the reconstructed basic grid.

[0183] (6) Encoding of vertex displacement

[0184] To encode vertex displacement, first adjust the order of vertex displacement according to the reconstructed basic mesh; then perform wavelet transform and quantization on the displacement to obtain quantized wavelet transform coefficients; then arrange the quantized wavelet transform coefficients into a two-dimensional image, and finally form a YUV video, which is encoded with the help of a video encoder to obtain the displacement code stream. The following is a detailed introduction to each module:

[0185] ①Adjust the displacement order

[0186] The main function of this module is to adjust the order of vertex displacement according to the reconstructed base mesh. The detailed process is: first, the reconstructed base mesh is subdivided, and then the displacement order is adjusted to be the same as the subdivided base mesh vertex order. There is also an optional step in this process, which is to convert the coordinate system of vertex displacement from the Cartesian coordinate system to the local coordinate system.

[0187] ② Wavelet transform

[0188] A transformation can be applied to the displacement vector to reduce the correlation between its data. An optional transformation is a linear wavelet transform, and its prediction process is defined as shown in formula (3):

[0189]

[0190] in, For edge The newly inserted midpoint on , as well as Vertex , and The corresponding displacement vector. After the displacement vector is predicted, it is updated. The update process is defined as shown in formula (4):

[0191]

[0192] in Vertex The set of all adjacent vertices of . The transformed displacement vector is called the wavelet coefficient.

[0193] ③ Coefficient quantization

[0194] The transformed displacement vector, i.e., the wavelet coefficient, can be quantized. There are many ways to quantize it. One method is shown in equations (5) and (6):

[0195]

[0196]

[0197] in, Indicates The transformed value of the vertex displacement vector, The displacement vector values, Indicates rounding down. The bit depth representing the current mesh vertex geometry position, Indicates As mentioned above, after the displacement vector is converted into a coordinate system, its normal component has a more significant effect on the quality than the tangential component, so a larger quantization parameter can be used for the tangential component.

[0198] At the same time, according to the characteristics of wavelet transform, different quantization parameters can be used for the newly generated vertices and the original vertices. That is, for the subdivided vertices, the quantization parameter update is shown in formula (7):

[0199]

[0200] in, Represents the coefficient of the quantization parameter of the current subdivision level.

[0201] ④Two-dimensional arrangement

[0202] After quantization, the quantized wavelet transform coefficients need to be arranged on a two-dimensional image. Before performing two-dimensional arrangement, the size of the arranged video needs to be determined first. The process of determining the video size is as follows:

[0203] First, determine whether the quantized wavelet transform coefficients of the current frame are the selected frame (also called the target frame). There are many ways to determine the selected frame. For example, the quantized wavelet transform coefficients of the first frame are used as the selected frame; or the quantized wavelet transform coefficients of multiple frames are divided into multiple GOPs, and the quantized wavelet transform coefficients of the first frame of each GOP are used as the selected frame; or the quantized wavelet transform coefficients of each I frame are used as the selected frame, etc. If the quantized wavelet transform coefficients of the current frame are the selected frame, it is necessary to calculate the size of the selected frame arrangement image. The calculation process is shown in equations (8), (9) and (10):

[0204]

[0205]

[0206]

[0207] in Represents the width of the array image; Represents the height of the arranged image; Represents the number of blocks in each row of the arranged video (this value can be specified by the user); Represents the size of the block in the arranged video (this value can be specified by the user); Represents the number of coefficients contained in the i-th LOD level of the wavelet transform coefficients after quantization of the selected frame; Represents a round-up operation; Represents the number of blocks required to arrange the coefficients in the i-th LOD level of the wavelet transform coefficients after quantization of the selected frame into a two-dimensional image; n represents the number of LOD levels, Represents a total of n+1 levels. After calculating the size of the arranged image of the currently selected frame, use it as the size of the video composed of all the arranged images from the currently selected frame to the next selected frame. After determining the size of the arranged video, store it in the Buffer (when the next selected frame is encountered, the video size in the Buffer needs to be updated) for the two-dimensional arrangement module. The two-dimensional arrangement module arranges the quantized wavelet transform coefficients of each frame into a two-dimensional image in a certain order. The embodiment of the present application does not specify a specific arrangement order. It only needs to ensure that the codec is consistent. For example, the Morton order can be used for arrangement within the block, and the raster scanning order can be used between blocks. During the arrangement process, a feasible arrangement method is: if the sum of the number of blocks required for each LOD level of the quantized wavelet transform coefficients of the current frame is less than or equal to the number of blocks of one frame of the arranged video (the number of blocks is equal to ), the pixels in the extra blocks are filled with default values ​​(this patent does not specify the specific filling value); if the total number of blocks required for each LOD level of the quantized wavelet transform coefficients of the current frame is greater than the number of blocks for arranging a frame of video, the redundant displacement data is removed. At the same time, the module outputs a flag to indicate whether the solution proposed in this patent is enabled.

[0208] ⑤Video Compression

[0209] After arranging the wavelet coefficients on a two-dimensional image, they can be directly encoded using a video encoder to obtain a displacement code stream.

[0210] (7) Reconstruction of deformed mesh

[0211] The displacement vector encoding module needs to obtain the reconstructed displacement vector value during encoding. First, the decoded and reconstructed video is obtained from the video encoder. Then, the pixel value is extracted from each frame of the decoded and reconstructed video in the same order as the two-dimensional arrangement order. If the flag indicator output by the two-dimensional arrangement module indicates that the scheme proposed in this patent is turned on, a feasible extraction method is: for each frame of the image to be extracted, the number of pixel values ​​to be extracted is determined according to the number of vertices of each LOD level of the corresponding subdivided and reconstructed basic grid; when the sum of the number of blocks required for the number of vertices of each LOD level of the subdivided and reconstructed basic grid is less than or equal to the number of blocks of the displacement image to be extracted, the same order as the arrangement order of the encoding end is used to extract the same pixel values ​​as the number of vertices of the subdivided and reconstructed basic grid from the image to be extracted; when the sum of the number of blocks required for the number of vertices of each LOD level of the subdivided and reconstructed basic grid is greater than the number of blocks of the image to be extracted, all non-filled pixel values ​​in the image to be extracted are extracted in the same order as the arrangement order of the encoding end, and a padding operation is performed at the end of the extracted data until the number of data is equal to the number of vertices of the subdivided and reconstructed basic grid. The filled value can be determined in many ways, such as equal to 0, equal to the mean of the extracted pixel values, and so on. If the flag output by the two-dimensional arrangement module indicates that the solution proposed by this patent is not enabled, then the same order as the two-dimensional arrangement order is directly used to extract the same number of pixel values ​​as the number of vertices of the reconstructed basic mesh after subdivision from each frame of the decoded reconstructed video. Finally, the extracted data is inversely quantized and inversely transformed to obtain a displacement vector consistent with the decoding end. After obtaining the reconstructed geometric displacement vector, the subdivided sub-mesh is reconstructed according to the corresponding geometric displacement vector to obtain the subdivided deformed mesh, which is passed to the attribute graph conversion module.

[0212] (8) Texture map conversion

[0213] The texture map conversion module performs texture map conversion according to the input original mesh, the input original texture map and the reconstructed deformed mesh. The texture map can also be called an attribute map.

[0214] The steps for attribute graph conversion are as follows:

[0215] Calculate the texture coordinates of each pixel on the attribute map to be generated, such as the texture coordinates corresponding to pixel A(i, j) is P(u, v).

[0216] Determine whether the texture coordinate is within a certain triangle face after the subdivision deformation mesh is parameterized.

[0217] If the texture coordinate does not belong to any triangle, the pixel is marked as an empty pixel and can be filled with a filling algorithm later.

[0218] If the texture coordinate belongs to a triangle, then

[0219] Mark the pixel as filled.

[0220] Calculate the center of gravity coordinates of the texture in the current triangle face based on the texture coordinates.

[0221] According to the barycentric coordinates and the corresponding triangular face, the two-dimensional texture coordinates are mapped to three-dimensional geometric coordinates, that is, mapped to the points on the subdivided deformation grid corresponding to the texture coordinates, as shown by M(x, y, z) in the figure.

[0222] Find the point closest to the three-dimensional coordinate on the input original grid, as shown in the figure (x, y, z).

[0223] The three-dimensional coordinates are calculated according to the triangular face where they are located, and their center coordinates are mapped to two dimensions to calculate their texture coordinates, that is, .

[0224] The texture coordinates are used to sample the original attribute map input to obtain the value of the corresponding pixel position. .

[0225] Assign this value to the corresponding pixel A(i, j) on the attribute map to be generated.

[0226] (9) Texture image compression

[0227] After obtaining the converted attribute map, the empty pixels therein can be filled using existing filling algorithms (such as the Push-Pull algorithm). Then, existing video encoders such as H.264 / AVC, H.265 / HEVC, and H.266 / VVC can be used to encode the empty pixels to obtain the bitstream of the output attribute map. In addition, color space conversion and chroma subsampling can be selectively applied to obtain better rate-distortion performance for video encoding, such as color space conversion from RGB 444 to YUV420. In the case of multiple attribute maps, the texture coordinates corresponding to each attribute map need to be identified by auxiliary information during encoding.

[0228] (10) Auxiliary information

[0229] During the encoding process, there are some alternative schemes in each module, such as the type of grid encoder, the type of video encoder, the grid subdivision scheme, the spatial displacement vector transformation scheme and the transformation parameters, etc. The proposed framework allows the use of different schemes. Therefore, the selected scheme needs to be passed to the decoding end to guide the correct decoding.

[0230] The auxiliary information includes the type of static grid encoder, the type of video encoder, the grid subdivision scheme, the number of grid iterations, the geometric displacement vector transformation scheme and transformation parameters, the coefficient arrangement scheme, and the color conversion scheme, etc. For the grid part encoded using the inter-frame coding mode, it also includes the corresponding reference frame list, etc.

[0231] After all modules are encoded, the basic grid part bitstream, texture coordinate part bitstream, displacement vector video bitstream, attribute map video bitstream and auxiliary information bitstream are mixed to obtain the encoded bitstream finally output by the encoding end.

[0232] The decoding device side solution proposed in this application is mainly applied in the decoding end of the VDMC framework in the process of "displacement decoding", such as Fig.10 As shown. Next, the entire application process is described in detail, as follows Fig.10 shown.

[0233] (1) Auxiliary information decoding

[0234] The auxiliary information includes the type of static grid encoder, the type of video encoder, grid subdivision scheme, grid iteration number, geometric displacement vector transformation scheme, coefficient arrangement scheme, displacement truncation mark, and color conversion scheme, etc. For the grid part encoded in inter-frame coding mode, it also includes the corresponding reference frame list, etc. It can guide the decoder to perform correct decoding, and the information is represented by the corresponding syntax parameters after decoding.

[0235] (2) Basic grid decoding

[0236] The decoding of the base grid can be divided into intra-frame mode and inter-frame mode. This module is only responsible for decoding the information representing the base grid. That is, for the intra-frame mode, the output of the module decoding is a three-dimensional grid, which contains geometric information and connection relationships, as well as possible attribute information and texture coordinate information. For the inter-frame mode, this module is responsible for decoding motion vectors and identifiers and possible intra-frame sub-grids, that is, the output information is motion vectors, identifiers, and intra-frame sub-grids.

[0237] ①Intra-frame mode

[0238] The decoding of the basic grid information in the intra-frame mode is also the decoding of the basic grid. This module uses the corresponding grid decoder to decode the basic grid through the static grid decoder type indicated by the auxiliary information.

[0239] ②Inter-frame mode

[0240] In inter-frame mode, it is necessary to decode motion vectors, identifiers, and possible intra-frame sub-grids. For intra-frame sub-grids, the grid decoder indicated by the auxiliary information can be used for decoding. The motion vector identifier is obtained by entropy decoding, and the motion vector is obtained by deprediction and dequantization after entropy decoding.

[0241] (3) Segmentation

[0242] The subdivision module operation is the same as the encoding end subdivision operation, and the auxiliary information is used to indicate the number of subdivision iterations of different areas of the current basic grid.

[0243] (4) Displacement vector video decoding

[0244] The displacement vector video bitstream is decoded according to the video decoder type indicated in the auxiliary information. Note that this includes geometric displacement vector video and possible displacement vector video with different attributes.

[0245] (5) Displacement vector decoding

[0246] This module is responsible for restoring the decoded displacement vector image to a displacement vector. First, it is necessary to extract the image value from the displacement vector image. If the flag indicator output by the two-dimensional arrangement module of the encoding end to the decoding end indicates that the scheme proposed by this patent is turned on, then a feasible extraction method is: for each frame of the image to be extracted, the number of pixel values ​​to be extracted is determined according to the number of vertices of each LOD level of the corresponding subdivided reconstructed basic grid; when the sum of the number of blocks required for the number of vertices of each LOD level of the subdivided reconstructed basic grid is less than or equal to the number of blocks of the displacement image to be extracted, the same pixel value as the number of vertices of the subdivided reconstructed basic grid is extracted from the image to be extracted in the same order as the arrangement order of the encoding end; when the sum of the number of blocks required for the number of vertices of each LOD level of the subdivided reconstructed basic grid is greater than the number of blocks of the image to be extracted, all non-filled pixel values ​​in the image to be extracted are extracted in the same order as the arrangement order of the encoding end, and the padding operation is performed at the end of the extracted data until the number of data is equal to the number of vertices of the subdivided reconstructed basic grid. There are many ways to determine the value of the padding, such as equal to 0, equal to the mean of the extracted pixel values, etc. If the flag indication outputted by the encoding end two-dimensional arrangement module to the decoding end indicates that the solution proposed in this patent is not enabled, the same sequence as the two-dimensional arrangement sequence is directly used to extract the same number of pixel values ​​as the number of vertices of the subdivided reconstructed basic mesh from each frame of the decoded reconstructed video. Then, the extracted data is subjected to inverse quantization, inverse transformation and other operations to restore the corresponding displacement vector.

[0247] (6) Reconstructing the deformed mesh

[0248] After the base mesh and displacement vector are decoded and reconstructed, the deformed mesh is reconstructed based on the two parts. The corresponding displacement vector is added to each vertex of the subdivided mesh, as shown in formula (11):

[0249]

[0250] in, That is, the current frame (index is ) The first subdivision of the base mesh The geometric coordinates of the vertices, That is The spatial displacement vector corresponding to each vertex is That is, the current frame after subdivision and deformation The geometric coordinates of the vertices.

[0251] The attribute displacement vector is applied to the corresponding attribute value in the same way, taking texture coordinates as an example, as shown in equation (12).

[0252]

[0253] in, Indicates the number of times the base grid of the current frame is subdivided. k texture coordinates, Represents the attribute displacement vector k values, This means that the current frame is subdivided and deformed. k The value of the texture coordinate.

[0254] (7) Attribute graph decoding

[0255] The attribute graph decoder is responsible for decoding the attribute graph bitstream. The attribute graph is decoded using the video decoder indicated in the auxiliary information. An optional color space conversion is performed on it to obtain the attribute graph that matches the input attribute graph of the encoder. Figure 1 The image format is consistent, and the attribute map of the final decoded output is obtained. For multiple attribute map bitstreams, after decoding, each attribute map is matched with the corresponding texture coordinates according to the identifier.

[0256] After each module is processed, the deformed mesh and the corresponding attribute map reconstructed by the decoder are finally obtained. Subsequent applications use the reconstructed deformed mesh and attribute map as input for processing.

[0257] In an optional implementation, when the scheme on the encoding device side and the method on the decoding device side proposed in this application are applied to the VDMC framework, the syntax structure corresponding to the VDMC framework needs to be enhanced. For example, the parameter "asve_custom_displacement_video_size_flag" is added to the syntax structure corresponding to the VDMC framework. When asve_custom_displacement_video_size_flag is equal to 1, it means that the method mentioned in this application is enabled in VDMC; when it is equal to 0, it means that the method mentioned in this application is not enabled in VDMC.

[0258] The displacement information video generation method and decoding method provided in the embodiments of the present application may be executed by a displacement information video generation device or a decoding device. In the embodiments of the present application, the displacement information video generation method or the decoding method is executed by a displacement information video generation method device or a decoding device as an example to illustrate the displacement information video generation method or the decoding method provided in the embodiments of the present application.

[0259] like Fig.11 As shown, it is a structural schematic diagram of the cross-domain emergency command and dispatch system provided by an embodiment of the present application, including a switch 110, multiple encoding devices 120, multiple decoding devices 130 and multiple display devices 140, the output port of each encoding device 120 is connected to the input port of the switch 110, the input port of each decoding device 130 is connected to the output port of the switch 110, the output port of each decoding device 130 is connected to the corresponding display device 140, different decoding devices 130 are deployed in different command places (taking N command places as an example), and each decoding device 130 and the corresponding display device 140 are deployed in the same command place;

[0260] The encoding device 120 is used to process multiple frames of displacement data to obtain multiple frames of first displacement data and determine the size of the displacement video corresponding to the multiple frames of displacement data; the displacement data is used to store the distance information from the subdivided basic network vertex to the nearest neighbor point of the original input network in the dynamic mesh encoding process;

[0261] and, according to the size of the displacement video, converting a plurality of frames of first displacement data into a displacement video; inputting the displacement video into a first neural network model, performing a first processing on the displacement video based on the first neural network model to obtain a compressed video; encoding the compressed video to obtain a displacement code stream, and sending the displacement code stream to the switch 110;

[0262] Among them, the first neural network model is a neural network model with N video compression processing functions, N is an integer greater than 1; the video compression processing method corresponding to the first processing is determined by the first neural network model.

[0263] Specifically, the encoding device 120 is specifically used to specify the size of the displacement video;

[0264] or,

[0265] Determine the size of the displacement video according to the size of the target frame displacement image;

[0266] The encoding device 120 is further used to determine the size of the target frame displacement image according to the number of data blocks in each row of the displacement video, the size of the data blocks in the displacement video and the number of displacement data in the target frame.

[0267] Furthermore, the encoding device 120 is specifically used to determine the width of the target frame displacement image according to the number of data blocks in each row of the displacement video and the size of the data blocks in the displacement video; and determine the height of the target frame displacement image according to the width of the target frame displacement image and the number of target frame displacement data.

[0268] The encoding device 120 is specifically configured to determine the width of the target frame displacement image according to a first formula; wherein the first formula is:

[0269]

[0270] in, Indicates the width of the target frame displacement image; Indicates the number of data blocks per line in the displacement video; Indicates the size of the data block in the displacement video.

[0271] According to the second formula, the height of the target frame displacement image is determined; wherein the second formula is:

[0272]

[0273] in, Indicates the height of the target frame displacement image; Indicates the number of target frame displacement data; Indicates the width of the target frame displacement image; Indicates a round-up operation.

[0274] In this embodiment, the encoding device 120 is further specifically configured to arrange each frame of first displacement data into a displacement image in a predetermined order according to the size of the displacement video; and multiple frames of displacement images constitute the displacement video;

[0275] Accordingly, the encoding device 120 is further configured to fill the excess pixels with default values ​​if the amount of displacement data of the current frame is less than the size of the displacement video;

[0276] or,

[0277] If the amount of displacement data of the current frame is greater than the size of the displacement video, the redundant displacement data is removed;

[0278] The encoding device is further configured to determine whether the current frame displacement data is the target frame displacement data in a first manner; wherein the first manner includes at least one of the following:

[0279] Presetting the Mth frame displacement data as the target frame displacement data;

[0280] Divide the multiple frames of displacement data into multiple groups of pictures GOP in advance, and set the Nth frame displacement data of each GOP as the target frame displacement data;

[0281] Wherein, M and N are integers greater than or equal to 1 respectively.

[0282] The encoding device 120 is further used to input target information into the first neural network model, wherein the target information is information for indicating a video compression processing method;

[0283] Wherein, the video compression processing mode corresponding to the first processing is determined by the first neural network model based on the target information;

[0284] The target information includes at least one of the following:

[0285] A first identifier, wherein an index value of the first identifier is mapped to a corresponding video compression processing method;

[0286] A second identifier, wherein the second identifier includes a multiple of the image width direction and a multiple of the image height direction;

[0287] The N video compression processing functions include at least two of a loop filtering function, a super-resolution function, and a post-processing filtering function;

[0288] The video compression processing method corresponding to the first processing includes one of a loop filtering processing method, a super-resolution processing method and a post-processing filtering processing method.

[0289] The decoding device 130 is used to receive the displacement code stream through the switch 110, decode the displacement code stream into the compressed video; input the compressed video into the second neural network model, perform a second process on the compressed video based on the second neural network model to obtain the displacement video; extract pixel values ​​from each displacement image frame of the displacement video in a target order to obtain target data; the target order is the same as the order in which the encoding device arranges the displacement images; process the extracted target data to obtain multiple frames of displacement data;

[0290] A display device 140 is used to display corresponding information resources and audio and video signals according to the multiple frames of displacement data;

[0291] Among them, the second neural network model is a neural network model with N video decompression processing functions, N is an integer greater than 1; the video decompression processing method corresponding to the second processing is determined by the second neural network model.

[0292] Specifically, the decoding device 130 is specifically configured to determine, for a frame of displacement image, the number of pixel values ​​that need to be extracted from the displacement image according to the number of vertices of the subdivided reconstructed basic mesh corresponding to the displacement image;

[0293] If the number of vertices of the reconstructed basic mesh after subdivision is less than or equal to the number of pixel values ​​required to be extracted from the displacement image, extracting pixel values ​​equal to the number of vertices of the reconstructed basic mesh after subdivision from the displacement image in the target order to obtain target data;

[0294] Alternatively, if the number of vertices of the basic mesh reconstructed after subdivision is greater than the number of pixel values ​​that need to be extracted from the displacement image, all pixel values ​​are extracted from the displacement image in the target order, and a padding operation is performed at the end of the extracted data until the number of data is equal to the number of vertices of the basic mesh reconstructed after subdivision, thereby obtaining the target data.

[0295] In an embodiment of the present application, multiple frames of displacement data are converted into displacement video, and the size of the displacement video is determined before generating the displacement video, thereby improving the flexibility, real-time and controllability of subsequent video encoding; a variety of video compression processing functions are realized through a unified neural network model, which is conducive to reducing the complexity of video compression processing and avoiding waste of resources.

[0296] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0297] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable the terminal or network side device to execute the methods described in each embodiment of the present application.

[0298] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A cross-domain emergency command and dispatch method, characterized in that: The invention is applied to a cross-domain emergency command and dispatch system including a switch, a plurality of encoding devices, a plurality of decoding devices and a plurality of display devices, wherein the output port of each encoding device is connected to the input port of the switch, the input port of each decoding device is connected to the output port of the switch, the output port of each decoding device is connected to the corresponding display device, different decoding devices are deployed in different command places, and each decoding device and the corresponding display device are deployed in the same command place, and the method comprises the following steps: The encoding device processes the multiple frames of displacement data to obtain multiple frames of first displacement data, and determines the size of the displacement video corresponding to the multiple frames of displacement data; the displacement data is used to store the distance information from the subdivided basic network vertex to the nearest neighbor point of the original input network in the dynamic mesh encoding process; The encoding device converts the plurality of frames of first displacement data into a displacement video according to the size of the displacement video; The encoding device inputs the displacement video into a first neural network model, and performs a first process on the displacement video based on the first neural network model to obtain a compressed video; The encoding device encodes the compressed video to obtain a displacement code stream, and sends the displacement code stream to the switch; Wherein, the first neural network model is a neural network model having N video compression processing functions, N is an integer greater than 1; the video compression processing mode corresponding to the first processing is determined by the first neural network model; The encoding device determines the size of the displacement video corresponding to the multiple frames of displacement data, including: The encoding device specifies the size of the displacement video; or, The encoding device determines the size of the displacement video according to the size of the displacement image of the target frame; The method further comprises: The encoding device determines the size of the target frame displacement image according to the number of data blocks in each row of the displacement video, the size of the data blocks in the displacement video and the number of displacement data of the target frame; The encoding device determines the size of the target frame displacement image according to the number of data blocks in each row of the displacement video, the size of the data blocks in the displacement video and the number of displacement data of the target frame, including: The encoding device determines the width of the displacement image of the target frame according to the number of data blocks in each row of the displacement video and the size of the data blocks in the displacement video; The encoding device determines the height of the target frame displacement image according to the width of the target frame displacement image and the amount of target frame displacement data.

2. The method according to claim 1, characterized in that The encoding device determines the width of the target frame displacement image according to the number of data blocks in each row of the displacement video and the size of the data blocks in the displacement video, including: The encoding device determines the width of the target frame displacement image according to a first formula; wherein the first formula is: in, Indicates the width of the target frame displacement image; Indicates the number of data blocks per line in the displacement video; Indicates the size of the data block in the displacement video.

3. The method according to claim 1, characterized in that The encoding device determines the height of the target frame displacement image according to the width of the target frame displacement image and the amount of target frame displacement data, including: The encoding device determines the height of the target frame displacement image according to a second formula; wherein the second formula is: in, Indicates the height of the target frame displacement image; Indicates the number of target frame displacement data; Indicates the width of the target frame displacement image; Indicates a round-up operation.

4. The method according to any one of claims 1 to 3, characterized in that: The encoding device converts a plurality of frames of first displacement data into a displacement video according to the size of the displacement video, including: The encoding device arranges each frame of first displacement data into a displacement image in a predetermined order according to the size of the displacement video; multiple frames of displacement images constitute the displacement video; The method further comprises: If the amount of displacement data of the current frame is less than the size of the displacement video, the encoding device fills the excess pixels with default values; or, If the amount of displacement data of the current frame is greater than the size of the displacement video, the encoding device removes the redundant displacement data; The method further comprises: The encoding device determines whether the current frame displacement data is the target frame displacement data in a first manner; wherein the first manner includes at least one of the following: Presetting the Mth frame displacement data as the target frame displacement data; Divide the multiple frames of displacement data into multiple groups of pictures GOP in advance, and set the Nth frame displacement data of each GOP as the target frame displacement data; Wherein, M and N are integers greater than or equal to 1 respectively.

5. The method according to claim 1, characterized in that The method further comprises: The encoding device inputs target information into the first neural network model, wherein the target information is information for indicating a video compression processing method; Wherein, the video compression processing mode corresponding to the first processing is determined by the first neural network model based on the target information; The target information includes at least one of the following: A first identifier, wherein an index value of the first identifier is mapped to a corresponding video compression processing method; A second identifier, wherein the second identifier includes a multiple of the image width direction and a multiple of the image height direction; The N video compression processing functions include at least two of a loop filtering function, a super-resolution function, and a post-processing filtering function; The video compression processing method corresponding to the first processing includes one of a loop filtering processing method, a super-resolution processing method and a post-processing filtering processing method.

6. The method according to claim 1, characterized in that After the encoding device sends the displacement code stream to the switch, the method further includes: The decoding device receives the displacement code stream through the switch, and decodes the displacement code stream into the compressed video; The decoding device inputs the compressed video into a second neural network model, and performs a second process on the compressed video based on the second neural network model to obtain the displacement video; The decoding device extracts pixel values ​​from each displacement image frame of the displacement video in a target order to obtain target data; the target order is the same as the order in which the displacement images are arranged by the encoding device; The decoding device processes the extracted target data to obtain multiple frames of displacement data; The display device displays corresponding information resources and audio and video signals according to the multiple frames of displacement data; Among them, the second neural network model is a neural network model with N video decompression processing functions, N is an integer greater than 1; the video decompression processing method corresponding to the second processing is determined by the second neural network model.

7. The method according to claim 6, characterized in that The decoding device extracts pixel values ​​from each displacement image frame of the displacement video in a target order to obtain target data, including: The decoding device determines, for a frame of displacement image, the number of pixel values ​​that need to be extracted from the displacement image according to the number of vertices of the subdivided reconstructed basic mesh corresponding to the displacement image; If the number of vertices of the reconstructed basic mesh after subdivision is less than or equal to the number of pixel values ​​required to be extracted from the displacement image, the decoding device extracts pixel values ​​equal to the number of vertices of the reconstructed basic mesh after subdivision from the displacement image in the target order to obtain target data; Alternatively, if the number of vertices of the basic mesh reconstructed after subdivision is greater than the number of pixel values ​​that need to be extracted from the displacement image, the decoding device extracts all pixel values ​​from the displacement image in the target order and performs a padding operation at the end of the extracted data until the number of data is equal to the number of vertices of the basic mesh reconstructed after subdivision, thereby obtaining the target data.

8. A cross-domain emergency command and dispatch system for executing the method according to claim 1, characterized in that: It includes a switch, a plurality of encoding devices, a plurality of decoding devices and a plurality of display devices, wherein the output port of each encoding device is connected to the input port of the switch, the input port of each decoding device is connected to the output port of the switch, the output port of each decoding device is connected to the corresponding display device, different decoding devices are deployed in different command locations, and each decoding device and the corresponding display device are deployed in the same command location; The encoding device is used to process multiple frames of displacement data to obtain multiple frames of first displacement data and determine the size of the displacement video corresponding to the multiple frames of displacement data; the displacement data is used to store the distance information from the subdivided basic network vertex to the nearest neighbor point of the original input network during the dynamic mesh encoding process; and, according to the size of the displacement video, converting the plurality of frames of first displacement data into the displacement video; Inputting the displacement video into a first neural network model, and performing a first process on the displacement video based on the first neural network model to obtain a compressed video; Encoding the compressed video to obtain a displacement code stream, and sending the displacement code stream to the switch; Among them, the first neural network model is a neural network model with N video compression processing functions, N is an integer greater than 1; the video compression processing method corresponding to the first processing is determined by the first neural network model.