Image transmission method and device, medium and program product

Abstract

The invention aims to provide an image transmission method and device, a medium and a program product. The method specifically comprises the following steps: determining a corresponding image enhancement feature according to an original image and a corresponding feature extraction model; and transmitting the original image and the image enhancement feature to a corresponding second user device, so that the second user device determines an enhanced image through the original image, the image enhancement feature and a corresponding image transformation model, and the image transformation model is matched with the feature extraction model. According to the invention, the image enhancement feature extraction and the transformation execution operation are respectively implemented to the first user equipment and the second user equipment, and the size of the coded bit stream is reduced by using a distributed operation method, so that the transmission bandwidth and the storage resources are saved, and a good image transmission environment is provided for users.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a technology for image transmission. Background Art

[0002] Image enhancement technology is a basic image processing technology. Its purpose is to extract and transform the global or local features of an image according to the image application or user visual needs, thereby improving the visual effect of the image. Most image enhancement technologies will add texture information to the image, sharpen the boundaries, or increase the contrast of color blocks. While these operations improve the visual effect, they significantly increase the difficulty of encoding the image or video, making the image or video difficult to compress and transmit. Summary of the invention

[0003] One object of the present application is to provide an image transmission method, device, medium and program product.

[0004] According to one aspect of the present application, there is provided an image transmission method, the method comprising:

[0005] Determine the corresponding image enhancement features according to the original image and the corresponding feature extraction model;

[0006] The original image and the image enhancement feature are transmitted to a corresponding second user device, so that an enhanced image is determined at the second user device through the original image, the image enhancement feature and a corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0007] According to another aspect of the present application, an image transmission method is provided, which is applied to a second user equipment, wherein the method includes:

[0008] Receiving an original image sent by a corresponding first user device and an image enhancement feature corresponding to the original image, wherein the image enhancement feature is determined by the original image and a corresponding feature extraction model;

[0009] A corresponding enhanced image is determined according to the original image, the image enhancement feature and a corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0010] According to one aspect of the present application, there is provided an image transmission method, wherein the method comprises:

[0011] The first user equipment determines a corresponding image enhancement feature according to the original image and a corresponding feature extraction model;

[0012] The first user equipment transmits the original image and the image enhancement feature to a corresponding second user equipment;

[0013] The second user device receives the original image and the image enhancement features corresponding to the original image, and determines a corresponding enhanced image according to the original image, the image enhancement features and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0014] According to one aspect of the present application, an image transmission device is provided, wherein the device comprises:

[0015] A module for determining corresponding image enhancement features according to an original image and a corresponding feature extraction model;

[0016] A second module is used to transmit the original image and the image enhancement feature to a corresponding second user device, so that the second user device determines an enhanced image through the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0017] According to another aspect of the present application, another image transmission device is provided, wherein the device comprises:

[0018] A module 21 is configured to receive an original image sent by a corresponding first user device and an image enhancement feature corresponding to the original image, wherein the image enhancement feature is determined by the original image and a corresponding feature extraction model;

[0019] A second module is used to determine a corresponding enhanced image according to the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0020] According to one aspect of the present application, a computer device is provided, wherein the device includes:

[0021] Processor; and

[0022] A memory arranged to store computer executable instructions, which when executed cause the processor to perform the steps of any of the methods described above.

[0023] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed, the system performs the steps of any of the methods described above.

[0024] According to one aspect of the present application, a computer program product is provided, including a computer program / instruction, characterized in that the computer program / instruction implements the steps of any of the above methods when executed by a processor.

[0025] Compared with the prior art, the present application implements the image enhancement feature extraction and transformation execution operations on the first user device and the second user device respectively, and uses a distributed operation method to reduce the size of the encoded bit stream, saving transmission bandwidth and storage resources, and providing users with a good image transmission environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 A flowchart of a system method for image transmission according to an embodiment of the present application is shown;

[0028] Figure 2 A flow chart of an image transmission method according to an embodiment of the present application is shown;

[0029] Figure 3 A flow chart of an image transmission method according to another embodiment of the present application is shown;

[0030] Figure 4 The functional modules of the first user equipment according to one embodiment of the present application are shown;

[0031] Figure 5 The functional modules of the second user equipment according to one embodiment of the present application are shown;

[0032] Figure 6 An exemplary system is shown that can be used to implement the various embodiments described in this application.

[0033] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0034] The present application is described in further detail below in conjunction with the accompanying drawings.

[0035] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (eg, a central processing unit (CPU)), an input / output interface, a network interface and a memory.

[0036] Memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.

[0037] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, Phase-Change Memory (PCM), Programmable Random Access Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), Read-Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0038] The devices referred to in this application include but are not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include but are not limited to any mobile electronic products that can interact with users (for example, interact with users through a touchpad), such as smart phones, tablet computers, etc. The mobile electronic products can use any operating system, such as Android operating system, iOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field Programmable Gate Array, FPGA), digital signal processors (Digital Signal Processor, DSP), embedded devices, etc. The network devices include but are not limited to computers, network hosts, single network servers, multiple network server sets or multiple servers. The cloud is composed of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. The network includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless self-organizing network (Ad Hoc network), etc. Preferably, the device may also be a program running on the user device, the network device, or a device formed by integrating the user device and the network device, the touch terminal, or the network device and the touch terminal through a network.

[0039] Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or future devices that are applicable to the present application should also be included in the scope of protection of the present application and are included here by reference.

[0040] In the description of the present application, “plurality” means two or more, unless otherwise clearly and specifically defined.

[0041] The image transmission method involved in the present application is generally applied to a first user device and a second user device, wherein there is a communication connection between the first user device and the second user device, through which the first user device can transmit images to the second user device, wherein the communication connection includes a direct communication connection established between the first user device and the second user device (for example, data transmission by wired or wireless means) or an indirect communication connection (for example, data transmission by forwarding via other devices or servers, etc.). In this solution, the feature map extraction and enhancement transformation execution operations in the image enhancement technology are respectively implemented on the first user device and the second user device, the first user device obtains the original image to be enhanced, determines the corresponding image enhancement features based on the original image through feature map extraction, and sends the original image and image enhancement features to the second user device, and the second user device can enhance the original image based on the received image enhancement features, thereby determining the corresponding enhanced image. The first user device includes but is not limited to any mobile electronic product that can interact with a user (for example, interact with a user through a touchpad), such as a smart phone, a tablet computer, etc. The first user device includes a corresponding data processing device for collecting, storing, retrieving, processing, transforming and transmitting image data; in some cases, the first user device also includes a corresponding display device for presenting a corresponding original image, such as a display screen; the first user device also includes a corresponding data import device for receiving or obtaining user input or imported processing data, such as a keyboard, mouse or touch screen input device. The device of the second user device is the same or similar to that of the first user device, and will not be described in detail. The second user device is used to generally refer to the receiving end of image enhancement acquisition, and the number of the second user device can be one or more, and there is no limitation here.

[0042] The traditional image enhancement device is a centralized system. In the complete process of multimedia content storage and transmission, the traditional image enhancement device can be installed before the image or video encoder to enhance the unencoded image or video frame at the sending end; it can also be installed after the image or video decoder to enhance the decoded image or video frame at the receiving end. Whether installed before the encoder or after the decoder, the image enhancement device is a centralized system, that is, the complete image enhancement mathematical operation is completed at the same end. Compared with the image enhancement device installed before the encoder, the present application can significantly reduce the encoding difficulty of the image or video encoder, resulting in a reduction in the bit rate or size of the transmitted file under the same picture quality; compared with the image enhancement device installed after the decoder, it can effectively reduce the processor computing overhead and additional transmission delay at the receiving end.

[0043] Figure 1A system method for image transmission according to the present application is shown, and the system method is applied to a system composed of a first user device and a second user device, and the method specifically includes:

[0044] The first user equipment determines a corresponding image enhancement feature according to the original image and a corresponding feature extraction model;

[0045] The first user equipment transmits the original image and the image enhancement feature to a corresponding second user equipment;

[0046] The second user device receives the original image and the image enhancement features corresponding to the original image, and determines a corresponding enhanced image according to the original image, the image enhancement features and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0047] refer to Figure 1 The system method shown in the figure, below we will introduce the image transmission method of the present application from the perspectives of the first user device and the second user device respectively.

[0048] First user device

[0049] Figure 2 An image transmission method according to one aspect of the present application is shown, which is applied to a first user device. The first user device is suitable for Figure 1 The system topology shown in the figure comprises steps S101 and S102. In step S101, a corresponding image enhancement feature is determined according to an original image and a corresponding feature extraction model; in step S102, the original image and the image enhancement feature are transmitted to a corresponding second user device, so that an enhanced image is determined on the second user device through the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0050] Specifically, in step S101, the corresponding image enhancement feature is determined according to the original image and the corresponding feature extraction model. For example, the original image includes image information to be transmitted and enhanced and stored in the first user device. The original image may be taken by a camera device of the first user device, may be crawled from the network through a specific interface, or may be transmitted to the first user device by other devices. The original image includes one or more image frames. In some cases, the original image includes a real-time image frame for real-time transmission in a video stream acquired in real time, and the original image is a current image frame in a frame sequence after video disassembly of the corresponding video stream. The corresponding image enhancement feature includes an image transformation parameter that generates visual features that meet user requirements by transforming the color and texture of the original image by mathematical means. The image transformation parameter is applied to the corresponding image enhancement scene, including but not limited to image denoising, color enhancement, tone mapping, image sharpening, etc. Here, we take color enhancement as an example to illustrate the following embodiments. Those skilled in the art should understand that the following embodiments are also applicable to other image enhancement scenes. Of course, those skilled in the art should understand that the above-mentioned image enhancement scenarios are only examples, and other existing or future image enhancement scenarios, if applicable to the present application, should also be included in the scope of protection of the present application and are included here by reference.

[0051] The feature extraction model includes an algorithm model for extracting the image enhancement features corresponding to the original image and the target enhanced image determined by the training of the corresponding image data set through a specific image enhancement algorithm to be implemented; the corresponding image data set includes sample image data for containing the target enhancement features. For example, in some embodiments, the method further includes step S103 (not shown), in which a plurality of sample images and enhanced images of the plurality of sample images are obtained; the plurality of sample images and the enhanced images of the plurality of sample images are input into the corresponding deep convolutional neural network for training to determine the feature extraction model and the image transformation model. For example, the first user device obtains training image data for data training of a specific or specified image enhancement algorithm, each pair of training image data includes a corresponding sample image and the target enhancement features of the sample image, and the corresponding target enhancement features can be image transformation parameters of the sample image, or can be an enhanced image containing sample image transformation parameters, etc. The data source in the corresponding image data set includes but is not limited to user import, network crawling or transmission from other devices, etc. The enhancement method described herein does not limit the size, source, quantity, style, etc. of the sample images in the image data set. Of course, the richer the scenes and the greater the number of sample images in the image data set, the better the generalization of the generated training set for the enhancement algorithm, and the closer the enhancement effect of the trained neural network is to the original algorithm. The feature extraction model is composed of a deep convolutional neural network, specifically including a global feature extraction network, a local feature extraction network, and a feature fusion network. After the first user device obtains the corresponding multiple sample images and the enhanced images of the multiple sample images, it inputs the corresponding deep convolutional neural network for training based on the multiple training image data sets to determine the corresponding feature extraction model and image transformation model. Among them, the corresponding feature extraction model is set on the first user device end, and the image transformation model is set on the second user device. Since the feature extraction model and the image transformation model are trained based on the same image data set, the two are adapted, such as the weights of the deep convolutional neural network in the feature extraction model and the weights of the deep convolutional neural network in the image transformation model are adapted, etc. Adaptation means that the feature extraction models and image transformation models trained by different image data sets cannot be mixed and matched. Among them, the first user end inputs the image data set into the deep convolutional neural network composed of the feature extraction model and the image transformation model in a certain order and batch, and calculates the loss of the deep convolutional neural network output and the corresponding enhanced image according to the loss function. The optimizer minimizes the loss by updating the network weights. When the loss drops below a preset loss threshold, it stops updating the network weights and model training stops.

[0052] In step S102, the original image and the image enhancement feature are transmitted to the corresponding second user device, so that the second user device determines the enhanced image through the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model. For example, after the first user device obtains the original image and the corresponding image enhancement feature, the original image and the image enhancement feature can be transmitted to the second user device through a communication connection with the second user device, so that the second user device can enhance the original image by using the image enhancement feature through the adapted image transformation model to obtain the corresponding enhanced image. The enhancement processing includes but is not limited to guided graph networks, guided graph slicing, channel expansion networks, bilateral grid generation, and pixel color affine transformation.

[0053] In some embodiments, in step S102, the original image is compressed to determine the corresponding original image bit stream; the image enhancement feature is compressed to determine the corresponding enhanced feature bit stream; the original image bit stream and the enhanced feature bit stream are transmitted in real time to the corresponding second user device. For example, the first user device can directly send the unenhanced original image and the corresponding image enhancement feature to the second user device, or, in order to prevent data loss or loss caused by matching the original image with the corresponding image enhancement feature, the first user device can compress the original image and the image enhancement feature and package them for transmission to the second user device. For example, the first user device compresses the original image and the image enhancement feature to determine the corresponding original image bit stream and the enhanced feature bit stream, and packages and encapsulates the original image bit stream and the enhanced feature bit stream and transmits them together to the second user device. Among them, the compression algorithm corresponding to the original image includes but is not limited to the publicly available video and image coding standards, such as JPEG, h.264, h.265, AV1, etc. The compression algorithms corresponding to the image enhancement features include, but are not limited to, lossless compression algorithms, lossy compression algorithms, etc., for example, the Huffman algorithm, the LZW (Lenpel-Ziv & Welch) compression algorithm, etc. Of course, those skilled in the art should understand that the above-mentioned compression algorithms are only examples, and other existing or future compression algorithms that may appear, if applicable to the present application, should also be included in the scope of protection of the present application and are included here by reference. The image enhancement feature map can be compressed independently, or it can be differentially compressed in the time domain depending on the feature map generated by the previous frame. After the second user device receives the corresponding compressed file, it can decompress the file to obtain the corresponding original image and the image enhancement features of the original image. The encapsulation operation is a reversible operation, and can be completely decapsulated on the second user end.

[0054] In some embodiments, the feature extraction model includes a local feature extraction module, a global feature extraction module and a feature fusion module; wherein, in step S101, the original image is input into the local feature extraction module, the local features in the original image are extracted, and the corresponding local features of the image are determined; the original image is input into the global feature extraction module, the global features in the original image are extracted, and the corresponding global features of the image are determined; the local features of the image and the global features of the image are input into the feature fusion module, the local features of the image are fused with the global features of the image, and the corresponding image enhancement features are determined. For example, the input data of the local feature module is a three-channel unenhanced original image, which is cascaded by difference downsampling, convolution layer, activation layer, and pooling layer, and is used to extract the local features of the image of the network. The feature map shape of the local features of the image extracted by the local feature extraction module is [BS, C0, H, W], wherein the resolution [H, W] of the feature image is lower than the resolution of the input original image, and the number of feature map channels C0 is higher than the number of channels of the input original image. BS is the number of batches in the neural network model, which is not required here. The reduction in resolution is achieved by difference downsampling, convolutional layers or pooling layers with a step size greater than or equal to 2, and a combination of these layers. The weights of the local feature module are determined by training.

[0055] The input of the global feature extraction module is a three-channel unenhanced original image, which is composed of a difference downsampling, a convolution layer, an activation layer, a pooling layer, and a fully connected layer, and is used to extract the global image features of the network. In order to extract the global features of the image, the feature extraction network is provided with a fully connected layer, and the feature map shape of the global features of the image is [BS, C0]. The weights of the network are all determined by training.

[0056] The feature fusion module is used to fuse the local features of the image and the global features of the image, and is specifically composed of a linear combination of the local features of the image and the global features of the image and an activation function. For example, in some embodiments, the feature fusion module is composed of the linear relationship between the local features of the image and the global features of the image and the corresponding activation function. For example, by setting a corresponding activation function and combining the local features of the image and the global features of the image (such as a linear combination or a nonlinear combination, etc.), the local features of the image and the global features of the image can be fused to determine the corresponding fused image enhancement features. Specifically, it can be performed by the following formula:

[0057]

[0058] Among them, G and L are the global and local features of the image respectively, and the number of their channels is C o . w′ cc′ With w cc′They are the weight matrices for the fusion of global image features and local image features, both of which have the shape of [C f ,C o ], where C f is the number of feature map channels of the image enhancement feature after fusion. c is the bias of each channel after fusion, and σ is the activation function. cc′ , w cc′ With b c Determined by training.

[0059] In some embodiments, the pixel value in the image enhancement feature is determined by all corresponding pixel values ​​in the global image feature and the multi-channel values ​​corresponding to the pixel position in the local image feature. For example, referring to the image fusion formula shown in formula (1), the feature map F of the fused image enhancement feature is c (x, y) has the same resolution as the feature map L(x, y) of the local features of the image. The value of the feature map of the fused image enhancement feature at a certain pixel position depends on the values ​​of all pixels in the global feature map and the multi-channel values ​​of the same pixel position in the local feature map, but does not depend on the multi-channel values ​​of other pixel positions in the local feature map.

[0060] Second user device

[0061] Figure 3 An image transmission method according to one aspect of the present application is shown, which is applied to a second user device. The second user device is suitable for Figure 1The system topology shown in the figure comprises steps S201 and S202. In step S201, an original image and an image enhancement feature corresponding to the original image sent by a corresponding first user device are received, wherein the image enhancement feature is determined by the original image and a corresponding feature extraction model; in step S102, a corresponding enhanced image is determined according to the original image, the image enhancement feature and a corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model. Here, the process of extracting the corresponding image enhancement feature is the same as the processing process of the first user device, and will not be repeated here. The corresponding image transformation model comprises an algorithm model for performing image enhancement on the original image based on a specific image enhancement algorithm and the image enhancement feature. Since the image transformation model and the feature extraction model are trained based on the same image data set, the two are adapted, such as the weights of the deep convolutional neural network in the feature extraction model are adapted to the weights of the deep convolutional neural network in the image transformation model. Adaptation means that the feature extraction model and the image transformation model trained by different image data sets cannot be mixed and matched. The second user device uses the image enhancement feature to enhance the original image through the adapted image transformation model to obtain a corresponding enhanced image, wherein the enhancement processing includes but is not limited to a guided graph network, guided graph slicing, channel expansion network, bilateral grid generation, and pixel color affine transformation. After the second user device determines the corresponding enhanced image, the enhanced image can be presented through a corresponding display device, such as by merging multiple enhanced images into a real-time video stream and displaying it through a corresponding liquid crystal display screen.

[0062] In some embodiments, the image transformation model includes an affine model; wherein, the determining the corresponding enhanced image according to the original image, the image enhancement feature and the corresponding image transformation model includes: determining the corresponding affine coefficient according to the original image, the image enhancement feature and the corresponding image transformation model; performing image enhancement on the original image according to the affine coefficient and the affine model to determine the corresponding enhanced image. For example, affine transformation refers to the process of transforming a vector space into another vector space by a linear transformation and a translation. In the case of finite dimensions, each affine transformation can be given by a matrix and a vector, which can be written as a matrix and an additional column. An affine transformation corresponds to the multiplication of a matrix and a vector. This scheme can obtain the corresponding enhanced image by solving the corresponding affine coefficients and using the affine coefficients to transform the original image memory. The second user device can input the corresponding image enhancement feature through the corresponding image transformation model, and after the processes of guided graph network, guided graph slicing, channel expansion network, bilateral grid generation, etc., the corresponding affine coefficient is determined, such as a matrix of 3*4 corresponding affine coefficients:

[0063]

[0064] Correspondingly, the three-channel pixel values ​​of the original image at the pixel coordinate (x, y) are (c1, c2, c3), and the three-channel pixel values ​​(v1, v2, v3) of the enhanced image at (x, y) are calculated by the following formulas:

[0065]

[0066] Therefore, based on the above formula (3), the pixel values ​​in the original image are determined by radial transformation to determine the pixel values ​​of the corresponding enhanced image.

[0067] In some embodiments, the image transformation model further includes a guide graph generation module and a slice parameter module; wherein the determining the corresponding affine coefficient according to the original image, the image enhancement feature and the corresponding image transformation model includes: inputting the original image into the guide graph generation module, converting the original image from a three-channel image to a single-channel image, and determining the corresponding guide image; inputting the image enhancement feature into the slice parameter module to determine the corresponding slice parameter; slicing the guide image based on the slice parameter to generate the corresponding affine coefficient. For example, the guide graph generation module is used to convert the unenhanced three-channel original image into a single-channel guide graph. Specifically, the guide graph generation module is composed of one or more point-by-point convolutional networks and activation layers, and the convolution kernel size of the point-by-point convolutional network is one. The guide graph generation module converts the unenhanced three-channel image into a single-channel guide image g(x,y). The resolution of the guide graph is the same as the resolution of the input three-channel image, the number of channels of the guide image is one, and the weight of the corresponding network can be determined by training, or it can be pre-set before training and remain unchanged during the training process. After the second user device determines the corresponding guide image, it obtains the slicing parameters and performs a slicing operation on the guide image to generate corresponding affine coefficients, etc. For example, the slicing operation is to cut the corresponding image into multiple small images using the slicing parameters. Here, the slicing coefficients are used to slice the guide image to generate affine coefficients (such as a color transformation coefficient map). For each unenhanced original image, the color transformation coefficient map is a three-dimensional matrix with the same resolution as the guide image and twelve channels. In other words, each pixel in the color transformation coefficient map corresponds to a vector with 12 elements. After deformation, the vector acts as a three-channel affine transformation matrix to transform the color of each channel of the original three-channel image. Specifically, the mathematical description of the slicing operation is:

[0068]

[0069] Where g(x,y) is the guidance image generated by the guidance graph generation module. B(i,j,k) is the slicing parameter, such as a bilateral grid, where i,j are the indices in the height and width [H,W] directions of the bilateral grid, respectively, and k is the index in the depth direction of the grid. τ is the linear interpolation operator τ(·)=max(1-|·|,0), s x is the ratio of the bilateral grid width to the width of the original unenhanced image. y is the ratio of the height of the bilateral grid to the height of the original image without enhancement. d is the predefined grid depth.

[0070] In some embodiments, the slice parameter module includes a channel expansion unit and a bilateral grid unit; wherein, the inputting the image enhancement feature into the slice parameter module to determine the corresponding slice parameter includes: inputting the image enhancement feature into the channel expansion unit to expand the number of channels of the image enhancement feature to determine the corresponding image enhancement expansion feature; inputting the image enhancement expansion feature into the bilateral grid unit to change the dimension corresponding to the number of channels of the image enhancement expansion feature to determine the corresponding slice parameter. For example, the slice parameter module includes a channel expansion unit and a bilateral grid unit, and the slice parameter module determines the slice parameters for slicing the guide image using the corresponding image enhancement feature, and specifically includes a channel expansion unit and a bilateral grid unit. The channel expansion unit is a point-by-point convolution network with a convolution kernel size of 1 and a step size of 1. The input of the channel expansion unit is a fusion feature map corresponding to the image enhancement feature extracted by the feature extraction module, and the resolution of the output feature map corresponding to the image enhancement expansion feature output by the unit is the same as the resolution of the fusion feature map. In some embodiments, the number of expansion channels of the channel expansion unit is determined by a preset coefficient and a corresponding grid depth. For example, the number of channels of the output feature map is an integer multiple of twelve, recorded as 12*d, where d is the grid depth, and the value of d can be pre-set, such as the number of expanded channels is determined by the first user end according to the video resolution and image quality requirements. The weights of the point-by-point convolutional network are determined by training. The bilateral grid unit is used to deform the [12*d] dimension of the output feature map of the image enhancement expansion feature after channel expansion into a shape of [12, d], while the shapes of other dimensions remain unchanged, thereby determining the corresponding slice parameters. The deformed feature map is used as a bilateral grid, recorded as B(i, j, k). For example, for a BS feature map, if the input feature map shape is [BS, 96, 16, 16], the output feature map is deformed to [BS, 12, 8, 16, 16], where d = 8 is the grid depth.

[0071] The above mainly introduces various embodiments of an image transmission method of the present application. In addition, the present application also provides specific devices that can implement the above embodiments. Figure 4 , Figure 5 Make an introduction.

[0072] Figure 4 A first user device 100 for image transmission according to one aspect of the present application is shown. The first user device is suitable for Figure 1 The system topology shown in the figure, the device 100 includes a first module 101 and a second module 102. The first module 101 is used to determine the corresponding image enhancement feature according to the original image and the corresponding feature extraction model; the first module 102 is used to transmit the original image and the image enhancement feature to the corresponding second user device, so that the second user device determines the enhanced image through the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0073] In some embodiments, the device 100 also includes a third module (not shown) for acquiring multiple sample images and enhanced images of the multiple sample images; inputting the multiple sample images and the enhanced images of the multiple sample images into the corresponding deep convolutional neural network for training to determine the feature extraction model and the image transformation model.

[0074] In some embodiments, module 102 is used to compress the original image to determine the corresponding original image bit stream; compress the image enhancement features to determine the corresponding enhanced feature bit stream; and transmit the original image bit stream and the enhanced feature bit stream to the corresponding second user device in real time.

[0075] In some embodiments, the feature extraction model includes a local feature extraction module, a global feature extraction module and a feature fusion module; wherein, module 101 is used to input the original image into the local feature extraction module, extract the local features in the original image, and determine the corresponding local features of the image; input the original image into the global feature extraction module, extract the global features in the original image, and determine the corresponding global features of the image; input the local features of the image and the global features of the image into the feature fusion module, perform feature fusion on the local features of the image and the global features of the image, and determine the corresponding image enhancement features. In some embodiments, the feature fusion module is composed of the linear relationship between the local features of the image, the global features of the image, and the corresponding activation function. In some embodiments, the pixel value in the image enhancement feature is determined by all the corresponding pixel values ​​in the global features of the image and the multi-channel values ​​corresponding to the pixel position in the local features of the image.

[0076] Here, the specific implementations corresponding to the one-one module 101, the one-two module 102 and the one-three module are the same or similar to the embodiments of the aforementioned step S101, step S102 and step S103, and are therefore not repeated here and are included herein by reference.

[0077] Figure 5 A second user device 200 for image transmission according to one aspect of the present application is shown. The second user device is suitable for Figure 1 The system topology shown in the figure, the device 200 includes a 21 module 201 and a 22 module 202. The 21 module 201 is used to receive an original image sent by a corresponding first user device and an image enhancement feature corresponding to the original image, wherein the image enhancement feature is determined by the original image and a corresponding feature extraction model; the 22 module 102 is used to determine a corresponding enhanced image according to the original image, the image enhancement feature and a corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

[0078] In some embodiments, the image transformation model includes an affine model; wherein, determining the corresponding enhanced image based on the original image, the image enhancement feature and the corresponding image transformation model includes: determining the corresponding affine coefficients based on the original image, the image enhancement feature and the corresponding image transformation model; and performing image enhancement on the original image based on the affine coefficients and the affine model to determine the corresponding enhanced image.

[0079] In some embodiments, the image transformation model also includes a guide graph generation module and a slice parameter module; wherein, determining the corresponding affine coefficients based on the original image, the image enhancement features and the corresponding image transformation model includes: inputting the original image into the guide graph generation module, converting the original image from a three-channel image to a single-channel image, and determining the corresponding guide image; inputting the image enhancement features into the slice parameter module to determine the corresponding slice parameters; slicing the guide image based on the slice parameters to generate corresponding affine coefficients.

[0080] In some embodiments, the slice parameter module includes a channel expansion unit and a bilateral grid unit; wherein, the inputting the image enhancement feature into the slice parameter module to determine the corresponding slice parameter includes: inputting the image enhancement feature into the channel expansion unit to expand the number of channels of the image enhancement feature to determine the corresponding image enhancement expansion feature; inputting the image enhancement expansion feature into the bilateral grid unit to change the dimension corresponding to the number of channels of the image enhancement expansion feature to determine the corresponding slice parameter. In some embodiments, the number of expanded channels of the channel expansion unit is determined by a preset coefficient and a corresponding grid depth.

[0081] Here, the specific implementations corresponding to the 21 module 201 and the 22 module 202 are the same or similar to the embodiments of the aforementioned step S201 and step S202, and thus will not be described in detail and are included herein by reference.

[0082] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium, which stores computer code. When the computer code is executed, the method described in any of the preceding items is executed.

[0083] The present application also provides a computer program product. When the computer program product is executed by a computer device, the method described in any of the preceding items is executed.

[0084] The present application also provides a computer device, the computer device comprising:

[0085] one or more processors;

[0086] a memory for storing one or more computer programs;

[0087] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any of the preceding items.

[0088] Figure 6 An exemplary system that can be used to implement various embodiments described in this application is shown;

[0089] like Figure 6 In some embodiments shown, the system 300 can be any of the above-mentioned devices in each of the described embodiments. In some embodiments, the system 300 may include one or more computer-readable media (e.g., system memory or NVM / storage device 320) with instructions and one or more processors (e.g., (one or more) processors 305) coupled to the one or more computer-readable media and configured to execute instructions to implement modules to perform the actions described in this application.

[0090] For one embodiment, system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 305 and / or any suitable device or component in communication with system control module 310 .

[0091] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0092] The system memory 315 may be used, for example, to load and store data and / or instructions for the system 300. For one embodiment, the system memory 315 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the system memory 315 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0093] For one embodiment, system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to NVM / storage device 320 and communication interface(s) 325 .

[0094] For example, NVM / storage device 320 may be used to store data and / or instructions. NVM / storage device 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0095] NVM / storage device 320 may include storage resources that are physically part of the device on which system 300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 320 may be accessed over a network via communication interface(s) 325.

[0096] Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and / or with any other suitable devices. System 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0097] For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 (e.g., the memory controller module 330). For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310 to form a system on chip (SoC).

[0098] In various embodiments, the system 300 may be, but is not limited to: a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the system 300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

[0099] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0100] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0101] Communication media include media by which communication signals containing, for example, computer readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media may include guided transmission media such as cables and wires (e.g., fiber optic, coaxial, etc.) and wireless (unguided transmission) media that can propagate energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared. Computer readable instructions, data structures, program modules, or other data may be embodied as a modulated data signal in, for example, a wireless medium such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology. The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

[0102] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information / data for use by a computer system.

[0103] Here, according to an embodiment of the present application, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to run the methods and / or technical solutions based on the aforementioned multiple embodiments of the present application.

[0104] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. An image transmission method, applied to a first user device, wherein: The method includes: Determine the corresponding image enhancement features according to the original image and the corresponding feature extraction model; The original image and the image enhancement feature are transmitted to a corresponding second user device, so that an enhanced image is determined at the second user device through the original image, the image enhancement feature and a corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

2. The method according to claim 1, wherein: The transmitting the original image and the image enhancement feature to a corresponding second user device includes: Compressing the original image to determine a corresponding original image bit stream; Compressing the image enhancement feature to determine a corresponding enhancement feature bit stream; The original image bit stream and the enhanced feature bit stream are transmitted to the corresponding second user equipment in real time.

3. The method according to claim 1, wherein: The method further comprises: Acquire a plurality of sample images and enhanced images of the plurality of sample images; The multiple sample images and the enhanced images of the multiple sample images are input into a corresponding deep convolutional neural network for training to determine the feature extraction model and the image transformation model.

4. The method according to claim 3, wherein: The feature extraction model includes a local feature extraction module, a global feature extraction module and a feature fusion module; wherein, determining the corresponding image enhancement features according to the original image and the corresponding feature extraction model includes: Inputting the original image into the local feature extraction module, extracting local features in the original image, and determining corresponding image local features; Inputting the original image into the global feature extraction module, extracting the global features in the original image, and determining the corresponding image global features; The local image features and the global image features are input into the feature fusion module, and the local image features are fused with the global image features to determine corresponding image enhancement features.

5. The method according to claim 4, wherein: The feature fusion module is composed of the linear relationship between the local features of the image, the global features of the image and the corresponding activation function.

6. The method according to claim 5, wherein: The pixel value in the image enhancement feature is determined by all corresponding pixel values ​​in the global feature of the image and the multi-channel value corresponding to the pixel position in the local feature of the image.

7. An image transmission method, applied to a second user device, wherein: The method includes: Receiving an original image sent by a corresponding first user device and an image enhancement feature corresponding to the original image, wherein the image enhancement feature is determined by the original image and a corresponding feature extraction model; A corresponding enhanced image is determined according to the original image, the image enhancement feature and a corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

8. The method according to claim 7, wherein: The image transformation model includes an affine model; wherein determining the corresponding enhanced image according to the original image, the image enhancement feature and the corresponding image transformation model includes: determining corresponding affine coefficients according to the original image, the image enhancement feature and the corresponding image transformation model; The original image is enhanced according to the affine coefficient and the affine model to determine a corresponding enhanced image.

9. The method according to claim 8, wherein: The image transformation model also includes a guide graph generation module and a slice parameter module; wherein the determining of the corresponding affine coefficients according to the original image, the image enhancement features and the corresponding image transformation model includes: Inputting the original image into the guidance map generation module, converting the original image from a three-channel image into a single-channel image, and determining a corresponding guidance image; Inputting the image enhancement feature into the slice parameter module to determine the corresponding slice parameter; The guide image is sliced ​​based on the slice parameters to generate corresponding affine coefficients.

10. The method according to claim 9, wherein: The slice parameter module includes a channel expansion unit and a bilateral grid unit; wherein, inputting the image enhancement feature into the slice parameter module to determine the corresponding slice parameter includes: inputting the image enhancement feature into the channel expansion unit to expand the number of channels of the image enhancement feature to determine the corresponding image enhancement expansion feature; The image enhancement and expansion features are input into the bilateral grid unit to change the dimension corresponding to the number of channels of the image enhancement and expansion features, and determine the corresponding slice parameters.

11. The method according to claim 10, wherein: The number of expansion channels of the channel expansion unit is determined by a preset coefficient and a corresponding grid depth.

12. An image transmission method, wherein: The method includes: The first user equipment determines a corresponding image enhancement feature according to the original image and the corresponding feature extraction model; The first user equipment transmits the original image and the image enhancement feature to a corresponding second user equipment; The second user device receives the original image and the image enhancement features corresponding to the original image, and determines a corresponding enhanced image according to the original image, the image enhancement features and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

13. An image transmission device, wherein: The equipment includes: A module for determining corresponding image enhancement features according to an original image and a corresponding feature extraction model; A second module is used to transmit the original image and the image enhancement feature to a corresponding second user device, so that the second user device determines an enhanced image through the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

14. An image transmission device, wherein: The equipment includes: A module 21 is configured to receive an original image sent by a corresponding first user device and an image enhancement feature corresponding to the original image, wherein the image enhancement feature is determined by the original image and a corresponding feature extraction model; A second module is used to determine a corresponding enhanced image according to the original image, the image enhancement feature and the corresponding image transformation model, wherein the image transformation model is adapted to the feature extraction model.

15. A computer device, wherein: The equipment includes: Processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 11.

16. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: The computer program / instructions, when executed, cause the system to perform the steps of the method as claimed in any one of claims 1 to 11.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images