An image compression and reconstruction method based on super-resolution and prior knowledge

Abstract

The application discloses an image compression and reconstruction method based on super-resolution and prior knowledge, comprising the following steps: obtaining a teaching image from a first terminal; inputting the teaching image into a preset double-path prior type acquirer to obtain a preliminary prior feature map; compressing the teaching image based on semantic information by using pyramid convolution to obtain a compressed image; embedding prior features in the preliminary prior feature map into a super-resolution deep reconstruction process, wherein the prior features are features in the teaching image that match preset prior conditions; performing shallow feature extraction on the compressed image and deep feature learning on the embedded image prior features based on a multi-scale salient feature extraction super-resolution based on an attention mechanism, and obtaining a reconstructed image by reconstruction. The application can improve the realization of image compression and reconstruction based on text face double prior, and can be widely applied to the field of image compression and reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image compression and reconstruction technology, and in particular to an image compression and reconstruction method based on super-resolution and prior knowledge. Background Technology

[0002] In the digital age, synchronous classrooms are a teaching format where supporting schools provide synchronous teaching services to recipient schools, breaking through the limitations of venue and space. Image compression and reconstruction are important steps in image transmission in synchronous classrooms. However, while existing image compression models alleviate the problems caused by image transmission, they still have some shortcomings, such as low compression efficiency and poor display quality of high-compression ratio images or video frames on the terminal.

[0003] Image compression and reconstruction essentially involve compressing an image and then adding high-frequency details to the compressed image to generate a high-resolution image. Image super-resolution aims to reconstruct a high-resolution image from a low-resolution image. However, most existing super-resolution research treats images uniformly as natural scene images. Although related works such as RCAN and HAN have achieved superior performance far exceeding that of bicubic interpolation, they neglect the prior information contained in the image content. Considering the special characteristics of synchronous classroom scenarios, the transmitted images usually consist of teacher and student portraits and text as the main content. Although some researchers have studied the prior information of text super-resolution and face super-resolution, such as TSRN and SISN, there are still problems such as limited application scenarios and inaccurate prior estimation. Image compression and reconstruction based on text and face dual priors in the context of synchronous classrooms urgently needs to be considered. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image compression and reconstruction method based on super-resolution and prior knowledge, so as to improve the realization of image compression and reconstruction based on text and face dual priors.

[0005] One aspect of this invention provides an image compression and reconstruction method based on super-resolution and prior knowledge, comprising:

[0006] Acquire teaching images from the first terminal;

[0007] The teaching image is input into a preset dual-path prior type acquirer to obtain a preliminary prior feature map. The dual-path prior type acquirer is used to classify the input image to obtain a first image mainly composed of text and a second image mainly composed of teacher and student portraits, and to convert the feature distribution of the first image or the second image into a feature distribution that matches the input image.

[0008] The teaching images are compressed using pyramid convolution and based on semantic information to obtain compressed images;

[0009] The prior features in the preliminary prior feature map are embedded into the super-resolution depth reconstruction process. The prior features are the features in the teaching image that match the preset prior conditions.

[0010] Attention-based multi-scale salient feature extraction super-resolution performs shallow feature extraction on compressed images and learns deep features by embedding prior image features, and then reconstructs the image.

[0011] Preferably, the step of inputting the teaching image into a preset dual-path prior type acquirer to obtain a preliminary prior feature map includes:

[0012] The teaching image is input into a preset dual-path prior type acquirer, and the text prior in the dual-path prior type acquirer identifies the teaching image as the first image; the probability vector sequence in the first image is extracted by the text prior; the probability vector sequence is converted into a feature distribution matching the teaching image to obtain a preliminary text prior feature map;

[0013] or,

[0014] The teaching image is input into a preset dual-path prior type acquirer, and the face prior in the dual-path prior type acquirer identifies the teaching image as a second image; the face distribution code in the second image is extracted by the face prior; the face distribution code is converted into a feature distribution matching with the teaching image to obtain a preliminary portrait prior feature map.

[0015] Preferably, the teaching image is compressed using pyramid convolution and based on semantic information to obtain a compressed image, including:

[0016] The teaching feature map is subjected to shallow convolution to obtain a convolutional feature map;

[0017] The convolutional feature map is input into the first branch of the pyramid convolution, and the high and low layer information of the convolutional feature map is fused to generate a first feature map containing multiple semantic layer information.

[0018] The convolutional feature map is input into the second branch of the pyramid convolution to obtain a second feature map based on the low channel capacity space;

[0019] The first feature map and the second feature map are aggregated, and the aggregated feature map is downsampled to obtain a compressed image.

[0020] Preferably, embedding the prior features from the preliminary prior feature map into the compressed image includes:

[0021] The compressed image is processed by a super-resolution module to generate a first multi-scale depth feature map;

[0022] The preliminary prior feature map is input into a dual-path prior feature converter to transform and align the features of the preliminary prior feature map and generate a second multi-scale depth feature map.

[0023] The first multi-scale depth feature map and the second multi-scale depth feature map are fused together, and the prior features in the preliminary prior feature map are embedded into the super-resolution depth reconstruction process.

[0024] Preferably, the attention-based multi-scale salient feature extraction super-resolution method performs shallow feature extraction and deep feature learning on the compressed image with embedded prior features, and reconstructs the image to obtain a reconstructed image, including:

[0025] Extracting shallow features from compressed images;

[0026] Deep residual multi-scale features are extracted from compressed images, and deep residual multi-scale features are aggregated through skip connections;

[0027] In the process of deep residual multi-scale feature extraction, the prior features in the preliminary prior feature map are fused to perform image depth reconstruction.

[0028] Preferably, it further includes:

[0029] A prior knowledge propagation scheme based on knowledge distillation is adopted, and a teacher network is trained according to the teaching image and the reconstructed image. The teacher network is used to generate teaching reconstructed images based on the input teaching image.

[0030] Based on the teacher network, prior information is propagated from the teacher network to the student network using knowledge distillation.

[0031] Based on the first loss between the teaching image and the teaching reconstructed image, the second loss between the teaching image and the reconstructed image of the student network, and the third loss between the teaching reconstructed image and the reconstructed image of the student network, a new reconstructed image corresponding to the teaching image is regenerated.

[0032] Preferably, it further includes:

[0033] The reconstructed image is transmitted to a second terminal for display.

[0034] Another aspect of the present invention provides an image compression and reconstruction apparatus based on super-resolution and prior knowledge, comprising:

[0035] An image acquisition unit is used to acquire teaching images from the first terminal;

[0036] The prior feature acquisition unit is used to input the teaching image into a preset dual-path prior type acquirer to obtain a preliminary prior feature map. The dual-path prior type acquirer is used to classify the input image to obtain a first image mainly composed of text and a second image mainly composed of teacher and student portraits, and to convert the feature distribution of the first image or the second image into a feature distribution matching the input image.

[0037] An image compression unit is used to compress the teaching image using pyramid convolution and based on semantic information to obtain a compressed image;

[0038] A prior feature embedding unit is used to embed prior features from the preliminary prior feature map into the super-resolution depth reconstruction process, wherein the prior features are features in the teaching image that match preset prior conditions.

[0039] The image reconstruction unit is used for multi-scale salient feature extraction based on attention mechanism to perform shallow feature extraction on compressed images and deep feature learning to embed prior image features, and then reconstructs the image.

[0040] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0041] The memory is used to store programs;

[0042] The processor executes the program to implement the above-described method.

[0043] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the above-described method.

[0044] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the method described above.

[0045] This invention provides an image compression and reconstruction method based on super-resolution and prior knowledge. Compared to existing image compression schemes that apply the same compression and reconstruction method to different types of images, ignoring the differences between them, this invention employs an effective compression framework that combines image priors and super-resolution to solve the compression and reconstruction problem of different image categories. Prior knowledge-guided image compression reconstruction can effectively utilize various image detail features. Furthermore, super-resolution-based image reconstruction can restore and supplement high-frequency details in compressed images, effectively improving the visual quality of the reconstructed images. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating an image compression and reconstruction method based on super-resolution and prior knowledge, provided for an embodiment of the present invention;

[0048] Figure 2 An example flowchart of an image compression and reconstruction method based on super-resolution and prior knowledge provided in an embodiment of the present invention;

[0049] Figure 3 Explanatory diagrams illustrating the variables and their meanings in the embodiments provided for the present invention;

[0050] Figure 4 This is a method architecture diagram for a synchronous classroom setting provided by an embodiment of the present invention;

[0051] Figure 5 This is an overall network model structure diagram of one embodiment of the present invention;

[0052] Figure 6 This is an example architecture diagram of a dual-path prior type acquirer provided in an embodiment of the present invention;

[0053] Figure 7 A flowchart illustrating an image prior knowledge embedding process provided in an embodiment of the present invention;

[0054] Figure 8 An example diagram of image compression based on semantic information extraction provided in an embodiment of the present invention;

[0055] Figure 9 An example diagram of image reconstruction based on super-resolution provided in an embodiment of the present invention;

[0056] Figure 10 This is a structural block diagram of an image compression and reconstruction device based on super-resolution and prior knowledge, provided for an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] Reference Figure 1 This invention provides an image compression and reconstruction method based on super-resolution and prior knowledge, specifically including the following steps:

[0059] Step S100: Obtain teaching images from the first terminal.

[0060] Step S110: Input the teaching image into a preset dual-path prior type acquirer to obtain a preliminary prior feature map. The dual-path prior type acquirer is used to classify the input image to obtain a first image mainly composed of text and a second image mainly composed of teacher and student portraits, and convert the feature distribution of the first image or the second image into a feature distribution matching the input image.

[0061] Specifically, it may include the following steps:

[0062] The teaching image is input into a preset dual-path prior type acquirer, and the text prior in the dual-path prior type acquirer identifies the teaching image as the first image; the probability vector sequence in the first image is extracted by the text prior; the probability vector sequence is converted into a feature distribution matching the teaching image to obtain a preliminary prior feature map.

[0063] This may include the following steps:

[0064] The teaching image is input into a preset dual-path prior type acquirer, and the face prior in the dual-path prior type acquirer identifies the teaching image as a second image; the face distribution code in the second image is extracted by the face prior; the face distribution code is converted into a feature distribution matching with the teaching image to obtain a preliminary prior feature map.

[0065] Step S120: Compress the teaching image using pyramid convolution and based on semantic information to obtain a compressed image.

[0066] Specifically, it may include the following steps:

[0067] S1. The teaching feature map is processed by shallow convolution to obtain a convolutional feature map.

[0068] S2. Input the convolutional feature map into the first branch of the pyramid convolution, and fuse the high and low layer information of the convolutional feature map to generate a first feature map containing multiple semantic layer information.

[0069] S3. Input the convolutional feature map into the second branch of the pyramid convolution to obtain a second feature map based on the low channel capacity space.

[0070] S4. Aggregate the first feature map and the second feature map, and downsample the aggregated feature map to obtain a compressed image.

[0071] Step S130: Embed the prior features in the preliminary prior feature map into the compressed image, wherein the prior features are features in the teaching image that match the preset prior conditions.

[0072] Specifically, it may include the following steps:

[0073] S1. The compressed image is processed by a super-resolution module to generate a first multi-scale depth feature map.

[0074] S2. Input the preliminary prior feature map into a dual-path prior feature converter to transform and align the features of the preliminary prior feature map and generate a second multi-scale depth feature map.

[0075] S3. Fuse the first multi-scale depth feature map and the second multi-scale depth feature map, and embed the prior features in the preliminary prior feature map into the super-resolution depth reconstruction process.

[0076] Step S140: Based on the attention mechanism, multi-scale salient feature extraction super-resolution performs shallow feature extraction on the compressed image and deep feature learning that embeds prior image features, and reconstructs the image.

[0077] Specifically, it may include the following steps:

[0078] S1. Extract shallow features from the compressed image.

[0079] S2. Extract deep residual multi-scale features from the compressed image and aggregate deep residual multi-scale features through skip connections.

[0080] S3. In the process of multi-scale feature extraction of depth residuals, the prior features in the preliminary prior feature map are fused to perform image depth reconstruction.

[0081] S4. Based on the upsampling layer and the convolutional layer of image reconstruction, the shallow features and the fused depth residual multi-scale features are used to reconstruct the reconstructed image corresponding to the compressed image.

[0082] Furthermore, after obtaining the reconstructed image, the present invention can also transmit the reconstructed image to a second terminal for the second terminal to display the reconstructed image.

[0083] Furthermore, the present invention may also include the following steps:

[0084] S1. Train a teacher network based on the teaching image and the reconstructed image, wherein the teacher network is used to generate a teaching reconstructed image based on the input teaching image.

[0085] S2. Based on the first loss between the teaching image and the teaching reconstructed image, the second loss between the teaching image and the reconstructed image of the student network, and the third loss between the teaching reconstructed image and the reconstructed image of the student network, a new reconstructed image corresponding to the teaching image is regenerated.

[0086] To describe the invention in more detail, specific examples will be used to illustrate its practical application.

[0087] Reference Figure 2 This invention provides an example flowchart of an image compression and reconstruction method based on super-resolution and prior knowledge. (Refer to...) Figure 3 The embodiments of the present invention provide explanatory diagrams of the variables used in the following processes and their meanings.

[0088] 1. Main assumptions.

[0089] In a synchronous classroom learning environment, from an overall structural perspective, it includes synchronous classrooms at supporting schools with both the main lecturer and local students, as well as synchronous classrooms at recipient schools with students from other locations, such as... Figure 4 As shown, the images or video frames from the supporting school's synchronous classroom are compressed and reconstructed using the image compression model (IC Model) and the image reconstruction model (SR Model) before being transmitted to the recipient school's synchronous classroom. Similarly, images or video frames from the recipient school's synchronous classroom can be transmitted to the supporting school's synchronous classroom through the same process.

[0090] From a network architecture perspective, the initial instructional images or video frames generated in the learning environment are defined as... The network is divided into teacher networks and student networks, where "teacher" and "student" are abstract concepts in the knowledge distillation process, such as... Figure 5 As shown. The teacher network is based on I. h Image priors (IPs) are assigned to images based on their inherent saliency characteristics. This IP is then compressed by the image compression module (IC) before proceeding to the super-resolution image reconstruction module (IR). Simultaneously, the corresponding text or face recognition model is extracted and invoked to obtain prior knowledge for that image type. Finally, this prior knowledge is embedded into the deep reconstruction process of IR, completing dual-path adaptive prior-guided super-resolution image reconstruction. The student network utilizes knowledge distillation to propagate prior information from the teacher network to the student network, effectively avoiding the impact of insufficient prior estimation on image reconstruction.

[0091] Specifically, the IP module is the prerequisite for extracting prior knowledge of images. It extracts key image features to classify text images and face images, and determines the prior information P = {P} for different categories after classification. t ,P fThis will be added and embedded in subsequent reconstruction modules to guide the reconstruction of high-resolution images.

[0092] The IC module is designed to compress images, reducing the amount of data transmitted to obtain low-resolution images. The key component responsible for image compression in the IC is the High-Efficiency Image Compression (EIC) module. The EIC consists of three parts: a traditional convolutional module, a pyramid feature fusion (PFM) module, and a downsampling (DM) module. This module can preserve the effective features of the image to the greatest extent possible while compressing it. Internally, the IC has two paths: Path A combines traditional compression methods with the EIC to achieve further compression and improve the compression ratio; Path B directly uses the EIC to compress the image to optimize the reconstruction of visual quality.

[0093] However, the IC (Integrated Image Capture) process actually causes some information loss in the image, which poses a significant challenge for image decompression. Image decompression essentially involves adding high-frequency details back to the compressed image to generate a high-resolution image, a process fundamentally similar to image super-resolution. Therefore, this invention combines super-resolution algorithms to reconstruct the compressed image to obtain a reconstructed image. Furthermore, the network employs an attention mechanism to model the dependencies between feature channels, focusing on the more important feature channels. Simultaneously, based on classification information in the IP, it adaptively selects prior models for different types of images, utilizing the corresponding prior model information P. t With P f Provides guidance for image reconstruction. Utilizes the reconstructed I S For prior P t With P f Refinement and continuous updates of prior information preserve more effective and key features of the image, enabling better reconstruction of compressed images.

[0094] 2. Construction of a teaching image compression and reconstruction model based on super-resolution algorithm and prior knowledge.

[0095] 2.1 Structure of the model.

[0096] 2.2 Model workflow.

[0097] 2.2.1 Image data acquisition.

[0098] Image or video frame data originates from teaching image data I h The dataset includes teaching images primarily composed of text and teaching images primarily composed of teacher and student facial images. After acquiring the image data, it is first preprocessed and standardized. Then, a model ranking method is used to select features from the image data, thereby generating two types of image data: text images and facial images, which form the training set.

[0099] 2.2.2 Image prior type acquisition and prior knowledge embedding.

[0100] 1. Image prior type acquisition.

[0101] To fully utilize prior information from various types of image data, such as text information and facial texture information, this invention designs a dual-path prior type acquirer, such as... Figure 6 As shown.

[0102] First, the features of the input image are determined. Represents an image prior feature classifier, which takes the input image I as an example. h Divided into facial images and text images, I t Images representing text types, I f Images representing face types are sent to different prior streams, as expressed by the formula:

[0103]

[0104] Next, in the dual-path prior stream, the rich information contained in the pre-trained model is used to generate prior information. Specifically, this invention designs a prior information generator PG that includes a text prior (TP) and a face prior (FP), extracting deep prior features P from text images and face images respectively. t With P f The network, guided by the deep prior knowledge P, can adaptively reconstruct compressed images and restore their detailed information during subsequent deep super-resolution image reconstruction. TP consists of a text recognition model CRNN, which can predict the probability of classifying text characters. TP generates a |T|-dimensional probability vector sequence, where |T| represents the number of characters learned by the prior model. FP consists of a face recognition model VGGFace. This pre-trained model encapsulates rich and diverse prior information and can match the closest potential face distribution code based on the input image.

[0105] Finally, since the pre-trained prior model is designed for image recognition tasks rather than image reconstruction tasks, it will generate prior information P. t With P f Integrating prior features into the image reconstruction process is challenging. To address the distribution discrepancies and feature mapping mismatches between prior features and actual image features, this invention designs a dual-path prior feature converter (DPT), where DPT-T and DPT-F can convert P... t With P f Transform into exploitable features and To be effectively embedded into the image reconstruction process, it can be expressed by the following formula:

[0106]

[0107]

[0108] Among them, (α) t ,β t ) and (α f ,β f ) are its transformation coefficients. This dual-path prior feature converter converts the output prior features into a feature distribution that is applicable to the image reconstruction features, thereby facilitating image reconstruction.

[0109] 2. Image prior knowledge embedding.

[0110] In the image reconstruction module, this invention proposes to utilize both the super-resolution reconstruction branch (SR) and the image prior branch to jointly complete the image reconstruction operation. An integrated module is designed to combine prior knowledge and original information to improve the recovery performance and generalization ability of image reconstruction. For example... Figure 7 As shown, the SR branch can reproduce the high-resolution image HR based on the input low-resolution image LR and the prior information generator PG guided features.

[0111] First, the compressed image LR is processed by the super-resolution module SRB of the SR branch to generate the output multi-scale depth feature map F. s The high-resolution image HR is processed by the prior information generator PG to generate a preliminary prior feature map, which is then transformed and aligned by the dual-path prior feature converter DPT to generate its depth feature map F. p .

[0112] Secondly, in order to reconstruct a high-fidelity and reliable restored image, this invention utilizes F with original image features. s With F that has prior knowledge p To adjust the generative model. The two deep feature maps are processed by an inconsistency function f. i After mapping, a feature map is obtained that integrates prior knowledge and the original image information. The formula for guiding image reconstruction is as follows:

[0113]

[0114] Finally, the feature maps are fused. By combining the original SR branch features, the rich and diverse details provided by prior knowledge are used to guide image reconstruction, reducing the difficulty of image restoration.

[0115] 2.2.3 Image compression based on semantic information extraction.

[0116] Image compression aims to reduce image data size while preserving key information. Maintaining effective high- and low-level semantic information during compression is beneficial for image reconstruction. The acquisition of semantic information typically relies on convolutional kernel extraction. CNN networks often use stacked convolutional layers with downsampling to obtain the receptive field. However, using the same convolutional kernel at all locations often only yields contextual information of a fixed size, limiting the kernel's feature extraction capabilities. Therefore, this invention introduces pyramid convolution into the network's compression module to retain more effective image features. Kernels with smaller receptive fields focus on more detailed image information, while kernels with larger receptive fields focus on more comprehensive contextual information. The two types of kernels complement each other while minimizing computational overhead.

[0117] The image compression module EIC can directly and efficiently compress images, or it can be used in conjunction with traditional compression schemes. For example... Figure 8 As shown, the high-resolution image input into EIC The feature map F0 is obtained after shallow convolution. It is then fed into two branches: a pyramid feature fusion module (PFM) that fuses high- and low-level information to generate a structure containing multiple semantic layers, whose feature representation is F. PFM And the shallow branch of the low channel capacity space, whose features are represented as F1. Then, these features are aggregated to obtain feature F3, which contains rich contextual information. The formula is as follows:

[0118] F3 = f concat (F1,F PFM )

[0119] The feature extraction process within the pyramid convolution of PFM is as follows:

[0120] 1. Set the number of layers in the pyramid convolution. Here, we set it to 4 layers from L1 to L4. The kernel size of each level K1 to K4 increases continuously, while the kernel depth decreases continuously.

[0121] 2. A grouped convolution method is used to group the input feature maps and apply kernels of different depths to generate feature maps. Among them MF i (·) represents the size of the input feature map, F p (·) Perform pyramidal convolution, K n 2 Indicates the space size of each level. The depth of each kernel level is represented by the following formula:

[0122]

[0123] 3. To extract image features more effectively, this invention introduces a channel attention module (SE) based on the pyramid convolution. The SE module can, based on the importance of the input feature channels, further enhance useful features, such as edge textures, while suppressing some unimportant features, while increasing the receptive field. The SE module first processes each feature map... After performing a compression operation and calculating the average, the global feature vector Z is obtained. g N is the number of layers, F sq (·) Perform the extrusion operation, as shown in the formula below:

[0124]

[0125] 4. Using Z g Learning FM g The feature weights of each channel are passed through two fully connected (FC) layers to obtain a weight vector Z between 0 and 1. w This process uses F fc (·)express.

[0126]

[0127] 5. Convert each channel of the original output channel With corresponding weights We perform weighting to obtain the new weighted features. F w (·) represents the operation of assigning weights. The formula is expressed as:

[0128]

[0129] 6. Concatenate the feature maps with reassigned weights to obtain a feature map that has undergone one pyramid convolution. The number of pyramid convolution groups is j, [i0, i1, ..., i j-1 [These represent the features generated by different numbers of pyramid convolutional groups.]

[0130] 2.2.4 Image reconstruction based on super-resolution.

[0131] The main direction for high-quality image compression and reconstruction is currently to design a reconstruction network that matches the compression process and utilize deep learning algorithms instead of increasing hardware costs. This invention designs a multi-scale salient feature extraction super-resolution network (MCAN) based on an attention mechanism to reconstruct compressed images. Its structure can be referenced from [reference needed]. Figure 9 This structure can extract multi-scale features from images, maximizing the recovery and reconstruction of compressed images and effectively improving the visual quality of the reconstructed images. MCAN consists of three parts: shallow feature extraction, deep feature learning, and reconstruction.

[0132] The first step is shallow feature extraction, which involves processing the input low-resolution image I. l Image shallow feature extraction F0 = f shallow (I l ), where f shallow (·) indicates a shallow feature extraction layer.

[0133] Secondly, there is deep feature extraction. This network includes multiple multi-scale feature extraction residual groups (MGs) and a deep feature fusion module (IFM). The IFM uses skip connections to aggregate more and more effective features. Deep residual multi-scale feature extraction consists of the following four steps:

[0134] 1. Initialize the MG layer Feature extraction is performed on shallow features to obtain initial features.

[0135] 2. Continuously iterate the MG layer to complete the deep extraction of image features, where This represents the (n-1)th iteration of the MG layer, where F1 represents the feature extraction of the initial MG layer. n The features extracted through n MG iterations are expressed by the formula:

[0136]

[0137] 3. Simultaneously, this invention introduces adaptive weights in the skip path, and multi-scale features enter IFM-MG via skip connections, w i F represents the multi-scale feature of the residual at layer i. i The weights are used to adaptively learn salient features. concat (·) Feature fusion is achieved, expressed by the formula:

[0138] F IFM =f concat [w1F1,w2F2,…,w n F n-1 ,F n ]

[0139] 4. IFM-MG fusion main path feature F n And the multi-hop feature yields F IFM F df Representing F IFM The depth features output by F0.

[0140] F df =f concat (F IFM ,F0)

[0141] The MG network internally consists of multiple multi-channel attention modules (MCAB) and an IFM, which improves the network's representational power through residual learning. Within the MCAB, multiple features are obtained through group convolution and multi-connection mechanisms, while a dual attention module (EAB) is introduced to obtain salient feature representations. The EAB contains a CA channel attention module and an ESA spatial attention module, which are cascaded together to adaptively readjust spatial and channel features. Feature extraction in the MCAB consists of the following three steps:

[0142] 1. Input features are used to generate features through convolutional layers of different scales. Introducing custom weights [λ1,λ2,λ3,λ4], feature F is obtained by fusing multiple features generated by convolutional layers of different scales. c This can be expressed as a public announcement:

[0143]

[0144] 2. Use the CA model to obtain the attention weights for each channel, and then combine these weights with the initialized features. Figure X Linear combination is used to obtain a channel-weighted feature map.

[0145] 3. Input the channel-weighted feature map into the SA model to obtain spatial attention weights, and linearly combine the input feature map and spatial attention weights to obtain the final output feature map F. β θ c With θ s Representing the channel attention weights and spatial attention weights, f(·) represents the value used to calculate the final modulation feature map F. β The modulation function is expressed by the formula:

[0146] F β =f(θ) c ,θ S ,X)

[0147] Finally, there is the reconstruction module, which extracts the final features F using an upsampling layer and an image reconstruction convolutional layer. ff F SR (·) represents super-resolution reconstruction of the image, expressed by the formula:

[0148] F ff =F SR (I l )

[0149] The image output by this network It achieves excellent visual reconstruction results. Based on the above functions, this super-resolution network can restore compressed images to high-quality images to a great extent.

[0150] 2.2.5 Prior knowledge propagation based on knowledge distillation.

[0151] To avoid inaccurate prior knowledge estimation and enable the network to utilize image prior knowledge without prior estimation, this invention designs a prior knowledge propagation scheme based on knowledge distillation. Specifically, the network is divided into a teacher network and a student network. Prior knowledge is propagated from the teacher to the student through knowledge distillation. The teacher and student networks have the same backbone architecture, but the teacher network embeds image prior information to effectively guide image reconstruction. Prior knowledge propagation based on knowledge distillation consists of the following steps:

[0152] 1. Input image I HR First, the system enters the teacher network; this process utilizes prior information about the image, specifically the P-process. f represents the reconstructed image generated by the teacher network. T (·) represents the teacher network image compression and reconstruction process. The teacher network is represented by the following formula:

[0153]

[0154] 2. After pre-training the teacher network, the characteristics and outputs of the teacher network are used to constrain and guide the training of the student network. Let f represent the reconstructed image generated by the student network, D represent the distillation information of the image, and f S (·) represents the image compression and reconstruction process of the student network. The student network can be represented as:

[0155]

[0156] 3. Three types of losses are used to optimize image reconstruction: the loss between the teacher network and the student network in reconstructing images. The loss between the teacher network reconstructed image and the real-world scene image The loss between the student network-reconstructed image and the real-world scene image

[0157] Reference Figure 10 This invention provides an image compression and reconstruction apparatus based on super-resolution and prior knowledge, comprising:

[0158] An image acquisition unit is used to acquire teaching images from the first terminal;

[0159] The prior feature acquisition unit is used to input the teaching image into a preset dual-path prior type acquirer to obtain a preliminary prior feature map. The dual-path prior type acquirer is used to classify the input image to obtain a first image mainly composed of text and a second image mainly composed of teacher and student portraits, and to convert the feature distribution of the first image or the second image into a feature distribution matching the input image.

[0160] An image compression unit is used to compress the teaching image using pyramid convolution and based on semantic information to obtain a compressed image;

[0161] A prior feature embedding unit is used to embed prior features from the preliminary prior feature map into the super-resolution depth reconstruction process, wherein the prior features are features in the teaching image that match preset prior conditions.

[0162] The image reconstruction unit is used for multi-scale salient feature extraction based on attention mechanism to perform shallow feature extraction on compressed images and deep feature learning to embed prior image features, and then reconstructs the image.

[0163] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0164] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0165] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0166] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0168] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0169] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0170] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0171] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0172] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. An image compression and reconstruction method based on super-resolution and prior knowledge, characterized in that, include: Acquire teaching images from the first terminal; The teaching image is input into a preset dual-path prior type acquirer to obtain a preliminary prior feature map. The dual-path prior type acquirer is used to classify the input image to obtain a first image mainly composed of text and a second image mainly composed of teacher and student portraits, and to convert the feature distribution of the first image or the second image into a feature distribution that matches the input image. The teaching images are compressed using pyramid convolution and based on semantic information to obtain compressed images; The prior features in the preliminary prior feature map are embedded into the super-resolution depth reconstruction process. The prior features are the features in the teaching image that match the preset prior conditions. Attention-based multi-scale salient feature extraction super-resolution performs shallow feature extraction on compressed images and learns deep features by embedding prior image features, and then reconstructs the image.

2. The image compression and reconstruction method based on super-resolution and prior knowledge according to claim 1, characterized in that, The step of inputting the teaching image into a preset dual-path prior type acquirer to obtain a preliminary prior feature map includes: The teaching image is input into a preset dual-path prior type acquirer, and the text prior in the dual-path prior type acquirer identifies the teaching image as the first image; the probability vector sequence in the first image is extracted by the text prior; the probability vector sequence is converted into a feature distribution matching the teaching image to obtain a preliminary text prior feature map; or, The teaching image is input into a preset dual-path prior type acquirer, and the face prior in the dual-path prior type acquirer identifies the teaching image as a second image; the face distribution code in the second image is extracted by the face prior; the face distribution code is converted into a feature distribution matching with the teaching image to obtain a preliminary face prior feature map.

3. The image compression and reconstruction method based on super-resolution and prior knowledge according to claim 1, characterized in that, The teaching images are compressed using pyramid convolution and based on semantic information to obtain compressed images, including: The teaching feature map is subjected to shallow convolution to obtain a convolutional feature map; The convolutional feature map is input into the first branch of the pyramid convolution, and the high and low layer information of the convolutional feature map is fused to generate a first feature map containing multiple semantic layer information. The convolutional feature map is input into the second branch of the pyramid convolution to obtain a second feature map based on the low channel capacity space; The first feature map and the second feature map are aggregated, and the aggregated feature map is downsampled to obtain a compressed image.

4. The image compression and reconstruction method based on super-resolution and prior knowledge according to claim 1, characterized in that, The step of embedding the prior features from the initial prior feature map into the super-resolution depth reconstruction process includes: The compressed image is processed by a super-resolution module to generate a first multi-scale depth feature map; The preliminary prior feature map is input into a dual-path prior feature converter to transform and align the features of the preliminary prior feature map and generate a second multi-scale depth feature map. The first multi-scale depth feature map and the second multi-scale depth feature map are fused together, and the prior features in the preliminary prior feature map are embedded into the super-resolution depth reconstruction process.

5. The image compression and reconstruction method based on super-resolution and prior knowledge according to claim 1, characterized in that, The attention-based multi-scale salient feature extraction super-resolution method performs shallow feature extraction and deep feature learning embedding prior image features on the compressed image, and reconstructs the image to obtain a reconstructed image, including: Extracting shallow features from compressed images; Deep residual multi-scale features are extracted from compressed images, and deep residual multi-scale features are aggregated through skip connections; In the process of deep residual multi-scale feature extraction, the prior features in the preliminary prior feature map are fused to perform image depth reconstruction.

6. The image compression and reconstruction method based on super-resolution and prior knowledge according to claim 1, characterized in that, Also includes: A prior knowledge propagation scheme based on knowledge distillation is adopted, and a teacher network is trained according to the teaching image and the reconstructed image. The teacher network is used to generate teaching reconstructed images based on the input teaching image. Based on the teacher network, prior information is propagated from the teacher network to the student network using knowledge distillation. Based on the first loss between the teaching image and the teaching reconstructed image, the second loss between the teaching image and the reconstructed image of the student network, and the third loss between the teaching reconstructed image and the reconstructed image of the student network, a new reconstructed image corresponding to the teaching image is regenerated.

7. The image compression and reconstruction method based on super-resolution and prior knowledge according to claim 1, characterized in that, Also includes: The reconstructed image is transmitted to a second terminal for display.

8. An image compression and reconstruction apparatus based on super-resolution and prior knowledge, characterized in that, include: An image acquisition unit is used to acquire teaching images from the first terminal; The prior feature acquisition unit is used to input the teaching image into a preset dual-path prior type acquirer to obtain a preliminary prior feature map. The dual-path prior type acquirer is used to classify the input image to obtain a first image mainly composed of text and a second image mainly composed of teacher and student portraits, and to convert the feature distribution of the first image or the second image into a feature distribution matching the input image. An image compression unit is used to compress the teaching image using pyramid convolution and based on semantic information to obtain a compressed image; A prior feature embedding unit is used to embed prior features from the preliminary prior feature map into the super-resolution depth reconstruction process, wherein the prior features are features in the teaching image that match preset prior conditions. The image reconstruction unit is used for multi-scale salient feature extraction based on attention mechanism to perform shallow feature extraction on compressed images and deep feature learning to embed prior image features, and then reconstructs the image.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images