Image generation method and device
By extracting the decoded image features and matching them with high-quality features, the target image is generated, and the distortion and artifact problems introduced during the encoding process are solved, improving image quality and system performance.
Patent Information
- Application Number
- CN202411026988.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-29
AI Technical Summary
The loss and error of the information introduced by the image or video during the encoding process lead to the decoded image or video quality, resulting in the loss of blur, noise, block effect, ringing effect, ribbon effect and other problems.
By acquiring code stream information, decoded image features are extracted, and corresponding features are extracted from high-quality second images, the target image is generated to repair and enhance the original image, and to remove distortions and artifacts in the encoding process.
Significantly improve the quality of the decoding image, reduce noise and artifacts, enhance image details and edge information, and improve system flexibility and adaptability.
Smart Images

Figure CN119031084A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of multimedia technology, and specifically relates to a method and device for generating an image. Background Art
[0002] Encoding an image or video means removing intra-frame and inter-frame redundancy of the encoded frames of the image or video through encoding technology to achieve efficient data compression.
[0003] However, image encoding will produce distortion. Distortion refers to the information loss or error introduced by data compression technology during the image or video encoding process. As a result, if the encoded bitstream information is decoded, the decoded image or video may have problems such as blur, noise, block effect, ringing effect, color banding effect, etc., resulting in a decrease in image or video quality. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a method and device for generating an image, which can solve the technical problem of low quality of images or videos after existing decoding.
[0005] In a first aspect, an embodiment of the present application provides a method for generating an image, the method comprising:
[0006] Obtaining code stream information, where the code stream information is information obtained when encoding the original image;
[0007] extracting a first image feature from a first image, where the first image is an image obtained by decoding the code stream information;
[0008] Extracting a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image;
[0009] A target image is generated according to the first image feature and the second image feature.
[0010] In a second aspect, an embodiment of the present application provides a device for generating an image, the device comprising:
[0011] An acquisition module, used to acquire code stream information, wherein the code stream information is information obtained when encoding the original image;
[0012] A first extraction module, used to extract a first image feature from a first image, where the first image is an image obtained by decoding the code stream information;
[0013] A second extraction module is used to extract a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image;
[0014] A generating module is used to generate a target image according to the first image feature and the second image feature.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method provided in the first aspect are implemented.
[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method provided in the first aspect are implemented.
[0017] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method provided in the first aspect.
[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method provided in the first aspect.
[0019] In the image generation method and device of the present application, the code stream information obtained by encoding the original image can be obtained, and then the decoded information can be decoded to obtain the first image, the first image feature can be extracted from the first image, and then the second image feature corresponding to the first image feature can be obtained, and finally the target image can be generated based on the first image feature and the second image feature. Since the second image feature is a high-quality image feature, the first image can be repaired and enhanced. Therefore, through this method, the distortion and artifacts caused by the compression of the original image during the encoding process can be removed, and the quality of the decoded image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is one of the flowcharts of a method for generating an image provided by an embodiment of the present application;
[0021] Figure 2 This is a second flow chart of a method for generating an image provided by an embodiment of the present application;
[0022] Figure 3This is a third flow chart of a method for generating an image provided by an embodiment of the present application;
[0023] Figure 4 This is a fourth flow chart of a method for generating an image provided by an embodiment of the present application;
[0024] Figure 5 is a structural schematic diagram of an image generation device provided by another embodiment of the present application;
[0025] Figure 6 is a structural schematic diagram of an electronic device provided by another embodiment of the present application;
[0026] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0029] In order to solve the above technical problems, the present application provides a method for generating an image. The following is a detailed description of the method for generating an image provided by the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.
[0030] like Figure 1 As shown, Figure 1 : is a flow chart of a method for generating an image provided by an embodiment of the present application. The embodiment of the present application provides a method for generating an image, which may include:
[0031] S101, obtaining code stream information, wherein the code stream information is information obtained when encoding the original image; in this embodiment, the code stream information refers to information obtained after encoding the original image, and the code stream information may include image information and encoding parameters of the compressed original image.
[0032] The encoding in this embodiment refers to the process of converting image or video data into a specific format (usually a compressed format), and the original image may be a single image, or a matrix composed of pixels such as a video frame. Specifically, the original image may be an uncompressed image or video frame obtained directly from a shooting device or an image generation tool, or an image or video frame restored from a compressed format. The original image may be a two-dimensional plane image or video frame, or a three-dimensional image or video frame.
[0033] During the encoding process, the original image can be encoded using common encoding methods such as H.266 / VVC, H.265 / HEVC, H.264 / AVC, JPEG, etc., or it can be encoded based on a neural network. The encoded bitstream information can be transmitted to the cloud through the network for storage, or it can be saved in a local file of an electronic device.
[0034] S102, extracting a first image feature from a first image, where the first image is an image obtained by decoding the code stream information;
[0035] In this embodiment, after acquiring the code stream information, the code stream information may be decoded to obtain a first image. Compared with the original image, the decoded first image has distortion and quality degradation caused by compression. After obtaining the first image, a feature extraction network may be used to extract first image features of the first image from the first image. The first image features are used to characterize the content and structure of the first image; they may also characterize the type and degree of distortion of the first image compared with the original image. The first image features may include features such as edges, textures, and frequency components of the first image.
[0036] S103, extracting a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image;
[0037] In this embodiment, since the first image is a distorted image of low quality, the image detail information in the first image feature is not rich enough and contains more noise and artifacts. In order to improve the image quality of the decoded image, a second image feature that is closer to the first image feature in content can be obtained, and the second image feature and the first image feature are used together to generate a new target image.
[0038] In this embodiment, the second image feature corresponding to the first image feature may be acquired by using an offline dictionary priori method or an online dictionary priori method.
[0039] For example, if the second image features are extracted by an offline dictionary a priori, an offline dictionary can be constructed in advance. Specifically, multiple high-definition third images can be obtained in advance, and then multiple fourth image features can be extracted from the multiple third images and stored in the offline dictionary. Then, whenever a low-quality first image feature is extracted from the decoded first image, a feature matching algorithm can be used in the offline dictionary to match the extracted first image feature with the multiple fourth image features in the offline dictionary, and find the second image feature that is most similar to the first image feature from the multiple fourth image features.
[0040] As an optional embodiment, Figure 2 As shown, the electronic device includes an encoding end and a decoding end. The specific process is as follows:
[0041] Encoding side
[0042] S201: Take the original image as input.
[0043] S202: The input original image is compressed by an encoder to generate encoded bit stream information.
[0044] S203: Transmit the encoded bitstream information through the network or store it in the cloud for use at the decoding end.
[0045] Decoding end
[0046] S204: At the decoding end, the received code stream information is decoded to generate a decoded frame (105). The decoded frame is the first image mentioned above. The decoded frame has distortion caused by compression.
[0047] S205: The decoded first image has distortion and quality degradation caused by compression.
[0048] S206: Extracting a degraded image feature, ie, a first image feature, from the decoded frame (105).
[0049] S207: The extracted first image features are searched for corresponding high-definition second image features in the offline dictionary prior (108).
[0050] S208: The pre-stored high-definition feature information is saved in the form of constructing a key-value pair, where the key-value pair includes a key-value pair of a first image feature and a second image feature.
[0051] S209: Expand and enhance the found second image feature to generate a high-quality third image feature.
[0052] S210: Fusing the first image feature and the expanded second image feature to obtain a final high-quality target image.
[0053] S211: Output the processed target image and play it.
[0054] For example, if the second image feature is extracted by an online dictionary prior, the second image feature needs to be extracted from the original image corresponding to the first image. In this case, the second image feature is the image feature of the original image. Figure 2 The offline dictionary method shown is the online dictionary method. Figure 3 The difference shown can be made on the encoding side before encoding the original image:
[0055] S320 online dictionary feature compression: extract the second image feature from the input original image, compress the second image feature, and then store the compressed second image feature in the code stream information, wherein the compression method of the second image feature can be neural network compression, principal component analysis, entropy coding, etc.
[0056] Then on the playback side you can do:
[0057] S321 online dictionary prior and S322 online dictionary feature processing, that is, decoding the compressed second image features in the code stream information, and obtaining the initial second image features through the inverse process of principal component analysis, neural network expansion, upsampling, feature extraction network, etc.
[0058] In this way, after the electronic device obtains the code stream information, it can decode the image information in the code stream information to obtain the first image, and decompress the compressed second image features, that is, the second image features can be obtained through online dictionary feature processing.
[0059] Among them, since the texture information, edge information and other details of the second image are richer than those of the first image, the content of the second image feature is close to the first image feature, and the image detail information of the second image feature is more than that of the first image feature, and the noise and artifacts contained in the second image feature are less than those of the first image feature, therefore, by combining the second image feature and the first image feature, a target image can be generated that is the same or similar to the first image in content but has richer details and less noise and artifacts.
[0060] By obtaining high-definition second image features through the above offline dictionary or online dictionary, the quality of the decoded image can be improved, noise and artifacts can be reduced, and the flexibility and adaptability of the system can be enhanced during the image and video decoding process. The offline dictionary does not need to change the encoding method of the image. Only adding an offline dictionary at the decoding end can improve the clarity of the decoded image. The online dictionary can further restore the target image to the original image as much as possible, while solving various inter-frame continuity and single-frame quality problems caused by encoding.
[0061] In some embodiments, the acquiring a second image feature corresponding to the first image feature includes:
[0062] In the case where the second image is a plurality of third images, extracting at least one fourth image feature from each of the third images to obtain a plurality of fourth image features;
[0063] A fourth image feature having the highest matching degree with the first image feature is obtained from the multiple fourth image features as the second image feature.
[0064] In this embodiment, if the second image feature is extracted by an offline dictionary a priori, an offline dictionary including multiple fourth image features can be constructed in advance, wherein the offline dictionary includes multiple fourth image features, multiple high-definition third images can be acquired in advance, and then at least one fourth image feature can be extracted from each third image to obtain multiple fourth image features.
[0065] Then, whenever a first image feature of lower quality is extracted from the decoded first image, a feature matching algorithm can be used in the offline dictionary to match the extracted first image feature with multiple fourth image features in the offline dictionary, and find the second image feature that is most similar to the first image feature from the multiple fourth image features. A neural network can be used for feature matching. The neural network can include a transformer network, a convolutional neural network, a recurrent neural network, or a combination thereof.
[0066] For example, in the offline dictionary, a unique index can be assigned to each fourth image feature, and then the index of the second image feature with the highest matching degree to the first image feature is predicted by training the neural network model, so as to obtain the second image feature based on the index.
[0067] In this embodiment, the high-definition feature corresponding to the first image feature can be quickly acquired through the mapping relationship in the offline dictionary.
[0068] S104: Generate a target image according to the first image feature and the second image feature.
[0069] In the embodiment of the present application, after obtaining the first image feature and the second image feature, the extracted first image feature and the corresponding second image feature can be used to generate a higher quality target image through a specific fusion or restoration algorithm. Thus, the high-quality second image feature is used to compensate for the information loss and distortion in the first image, thereby significantly improving the image quality.
[0070] In the present application, the code stream information obtained by encoding the original image can be obtained, and then the decoded information can be decoded to obtain the first image, the first image feature can be extracted from the first image, and then the second image feature corresponding to the first image feature can be obtained, and finally the target image can be generated based on the first image feature and the second image feature. Since the second image feature is a high-quality image feature, the first image can be repaired and enhanced. Therefore, through this method, the distortion and artifacts caused by the compression of the original image during the encoding process can be removed, and the quality of the decoded image can be improved.
[0071] In some embodiments, S104 includes:
[0072] Inputting the second image feature into a feature expansion network to obtain a third image feature output by the feature expansion network, wherein the dimension of the third image feature is greater than the dimension of the second image feature;
[0073] generating a target image feature according to the third image feature and the first image feature;
[0074] The target image is generated according to the target image features.
[0075] In this embodiment, the second image feature is an image feature stored in a pre-constructed dictionary. For the convenience of storage, the second image feature may be a compressed low-dimensional feature. Therefore, after obtaining the second image feature, the second image feature can be expanded to obtain a third image feature of a higher dimension. Image expansion can reconstruct or enhance the details and quality in the feature. Among them, the feature expansion network can be a neural network structure for expanding compressed or low-dimensional features into high-dimensional features. It gradually restores high-frequency details and structural information in the image through a series of convolution, deconvolution, upsampling and other operations. The dimension of the image feature refers to the number of elements or data points in the quantity. The larger the dimension of the image feature, the more information in the image feature. For example, compared with the second image feature, the third image feature can contain more image information in dimensions such as edges, textures, shapes, and colors.
[0076] For example, image features can be extracted from an image through a feature extraction network, the feature extraction network extracts features from the image through multi-layer convolution operations, and the feature expansion network expands low-dimensional features into high-dimensional features through multi-layer deconvolution operations to restore high-frequency details and structural information in the image. Specifically, feature expansion can be expanded through a first decoder, and extracting first image features from a first image can be accomplished through a first encoder, and the form of the first decoder and the form of the first encoder can be symmetrical.
[0077] After the third image feature is obtained by feature expansion, the third image feature and the first image feature can be fused into a target image feature, and then a decoder or a generator network is used to convert the target image feature representation into a target image.
[0078] In this embodiment, since the second image feature is originally a high-definition image feature, the texture and edge information of the image can be further supplemented and enhanced during its expansion. Therefore, the target image corresponding to the target image feature obtained by combining the third image feature and the first image feature can effectively remove artifacts and noise in the image compared to the first image, thereby obtaining a high-quality target image.
[0079] In some embodiments, generating a target image feature according to the third image feature and the first image feature includes:
[0080] Acquire a fusion parameter of the first image feature and the second image feature;
[0081] Obtaining a weight parameter of the third image feature;
[0082] The target image feature is generated according to the fusion parameter, the third image feature and the weight parameter.
[0083] In this embodiment, after obtaining the first image feature and the third image feature, the third image feature and the first image feature may be fused into a target image feature, so that the target image feature includes both the content of the first image feature and the content of the third image feature.
[0084] Specifically, the fusion process can be expressed as follows:
[0085] F hd =F d +(a*F d +b)*w (1)
[0086] Among them, a and b are the fusion parameters of the first image feature and the second image feature, which can be extracted through a neural network. w is the weight parameter of the third image feature, and its value is between 0 and 1. d is the third image feature, F hd is the target image feature.
[0087] In the above formula, parameter a is used to control the contribution of the second image feature in the fusion process, while parameter b is used to further adjust the fusion process to ensure that the fused features can better match the distribution of the target image; the weight parameter w is used to adjust the proportion of the second image feature and the first image feature in the final fusion result. The value is between 0 and 1, and the value close to 1 indicates a preference for the second image feature, and the value close to 0 indicates a preference for the first image feature.
[0088] Therefore, users can find a balance between fidelity and image quality by flexibly adjusting the weight parameter w, so that the output target image can meet the user's needs.
[0089] In some embodiments, the bitstream information includes a quantization parameter of the original image;
[0090] Before generating the target image according to the first image feature and the second image feature, the method further includes:
[0091] When the quantization parameter does not belong to the quantization threshold interval, adjusting the parameter value of the weight parameter of the third image feature to a first parameter value, wherein the quantization parameter corresponding to the first parameter value belongs to the quantization threshold interval;
[0092] The generating a target image according to the first image feature and the second image feature comprises:
[0093] The target image is generated according to the first parameter value, the first image feature and the second image feature.
[0094] In this embodiment, in the process of encoding the original image, i.e., compressing, quantization is the process of simplifying the transformed coefficients. The quantization parameter (QP) determines the size of the quantization step. The larger the quantization parameter, the more information is lost during the image encoding process, the higher the compression ratio, but the greater the distortion. Therefore, in the process of encoding the original image, a higher quantization parameter will lead to a higher compression rate, but will introduce greater distortion, image quality degradation, and loss of details. A lower quantization parameter will retain more details, with less distortion, but a lower compression rate.
[0095] After the original image is encoded with a preset quantization parameter, the encoded quantization parameter can be stored in the bitstream information. If the bitstream information needs to be decoded, the quantization parameter in the bitstream information can be first obtained, and then it can be determined whether the quantization parameter belongs to the preset quantization threshold interval. If the quantization parameter belongs to the preset quantization threshold interval, the bitstream information can be decoded based on the quantization parameter; if the quantization parameter does not belong to the quantization threshold interval, the quantization parameter needs to be adjusted so that the adjusted quantization parameter enters the quantization threshold interval, and the bitstream information is decoded based on the adjusted quantization parameter. Among them, a quantization parameter can correspond to an image or a partial area in an image.
[0096] Specifically, the quantization parameter can be adjusted by adjusting the weight parameter w of the third image feature, because the weight parameter is used to adjust the proportion of the second image feature and the first image feature in the final fusion result. When the weight parameter is increased, the quantization parameter becomes smaller accordingly; when the weight parameter is decreased, the quantization parameter becomes larger accordingly.
[0097] In addition, the quantization parameter of the original image may be normalized to obtain a normalized value, and whether the quantization parameter belongs to the quantization threshold range may be determined by the size of the normalized value.
[0098] Therefore, when the quantization parameter does not belong to the quantization threshold interval, the parameter value of the weight parameter of the third image feature can be adjusted from the initial parameter value to the first parameter value so that the quantization parameter corresponding to the first parameter value belongs to the quantization threshold interval.
[0099] In the above manner, by adjusting the weight parameter to dynamically adjust the quantization parameter in the decoding process, the best balance between the compression rate and quality of the image or video can be found.
[0100] In some embodiments, after generating the target image according to the first image feature and the second image feature, the method further includes:
[0101] displaying the target image;
[0102] In the case of receiving a parameter modification input of the target image from the user, in response to the parameter modification input, adjusting the parameter value of the weight parameter of the third image feature to a second parameter value;
[0103] The target image is updated according to the second parameter value, the first image feature and the second image feature.
[0104] In this embodiment, similarly, compared with Figure 2 In the manner shown, this embodiment has Figure 4 As shown in the figure, after the original image is encoded with a preset quantization parameter, the encoded quantization parameter can be stored in the bitstream information. If the bitstream information needs to be decoded, the quantization parameter QP in the bitstream information can be first obtained (S215), and the target image is generated and displayed based on the quantization parameter.
[0105] In the process of decoding the bitstream information, a higher quantization parameter will introduce higher distortion and noise, but can ensure that the target image and the original image have a higher similarity; a lower quantization parameter will cause the target image and the original image to have a lower similarity, but the distortion of the target image is smaller.
[0106] After the target image is displayed, the user can observe the target image on the display interface, and receive the parameter modification input of the user intensity adjustment (S216), and adjust the quantization parameter QP (S217) in response to the parameter modification input. If the user believes that the authenticity of the target image is too low or the clarity is too high, the parameter modification input can be made to the electronic device, and the quantization parameter can be reduced by increasing the value of the weight parameter, thereby increasing the proportion of the first image feature in the target image to improve the similarity between the target image and the original image or reducing the clarity. On the contrary, if the user believes that the clarity of the target image is too low, the quantization parameter can also be increased by reducing the value of the weight parameter by making a parameter modification input to the electronic device, thereby increasing the proportion of the second image feature in the target image to improve the clarity of the target image.
[0107] Specifically, the user may adjust the parameter value of the weight parameter of the third image feature from the initial parameter value to the second parameter value by modifying the input, so that the target image corresponding to the second parameter value matches the user's needs.
[0108] In the above manner, the user can dynamically adjust the quantization parameters in the decoding process by manually adjusting the weight parameters, so that the decoded image or video better meets the user's needs.
[0109] Figure 5 is a schematic diagram of the structure of an image generation device provided by another embodiment of the present application, such as Figure 5 As shown, the image generating device may include:
[0110] The acquisition module 501 is used to acquire code stream information, where the code stream information is information obtained when encoding the original image;
[0111] A first extraction module 502, configured to extract a first image feature from a first image, where the first image is an image obtained by decoding the bitstream information;
[0112] A second extraction module 503 is configured to extract a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image;
[0113] The generating module 504 is configured to generate a target image according to the first image feature and the second image feature.
[0114] In the present application, the code stream information obtained by encoding the original image can be obtained, and then the decoded information can be decoded to obtain the first image, the first image feature can be extracted from the first image, and then the second image feature corresponding to the first image feature can be obtained, and finally the target image can be generated based on the first image feature and the second image feature. Since the second image feature is a high-quality image feature, the first image can be repaired and enhanced. Therefore, through this method, the distortion and artifacts caused by the compression of the original image during the encoding process can be removed, and the quality of the decoded image can be improved.
[0115] In another optional example, the generating module 504 includes:
[0116] an expansion unit, configured to input the second image feature into a feature expansion network to obtain a third image feature output by the feature expansion network, wherein the dimension of the third image feature is greater than the dimension of the second image feature;
[0117] A first generating unit, configured to generate a target image feature according to the third image feature and the first image feature;
[0118] The second generating unit is used to generate the target image according to the target image feature.
[0119] In another optional example, the bitstream information includes a quantization parameter of the original image;
[0120] The image generating device comprises:
[0121] A first adjustment module, configured to adjust the parameter value of the weight parameter of the third image feature to a first parameter value when the quantization parameter does not belong to the quantization threshold interval, wherein the quantization parameter corresponding to the first parameter value belongs to the quantization threshold interval;
[0122] The generating module 504 further includes:
[0123] A third generating unit is used to generate the target image according to the first parameter value, the first image feature and the second image feature.
[0124] In another optional example, the image generating device includes:
[0125] A display module, used for displaying the target image;
[0126] A second adjustment module is configured to, upon receiving a parameter modification input of the target image from a user, adjust the parameter value of the weight parameter of the third image feature to a second parameter value in response to the parameter modification input;
[0127] An updating module is used to update the target image according to the second parameter value, the first image feature and the second image feature.
[0128] In another optional example, the second extraction module 503 includes:
[0129] an extraction unit, configured to extract at least one fourth image feature from each of the third images to obtain a plurality of fourth image features when the second image is a plurality of third images;
[0130] An acquisition unit is used to acquire, from the multiple fourth image features, a fourth image feature that has the highest matching degree with the first image feature as the second image feature.
[0131] The image generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.
[0132] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0133] The image generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0134] Alternatively, if Figure 6 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 110, a memory 119, and a program or instruction stored in the memory 119 and executable on the processor 110. When the program or instruction is executed by the processor 110, each process of the above-mentioned image generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0135] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
[0136] Please refer to Figure 7, Figure 7 The hardware structure diagram of an electronic device for implementing an embodiment of the present application is shown in FIG. The electronic device 100 includes, but is not limited to, a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.
[0137] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 120 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0138] The network module 122 is used to obtain code stream information, which is information obtained when encoding the original image;
[0139] The processor 120 is configured to extract a first image feature from a first image, where the first image is an image obtained by decoding the code stream information;
[0140] The processor 120 is configured to extract a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image;
[0141] The processor 120 is configured to generate a target image according to the first image feature and the second image feature.
[0142] In the present application, the code stream information obtained by encoding the original image can be obtained, and then the decoded information can be decoded to obtain the first image, the first image feature can be extracted from the first image, and then the second image feature corresponding to the first image feature can be obtained, and finally the target image can be generated based on the first image feature and the second image feature. Since the second image feature is a high-quality image feature, the first image can be repaired and enhanced. Therefore, through this method, the distortion and artifacts caused by the compression of the original image during the encoding process can be removed, and the quality of the decoded image can be improved.
[0143] In another optional example, the processor 120 is further configured to:
[0144] Inputting the second image feature into a feature expansion network to obtain a third image feature output by the feature expansion network, wherein the dimension of the third image feature is greater than the dimension of the second image feature;
[0145] generating a target image feature according to the third image feature and the first image feature;
[0146] The target image is generated according to the target image features.
[0147] In another optional example, the bitstream information includes a quantization parameter of the original image;
[0148] The processor 120 is further configured to:
[0149] In a case where the quantization parameter does not belong to the quantization threshold interval, adjusting the parameter value of the weight parameter of the third image feature to a first parameter value, wherein the quantization parameter corresponding to the first parameter value belongs to the quantization threshold interval;
[0150] The target image is generated according to the first parameter value, the first image feature and the second image feature.
[0151] In another optional example, the display unit 126 is used to display the target image;
[0152] The user input unit 127 is used for adjusting the parameter value of the weight parameter of the third image feature to the second parameter value in response to receiving the parameter modification input of the user for the target image;
[0153] The processor 120 is configured to update the target image according to the second parameter value, the first image feature, and the second image feature.
[0154] In another optional example, the processor 120 is configured to, when the second image is a plurality of third images, extract at least one fourth image feature from each of the third images to obtain a plurality of fourth image features;
[0155] The processor 120 is configured to obtain, from the multiple fourth image features, a fourth image feature that has the highest matching degree with the first image feature as the second image feature.
[0156] It should be understood that in the embodiment of the present application, the input unit 124 may include a graphics processor (Graphics Processing Unit, GPU) 1241 and a microphone 1242, and the graphics processor 1241 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 126 may include a display panel 1261, and the display panel 1261 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 127 includes a touch panel 1271 and at least one of other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include two parts: a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0157] The memory 129 can be used to store software programs and various data. The memory 129 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 129 may include a volatile memory or a non-volatile memory, or the memory 129 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 129 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0158] The processor 120 may include one or more processing units; optionally, the processor 120 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 120.
[0159] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0160] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0161] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0162] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0163] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0164] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0166] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for generating an image, characterized in that: include: Obtaining code stream information, where the code stream information is information obtained when encoding the original image; extracting a first image feature from a first image, where the first image is an image obtained by decoding the code stream information; Extracting a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image; A target image is generated according to the first image feature and the second image feature.
2. The method according to claim 1, characterized in that The generating a target image according to the first image feature and the second image feature comprises: Inputting the second image feature into a feature expansion network to obtain a third image feature output by the feature expansion network, wherein the dimension of the third image feature is greater than the dimension of the second image feature; generating a target image feature according to the third image feature and the first image feature; The target image is generated according to the target image features.
3. The method according to claim 2, characterized in that The bitstream information includes a quantization parameter of the original image; Before generating the target image according to the first image feature and the second image feature, the method further includes: When the quantization parameter does not belong to the quantization threshold interval, adjusting the parameter value of the weight parameter of the third image feature to a first parameter value, wherein the quantization parameter corresponding to the first parameter value belongs to the quantization threshold interval; The generating a target image according to the first image feature and the second image feature comprises: The target image is generated according to the first parameter value, the first image feature and the second image feature.
4. The method according to claim 2, characterized in that: After generating the target image according to the first image feature and the second image feature, the method further includes: displaying the target image; In the case of receiving a parameter modification input of the target image from the user, in response to the parameter modification input, adjusting the parameter value of the weight parameter of the third image feature to a second parameter value; The target image is updated according to the second parameter value, the first image feature and the second image feature.
5. The method according to claim 1, characterized in that The extracting, from the second image, a second image feature corresponding to the first image feature comprises: In the case where the second image is a plurality of third images, extracting at least one fourth image feature from each of the third images to obtain a plurality of fourth image features; A fourth image feature having the highest matching degree with the first image feature is obtained from the multiple fourth image features as the second image feature.
6. An image generating device, characterized in that: include: An acquisition module, used to acquire code stream information, wherein the code stream information is information obtained when encoding the original image; A first extraction module, used to extract a first image feature from a first image, where the first image is an image obtained by decoding the code stream information; A second extraction module is used to extract a second image feature corresponding to the first image feature from a second image, wherein the second image is the original image or a third image other than the original image, the third image has more texture information and edge information than the first image, and the third image has less noise and artifacts than the first image; A generating module is used to generate a target image according to the first image feature and the second image feature.
7. The device according to claim 6, characterized in that The generation module comprises: an expansion unit, configured to input the second image feature into a feature expansion network to obtain a third image feature output by the feature expansion network, wherein the dimension of the third image feature is greater than the dimension of the second image feature; A first generating unit, configured to generate a target image feature according to the third image feature and the first image feature; The second generating unit is used to generate the target image according to the target image feature.
8. The device according to claim 7, characterized in that The bitstream information includes a quantization parameter of the original image; The image generating device comprises: A first adjustment module, configured to adjust the parameter value of the weight parameter of the third image feature to a first parameter value when the quantization parameter does not belong to the quantization threshold interval, wherein the quantization parameter corresponding to the first parameter value belongs to the quantization threshold interval; The generating module also includes: A third generating unit is used to generate the target image according to the first parameter value, the first image feature and the second image feature.
9. The device according to claim 7, characterized in that The image generating device comprises: A display module, used for displaying the target image; A second adjustment module is configured to, upon receiving a parameter modification input of the target image from a user, adjust the parameter value of the weight parameter of the third image feature to a second parameter value in response to the parameter modification input; An updating module is used to update the target image according to the second parameter value, the first image feature and the second image feature.
10. The device according to claim 6, characterized in that The second extraction module comprises: an extraction unit, configured to extract at least one fourth image feature from each of the third images to obtain a plurality of fourth image features when the second image is a plurality of third images; An acquisition unit is used to acquire, from the multiple fourth image features, a fourth image feature that has the highest matching degree with the first image feature as the second image feature.
Citation Information
Patent Citations
Image reconstruction method, related device and system
CN114697543A
Coding and decoding method and device
CN115700771A
Image processing method, device and equipment and computer readable storage medium
CN116452466A
Image restoration method and device, computer equipment, medium and program
CN117036178A
Image coding method, image decoding method, image coding device and image decoding device
CN117294854A
