Bitstream Processing Method, Apparatus, Terminal Device, and Storage Medium

Abstract

A bitstream processing method, apparatus, terminal device and storage medium, the method including determining an image quality evaluation area (S110), determining quality evaluation data for the quality evaluation area (S120), determining image quality information including the quality evaluation data (S130), and adding the image quality information to a bitstream to obtain a processed bitstream (S140).

Description

Technical Field

[0001] This application is filed based on a Chinese patent application with an application number of 202110832744.X and a filing date of July 22, 2021, and claims the benefit of priority to the Chinese patent application, and all the contents of the Chinese patent application are incorporated herein by reference.

[0002] This application relates to the field of communication technologies, for example, to a bitstream processing method, apparatus, terminal device, and storage medium.

Background Art

[0003] With the development of immersive media technology, more and more video applications can provide 360-degree panoramic videos for users to watch. In order to provide users with more flexible and appropriate video image services, image quality evaluation can be performed on images.

[0004] In some cases, when performing quality evaluation on an image, the image quality is calculated with one image as a unit, and the calculation amount is large. Therefore, how to improve the flexibility of image quality evaluation is an urgent technical problem to be solved currently.

Summary of the Invention

Problems to be Solved by the Invention

[0005] This application provides a bitstream processing method, apparatus, terminal device, and storage medium.

Means for Solving the Problems

[0006] In aspect 1, embodiments of this application include: determining an image quality evaluation area, determining quality evaluation data of the quality evaluation area, determining image quality information including the quality evaluation data, and adding the image quality information to a bitstream to obtain a processed bitstream. A bitstream processing method is provided.

[0007] In Embodiment 2, the embodiments of the present application include: obtaining a processed bitstream, analyzing the processed bitstream to obtain image quality information, and determining an image quality evaluation region and corresponding quality evaluation data based on the image quality information. A bitstream processing method is provided.

[0008] In Embodiment 3, the embodiments of the present application include: a first determination module configured to determine an image quality evaluation region, a second determination module configured to determine quality evaluation data for the quality evaluation region, a third determination module configured to determine image quality information including the quality evaluation data, and an addition module configured to add the image quality information to a bitstream to obtain a processed bitstream. A bitstream processing apparatus is provided.

[0009] In Embodiment 4, the embodiments of the present application include: an acquisition module configured to obtain a processed bitstream, an analysis module configured to analyze the processed bitstream to obtain image quality information, and a determination module configured to determine an image quality evaluation region and corresponding quality evaluation data based on the image quality information. A bitstream processing apparatus is provided.

[0010] In Embodiment 5, the embodiments of the present application include: a terminal device including one or more processors and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the bitstream processing method according to the embodiments of the present application. A terminal device is provided.

[0011] In Embodiment 6, an embodiment of the present application is a computer program is stored, and when the computer program is executed by a processor, any bitstream processing method in the embodiments of the present application is realized, a storage medium is provided.

[0012] For the above embodiments, other aspects and their implementation forms of the present application, more explanations will be provided in the description of the drawings, the mode for carrying out the invention, and the scope of the claims.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0014] To make the object, technical solution and advantages of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. Unless there is no conflict, the embodiments and features in the embodiments according to the present application can be arbitrarily combined with each other.

[0015] The steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer-executable instructions. And although the flowchart shows a logical order, in some cases, steps shown or described in an order different from the order here may be executed.

[0016] In one embodiment, FIG. 1 is a flowchart of a bitstream processing method according to an embodiment of the present application. The method can be applied when improving the flexibility of image quality evaluation by processing a bitstream. The method can be executed by a bitstream processing device, which is realized by software and / or hardware and can be integrated into a terminal device, and the terminal device may be a transmitting device that transmits a bitstream.

[0017] As shown in FIG. 1, the bitstream processing method according to the embodiment of the present application includes the following steps S110, S120, S130, and S140.

[0018] In S110, determine an image quality evaluation area.

[0019] The image quality evaluation area can be understood as the entire image to be evaluated, or a plurality of images to be evaluated, or a part of the area in the entire image to be evaluated. Here, without limiting to a part of the area, it can be determined according to a specific scene. Here, a part of the area may be an area formed after dividing the image to be evaluated into areas.

[0020] The number of image quality evaluation areas may be one or more. One quality evaluation area can correspond to one or more quality evaluation data.

[0021] This step does not limit how to determine the image quality evaluation area. For example, the image quality evaluation area can be determined based on the image features of the image to be evaluated.

[0022] In S120, determine the quality evaluation data of the quality evaluation area.

[0023] The quality evaluation data may be considered as the data obtained after performing quality evaluation on the quality evaluation area.

[0024] To determine the quality evaluation data, an image quality evaluation method can be adopted. The image quality evaluation method includes a full-reference quality evaluation method, a reduced-reference quality evaluation method, and a no-reference quality evaluation method. Here, it is not limited how to determine the image evaluation data based on the image quality evaluation method.

[0025] The quality evaluation data of the quality evaluation area may be one or more.

[0026] In S130, image quality information including the quality evaluation data is determined.

[0027] The image quality information may be considered as information representing the image quality to be evaluated.

[0028] After determining the quality evaluation data, in this step, the quality evaluation data can be added to the image quality information. Here, the content included in the image quality information is not limited.

[0029] In S140, the image quality information is added to the bitstream to obtain a processed bitstream.

[0030] After determining the image quality information, in this step, the image quality information can be added to the bitstream so that the terminal device receiving the processed bitstream can analyze the image quality information, and a processed bitstream can be obtained. The bitstream in this embodiment may be a media bitstream.

[0031] The bitstream processing method according to the embodiment of the present application improves the flexibility of image quality evaluation of the processed bitstream by determining image quality information including quality evaluation data and then adding the image quality information to the bitstream.

[0032] Based on the above embodiment, a modified example of the above embodiment is presented. For the sake of simplicity of description, only the differences from the above embodiment are described in the modified example.

[0033] In one embodiment, the number of the image quality evaluation regions is at least one, the image quality evaluation region includes the entire image region to be evaluated or a plurality of image regions to be evaluated, or the image quality evaluation region includes a part of the regions formed after region-dividing the image to be evaluated, and the image to be evaluated determines a division policy based on image features.

[0034] The image features may be information representing the image features, and here, the specific content of the image features is not limited. The image features may be the image type or the specific content in the image, etc.

[0035] The image quality evaluation region may be the entire image region to be evaluated, or a plurality of image regions to be evaluated, or a part of the regions formed after region-dividing one or more images to be evaluated. The part of the regions may be considered as one or more of the plurality of regions formed after region-dividing the image to be evaluated. The number of the regions included in the part of the regions is not limited, and which region in the regions after region-dividing the part of the regions is also not limited, and can be determined according to the actual scene.

[0036] In one embodiment, the image quality information includes an image quality evaluation method adopted to determine the quality evaluation data, a time stamp for generating the quality evaluation data, a device identifier for generating the quality evaluation data, an identifier of the quality evaluation data, i.e., a number of the quality evaluation data, the number of image quality evaluation regions, information of the original image, including a link to the original image or statistical information of a part of the original image, information of the original image where the image to be evaluated is a reconstructed image generated after processing the original image, a mathematical model adopted to perform no-reference image quality evaluation, image quality evaluation region division instruction information such as pqm_picture_division_idc, which may be image quality evaluation region division instruction information that is an image feature of the image to be evaluated such as pqm_picture_type, instruction information on the commonality of quality evaluation methods adopted for different image quality evaluation regions, such as pqm_mixed_measurement_flag, for indicating whether the quality evaluation methods adopted for different image quality evaluation regions are the same, instruction information on components to be evaluated in the image quality evaluation regions, information of scalable coding layers, including the number of scalable coding layers, the scalable coding layer to which the image quality evaluation region belongs, a quality evaluation method adopted for the i-th layer image in the image to be evaluated, such as pqm_PSNR[i], a quality evaluation method adopted for the n-th image quality evaluation region of the k-th layer image in the image to be evaluated, such as pqm_PSNR[i][j], where i, k, and n are positive integers, information of the image sequence including a plurality of images to be evaluated corresponding to the image quality evaluation region, the number of images included in the image sequence, the number of quality evaluation regions in each image within the image sequence, a quality evaluation method adopted to determine the j-th (j is a positive integer) image quality evaluation region within the image sequence, information of the m-th (m is a positive integer) quality evaluation region, a quality evaluation method adopted to determine the quality evaluation data of the h-th (h is a positive integer) quality evaluation region, and one or more of the number of quality evaluation regions in the g-th (g is a positive integer) image to be evaluated within the image sequence.

[0037] The image quality evaluation area division instruction information can be used to indicate the number of image quality evaluation areas and the calculation method of the quality evaluation data.

[0038] This application can calculate the quality evaluation score of one image (area), that is, the quality evaluation data, and may also calculate the quality evaluation scores of multiple images (areas) waiting for evaluation. The multiple images waiting for evaluation are, that is, an image sequence.

[0039] Multiple quality evaluation areas correspond to a group of quality evaluation data. The multiple quality evaluation areas may be multiple quality evaluation areas in one image waiting for evaluation, or may be quality evaluation areas in multiple images waiting for evaluation. The multiple quality evaluation areas may be considered as a quality evaluation area series.

[0040] This embodiment does not limit the combination method of the content included in the quality information, and those skilled in the art can determine it according to the actual application scenario.

[0041] In one embodiment, the image quality information further includes the division information of the image quality evaluation area and the instruction information indicating whether the division information exists. The division information of the image quality evaluation area includes one or more of the identifier of the quality evaluation area, the position of the quality evaluation area, and the size of the quality evaluation area.

[0042] In one embodiment, determining the image quality evaluation area includes determining the image quality evaluation area of the image waiting for evaluation based on the image features of the image waiting for evaluation.

[0043] Different image features correspond to different image quality evaluation areas. The specific method for determining the image quality evaluation area is determined based on the selected image features and is not limited here.

[0044] In one embodiment, adding the image quality information to the bitstream to obtain the processed bitstream includes the following steps.

[0045] Generate supplementary enhancement information based on the image quality information.

[0046] Add the supplementary enhancement information to the bitstream to obtain a processed bitstream.

[0047] When generating supplementary enhancement information based on the image quality information, the image quality information can be added to the supplementary enhancement information.

[0048] In one embodiment, obtaining a processed bitstream by adding the image quality information to the bitstream includes the following steps.

[0049] Write the image quality information to the media file, or write the image quality information to the media file as the level information of the corresponding image quality evaluation area.

[0050] Add the media file to the bitstream to obtain a processed bitstream.

[0051] In one embodiment, one image quality evaluation area corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting the same or different quality evaluation methods for one image quality evaluation area.

[0052] In one embodiment, the quality evaluation methods adopted for different image quality evaluation areas are the same or different.

[0053] In one embodiment, FIG. 2 is a flowchart of a further bitstream processing method according to an embodiment of the present application. The method can be applied when improving the flexibility of image quality evaluation. The method can be executed by a bitstream processing device, which is executed by software and / or hardware and can be integrated into a terminal device that receives the processed bitstream.

[0054] As shown in FIG. 2, the bitstream processing method according to an embodiment of the present application includes the following steps S210, S220, and S230.

[0055] In S210, obtain the processed bitstream.

[0056] In S220, analyze the processed bitstream to obtain image quality information.

[0057] Here, the specific means of analysis is not limited. The specific means of analysis is determined based on the specific means by which the terminal device that transmits the processed bitstream generates the processed bitstream. For example, obtain image quality information from the media file of the processed bitstream or obtain image quality information from the supplementary enhancement information.

[0058] In S230, determine an image quality evaluation area and corresponding quality evaluation data based on the image quality information.

[0059] In this step, the content included in the image quality information can be extracted to determine the image quality evaluation area and corresponding quality evaluation data. For example, by obtaining content such as image quality evaluation area division instruction information, the number of image quality evaluation areas, and the quality evaluation method to be adopted, the image quality evaluation area and quality evaluation data can be determined.

[0060] Contents not described in detail in this embodiment can be referred to the above embodiments, and the description is omitted here.

[0061] The bitstream processing method according to the embodiment of the present application effectively improves the flexibility of image quality evaluation by obtaining a processed bitstream including image quality information, analyzing the processed bitstream, and obtaining corresponding image quality evaluation regions and quality evaluation data.

[0062] Hereinafter, the present application will be described with examples. The bitstream processing method according to the embodiment of the present application may be considered as a processing method for adding image quality information to a media bitstream. The present application determines an image quality metric region, that is, an image quality evaluation region based on image features, that is, image characteristics, calculates a quality evaluation score for each image quality metric region, that is, quality evaluation data, and writes image quality information including the quality evaluation score corresponding to the image quality metric region into the media bitstream. An application that receives the media bitstream provides the image quality information corresponding to the image quality evaluation region in the media bitstream so that a more flexible and appropriate video image service can be provided based on the image quality information based on the image quality metric region (that is, the image quality evaluation region) provided by the encoding side.

[0063] With the development of immersive media technology, more and more video applications can provide 360-degree panoramic videos for users to watch. However, the human visual field range is generally about 120 degrees. The video screen that a user is actually looking at at a certain point in time is not a complete 360-degree panoramic video screen, but only a part of the screen corresponding to the user's current viewing point. At the same time, the perceptions of different users towards the screen are not exactly the same, and different interests are also held for different regions, different target objects or different colors on the screen. Therefore, when watching immersive videos, if it is possible to provide a more flexible viewing or processing method according to the user's needs and preferences in combination with the characteristics of the scene, or based on pre-settings, a video screen with more consistent image quality can be provided to the user, and the user can obtain a better experience effect. In ISO / IEC 23090-2 OMAF, quality levels related to quality are defined, indicating the relative quality order of the quality level regions. The ranking values can be derived from specific quality metrics. In ISO / IEC 23090-6, immersive media metrics also use quality rankings to measure the switching delay of viewpoints.

[0064] Image quality assessment is used to measure various distortions of images in the processes of collection, processing, compression, storage, transmission, and reproduction. The objective evaluation method of image quality means giving results based on digital operations by a certain mathematical model. Based on whether it is necessary to refer to the information of the image when calculating the image quality, the objective evaluation methods of image quality can be divided into three types: full-reference quality assessment, partial-reference quality assessment, and no-reference quality assessment. Full-reference and partial-reference quality assessments mainly calculate the quality of the reconstructed image by analyzing the features of the image and quantifying the differences between the original image and the reconstructed image. Currently, in the field of video encoding and decoding, the quality of video images is often evaluated by the full-reference image quality assessment method. Common objective evaluation methods for full-reference image quality include PSNR (Peak Signal to Noise Ratio), SSIM (structural similarity), MS-SSIM (Multi-Scale Structural SIMilarity), Video Quality Metric (VQM), and Mean Squared Error (MSE), etc. Based on the above image quality assessment methods, a series of improved quality assessment methods have been further derived, such as weighted Peak Signal-to-noise Ratio (wPSNR), weighted spherical Peak Signal-to-noise Ratio (WS-PSNR), Spherical Peak Signal-to-noise Ratio (S-PSNR), etc. Also, in ISO / IEC 23001-10, the usage method of media quality metrics is specified. In ITU-T J.143, the user needs for the objective quality assessment of video in digital cable television are specified. In ITU-T P.914, the display requirements for 3D video quality assessment are specified.In ITU-T P.1204, it is specified to perform video quality evaluation for high-reliability transmission streaming services with 4K resolution, etc.

[0065] Currently, video encoding is not limited to merely meeting the viewing needs of the user side. In the current situation where the user's needs are increasing and changing, video encoding analysis after transmission and other visual applications will also become richer. Such objective evaluation results of image quality based on such references can only be obtained by calculation on the encoder side. By carrying these information in the media bitstream, it is possible to provide the application program with the encoded image quality evaluation information. The application program can further provide more flexible and appropriate video services and exploration research work related to quality evaluation based on the quality evaluation information. For example, the complete reference quality metric can provide improved fidelity measurement accuracy for quality-oriented application programs. Since most of these application programs usually cannot obtain a complete reference, they can only perform quality evaluation based on parameters such as resolution, bitrate, quantization parameter, or quality ranking. Also, user-generated content from mobile devices or drones is usually pre-compressed and the capture quality is unknown. It is very necessary and meaningful to transmit the original quality metrics of user-generated content to the client and the service. On the other hand, in video communication, one very practical scenario is to convert an existing compressed video stream into another format / bitrate. After decoding and then re-encoding, there is a possibility of increasing compression and scaling processing in this process. In the video transcoding process, there is also a possibility of performing further pixel processing such as noise removal, contrast / brightness adjustment, etc. In the above scenarios, when performing the corresponding operations, the expected goal is not to reduce the quality, but to be able to improve the perceived video quality. Therefore, providing image quality evaluation information, that is, image quality information, that is, the original quality evaluation information and the subsequently generated quality evaluation information in the media bitstream is also meaningful for the evaluation of the entire system and service.

[0066] At the JVET meeting, related proposals were submitted in the form of image quality metric supplemental enhancement information (SEI) information. In this form, only the metric type (i.e., the objective evaluation method of the image quality adopted, such as PSNR, SSIM, etc.) and the quality metric values of the luminance component and / or chrominance component in this metric type are included in the image quality metric SEI information.

[0067] In some cases, for the objective quality evaluation of an image, only the calculation of the objective quality of the image on a per-image basis is considered. In a panoramic video, when the user is only viewing a specific area of the image screen, the quality value calculated for the entire image is clearly meaningless. It is necessary to determine the quality metric regions based on image features and calculate the quality evaluation scores for each quality metric region. With the daily development of video technology, application scenarios that reproduce video image screens corresponding to the user's viewing area according to the user's perspective, such as panoramic videos and free-viewpoint videos, are increasing. Calculating the image quality evaluation scores according to regions becomes more meaningful. Usually, the objective evaluation results of the image quality based on a reference can only be obtained by calculation on the encoder side. However, the SEI information carrying this information can provide relevant information to the application program so that the application program can further provide more flexible and appropriate video services based on the quality information.

[0068] This application realizes bitstream processing through the following embodiments.

Embodiment

[0069] The embodiment of this application provides a method for media bitstream processing, which determines an image quality evaluation region based on the image features of an image to be quality-evaluated, i.e., an image waiting for evaluation, calculates the quality evaluation score corresponding to the quality evaluation region, and writes it into the media bitstream.

[0070] This embodiment includes the following steps: step S101, step S102, step S103, step S104, and step S105.

[0071] In step S101, an image waiting for quality evaluation is acquired.

[0072] The image waiting for quality evaluation is a reconstructed image generated after the original image has passed through the encoding and decoding flows respectively. The original image means the image before encoding and compression. Unless otherwise specified, the image may be a single still image or the image data of one (frame) in a video.

[0073] In step S102, an image quality evaluation region is determined based on the image features.

[0074] The image features may be the image type, and the image type may be a normal image and an abnormal image. The abnormal image includes at least a 360-degree panoramic video image, a free-view video image, and other multi-viewpoint video images.

[0075] When the image is a normal image, the image quality evaluation region is the entire image region. When the image is an abnormal image, the image can be divided into different regions, and each region corresponds to one image quality evaluation region. Usually, the division of the image quality evaluation region corresponds to the viewpoint region that the user can select and view, but this form does not limit the division method of the image quality evaluation region.

[0076] In this embodiment, the division of the image type into normal images and abnormal images is only an example, and it may be classified according to the actual type of the image, or according to other image classification methods that can be used for the division of the image viewpoint region.

[0077] Further, the image feature may be the content of the image screen. For example, the image quality evaluation area is divided according to whether the image screen content includes screen contents such as a person or a vehicle. Alternatively, the image quality evaluation area or the like is divided according to whether there is movement or other changes in the image screen content.

[0078] In the High Efficiency Video Coding (H.265 / HEVC) standard and the Versatile Video Coding (VVC) standard, one image can be divided into tiles and slices. In VVC, dividing one image into subpictures is also supported. Therefore, one quality evaluation area in this embodiment may correspond to one image, or may be one subpicture. Alternatively, one quality evaluation area may include one or more tiles, or one quality evaluation area may include one or more slices.

[0079] In step S103, a quality evaluation score corresponding to the image quality evaluation area is calculated to determine the image quality information.

[0080] To calculate the quality evaluation score corresponding to the image quality evaluation area, first, it is necessary to determine the image quality evaluation method to be adopted. When the image quality evaluation method to be adopted is a full-reference quality evaluation method, the image quality evaluation area and the pixel values of the area corresponding to the image quality evaluation area of the original image are used, and the quality evaluation score corresponding to the image quality evaluation area needs to be calculated according to the quality evaluation mathematical model given by the full-reference image quality evaluation method. When the image quality evaluation method to be adopted is a partial-reference quality evaluation method, the image quality evaluation area and a part of the statistical information extracted from the original image required by the partial-reference image quality evaluation method are used, and the quality evaluation score corresponding to the image quality evaluation area needs to be calculated according to the quality evaluation mathematical model given by the method. For example, the digital watermark in the image can be extracted as a part of the statistical information, or the visual sensitivity coefficient in the image can be extracted as a part of the statistical information, etc. When the image quality evaluation method to be adopted is a no-reference quality evaluation method, it is not necessary to use the original image, and the quality evaluation score corresponding to the image quality evaluation area is directly calculated according to the quality evaluation mathematical model given by the no-reference quality evaluation method. When the no-reference quality evaluation method is based on a neural network, the mathematical model should at least include information such as the neural network model and network parameters, and it is also possible to further provide the dataset or training set adopted by the neural network model.

[0081] The quality evaluation score may correspond to one quality evaluation area or multiple quality evaluation areas. The multiple quality evaluation areas may be quality evaluation areas at different positions in the same image or quality evaluation areas in multiple images.

[0082] The image quality information should include the quality evaluation score corresponding to at least one image quality evaluation area.

[0083] The image quality information may include an image quality evaluation method that is adopted corresponding to a quality evaluation score corresponding to at least one image quality evaluation region.

[0084] When there are two or more image quality evaluation regions, the number of image quality evaluation regions should be included in the image quality information.

[0085] When there are two or more image quality evaluation regions, the image quality information may include image quality evaluation region division information, and the image quality evaluation region division information may include one or more of attributes such as the number of the image quality evaluation region, the position of the image quality evaluation region, and the size of the image quality evaluation region.

[0086] When the media bitstream is a scalable encoded bitstream, the image quality information may include information on the scalable encoding layer, and the information on the scalable encoding layer may include the number of scalable encoding layers and the scalable encoding layer to which the image quality evaluation region belongs.

[0087] In some embodiments, when the image quality evaluation method included in the image quality information is a full-reference or partial-reference image quality evaluation method, the image quality information may include information on the original image, and the information on the original image may be a link to obtain the original image or statistical information on a part of the original image.

[0088] When the image quality evaluation method included in the image quality information is a no-reference image quality evaluation method, the image quality information may include the mathematical model adopted by the no-reference image quality evaluation method. When the no-reference quality evaluation method is based on a neural network, the mathematical model may include information such as a neural network model, network parameters, etc. and / or dataset information adopted by the neural network model.

[0089] As described in the background art, when a video bitstream may undergo a transcoding operation or a process of decoding and then re-encoding during transmission, there may be one or more quality evaluation scores in one quality evaluation area. When there are two or more quality evaluation scores corresponding to the same quality evaluation area, the image quality information should include image quality evaluation methods adopted by the multiple quality evaluation scores corresponding to the one image quality evaluation area respectively. The image quality evaluation methods may be of different evaluation types, and the evaluation types include full reference, partial reference, and no reference. Also, the image quality information may include one or more of information such as the time stamp corresponding to the generation of each quality evaluation score, the device ID that generates each quality evaluation score, and the number that generates each quality evaluation score.

[0090] Different quality evaluation areas can adopt different image quality evaluation methods, and the selection of the image quality evaluation method can be determined based on the image features included in the quality evaluation area. For example, when the quality evaluation area is divided based on the image screen content as the image feature, it can be divided into a quality evaluation area including important figures and a background image quality evaluation area without important content, and different image quality evaluation methods can be adopted respectively.

[0091] Different image quality evaluation methods usually have their respective specific evaluation scales. For example, PSNR usually varies within the range of 20 to 60 dB (decibel), and SSIM varies within the range of 0 to 1. The quality evaluation score in this embodiment may be a score calculated based on the image quality evaluation method, or a value converted by a mathematical formula. For example, an integer rounding operation is performed on the original score, or level classification, etc. is performed on the original score.

[0092] In step S104, the image quality evaluation information, that is, the image quality information, is written into the media bitstream.

[0093] As one implementation form, corresponding image quality evaluation SEI information is generated based on the image quality evaluation information and written into the media bitstream. The image quality evaluation SEI information can be understood as SEI information including image quality information.

[0094] In addition to adopting the generation of corresponding image quality evaluation SEI information based on the image quality evaluation information, the image quality evaluation area and the corresponding quality evaluation score are used as the level information of the image quality evaluation area, and if necessary, they may be written into different parts of the media bitstream, for example, the media description part. The image quality evaluation information may be written into a corresponding parameter set as needed, and the present application does not limit the method of writing the image quality evaluation information into the media bitstream.

[0095] In step S105, the media bitstream is transmitted or stored.

[0096] The media bitstream of the image quality evaluation area and the corresponding quality evaluation score is transmitted or stored. When the image quality evaluation area and the corresponding quality evaluation score are used as the level information of the image quality evaluation area, the media bitstream may be transmitted or stored based on different attributes of the level information instead of being transmitted or stored in the scan order.

Embodiment

[0097] This embodiment gives an example of a form for realizing "writing the image quality information into the media bitstream" in step S104 in Embodiment 1 by using the image quality evaluation SEI information.

[0098] As one implementation form, the image quality information can be written into the supplementary enhancement information in the video bitstream. Table 1 shows a schematic table for writing the image quality information according to the embodiment of the present application into the supplementary enhancement information, and a specific example has the structure shown in the following table.

[0099]

Table 1

[0100] The SEI information includes a payload type "payloadType" and a payload size "payloadSize". Here, "payloadType" is used to indicate the type of the SEI information. For example, when "payloadType" is "PIC_QUALITY_MEASURE_INFO", it indicates that the SEI information is image quality evaluation SEI information. "payloadSize" includes the actual image quality information carried. Table 2 shows a schematic table of the payload size carrying the image quality information according to the embodiments of the present application. It is as shown in the following table.

[0101]

Table 2

[0102] "pqm_picture_type" indicates the image type related to the SEI information. As described in steps S102 and S103 in Embodiment 1, when "pqm_picture_type" = 0, it indicates that the image related to the SEI information is a normal image, representing that the image quality evaluation area is the entire image area. When "pqm_picture_type" = 1, it indicates that the image related to the SEI information is an abnormal image, and the image quality evaluation area may be different quality evaluation areas divided from the entire image or different quality evaluation areas divided from a sub-image.

[0103] Here, only an example with "pqm_picture_type" as an image feature is shown. According to Embodiment 1, the image feature may be based on the content in the image screen. It is determined whether to divide the quality evaluation area for the image based on the content in the screen, and the image quality evaluation area division instruction information is determined. Table 3 is a schematic table of the image quality evaluation area division instruction information according to the embodiments of the present application. The following table shows one possible example.

[0104]

Table 3

[0105] Also, when it is necessary to divide one image into a plurality of image quality evaluation regions and perform quality evaluation on each of them, the total of the plurality of image quality evaluation regions may be a complete one image, or may be only a part of the regions in one image.

[0106] pqm_mixed_measurement_flag indicates whether to support using different quality evaluation methods for different image quality evaluation regions. When pqm_mixed_measurement_flag = 0, it means that the same quality evaluation method is used for all image quality evaluation regions. When pqm_mixed_measurement_flag = 1, it also means that the same quality evaluation method is used for all image quality evaluation regions.

[0107] As described in step S102 in Embodiment 1, when the image quality evaluation region is divided based on the image screen content as an image feature and is divided into an image quality evaluation region including important persons and a background image quality evaluation region not including important content, different image quality evaluation methods can be adopted respectively. Although not specifically stated in subsequent embodiments, by default, pqm_mixed_measurement_flag = 1 is supported, that is, the same quality evaluation method is used for all image quality evaluation regions.

[0108] pqm_measurement_method indicates the image quality evaluation method to be adopted. Table 4 is a schematic table for performing image quality evaluation method indication according to the embodiments of the present application. For the specific indication method, refer to the following table.

[0109]

Table 4

[0110] This embodiment only gives an example of the value of the image quality evaluation method, and any quality evaluation method for evaluating the image quality is allowed. It includes simultaneously adopting two or more image quality evaluation methods, or identifying different quality evaluation methods using an external quality evaluation method registration mechanism, etc.

[0111] The pqm_single_component_flag indicates whether to perform quality evaluation only on a single component in the image quality evaluation area. When pqm_single_component_flag = 1, it means that quality evaluation is performed only on a single component within the color space of the image quality evaluation area. When pqm_single_component_flag = 0, it means that quality evaluation needs to be performed on all components within the color space of the image quality evaluation area respectively. It is supported to adopt different policies for multiple quality evaluation areas, that is, some quality evaluation areas perform quality evaluation only on a single component, and some quality evaluation areas perform quality evaluation on all components.

[0112] There are multiple types of color space displays. For example, the commonly seen RGB color space describes colors with the three primary colors. Component R represents red (Red), component G represents green (Green), and component B represents blue (Blue). The YUV color space describes colors by luminance and chrominance. The luminance component is Y, and the color difference components are composed of two independent signals, usually represented as U, V or Cb, Cr, etc. In the following embodiments, YUV is described as an example, and unless otherwise specified, quality evaluation is performed only on the luminance component Y by default.

[0113] pqm_PSNR[i] indicates calculating the quality evaluation score of the image luminance component Y using the quality evaluation method PSNR, or calculating the quality evaluation scores of the image color difference components U and V using the quality evaluation method PSNR.

[0114] pqm_SSIM[i] indicates obtaining the quality evaluation score of the image luminance component Y using the quality evaluation method SSIM, or calculating the quality evaluation scores of the image color difference components U and V using the quality evaluation method SSIM.

[0115] pqm_VQM[i] instructs to calculate the quality evaluation score of the image luminance component Y using the quality evaluation method VQM, or to calculate the quality evaluation scores of the image color difference components U and V using the quality evaluation method VQM.

[0116] pqm_MSSSIM[i] instructs to calculate the quality evaluation score of the image luminance component Y using the quality evaluation method MS-SSIM, or to calculate the quality evaluation scores of the image color difference components U and V using the quality evaluation method MS-SSIM.

[0117] num_quality_regions indicates the number of image quality evaluation regions.

[0118] pqm_quality_region_info_present_flag indicates whether the quality evaluation region division information exists. When pqm_quality_region_info_present_flag = 1, it means that the quality evaluation region division information exists. When pqm_quality_region_info_present_flag = 0, it means that the quality evaluation region division information does not exist. In this case, the region division information in the panorama image information may be adopted by default, or the information on the quality evaluation region division may be transmitted independently in the media bit stream.

[0119] pqm_quality_region_info[i]() indicates the information of the i-th quality evaluation region, that is, the segmentation information of the quality evaluation region. The information of the quality evaluation region may further include one or more of attributes such as the number of the image quality evaluation region, the position of the image quality evaluation region, and the size of the image quality evaluation region. Here, the position of the image quality evaluation region may include the horizontal position of the image quality evaluation region in the entire image and the vertical position of the image quality evaluation region in the entire image. Also, the position of the image quality evaluation region may include the horizontal coordinate of the upper left pixel point of the image quality evaluation region in the entire image and the vertical coordinate of the upper left pixel point of the image quality evaluation region in the entire image. The size of the image quality evaluation region may include the width and height of the image quality evaluation region.

[0120] pqm_PSNR[i][j] indicates calculating the quality evaluation score of the luminance component Y in the i-th image quality evaluation region using the quality evaluation method PSNR, or calculating the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region using the quality evaluation method PSNR.

[0121] pqm_SSIM[i][j] indicates obtaining the quality evaluation score of the luminance component Y in the i-th image quality evaluation region using the quality evaluation method SSIM, or calculating the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region using the quality evaluation method SSIM.

[0122] pqm_VQM[i][j] indicates calculating the quality evaluation score of the luminance component Y in the i-th image quality evaluation region using the quality evaluation method VQM, or calculating the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region using the quality evaluation method VQM.

[0123] pqm_MSSSIM[i][j] instructs to calculate the quality evaluation score of the luminance component Y in the i-th image quality evaluation region using the quality evaluation method MS-SSIM, or to calculate the quality evaluation scores of the chromatic difference components U and V in the i-th image quality evaluation region using the quality evaluation method MS-SSIM.

[0124] In this embodiment, the quality evaluation region corresponding to the quality evaluation score is one single image or a certain quality evaluation region included in the single image. In this case, the durability range of the image quality evaluation SEI information carrying the image quality evaluation information is the picture unit (PU) including the image quality evaluation SEI information, indicating that the image quality evaluation SEI information acts only on one (frame) of the images related thereto.

Embodiment

[0125] The high-performance video coding (H.265 / High Efficiency Video Coding, HEVC) standard and the versatile video coding (H.266 / Versatile Video Coding, VVC) standard formulated by the Joint Video Expert Teams (JVET) under ISO / IEC and ITU both support the concept of "multi-layer video coding / scalable video coding". FIG. 3 is a schematic diagram of the media bitstream according to the embodiment of the present application. As shown in FIG. 3, the media bitstream may include one base layer and at least one enhancement layer, and can provide video image screens of different qualities. In this case, taking Image 1 as an example, one quality evaluation region in Image 1 may have a plurality of quality evaluation scores, corresponding to the base layer, the first enhancement layer, and the second enhancement layer respectively.

[0126] This embodiment provides an example of a method for realizing "writing the image quality information into the media bitstream" in step S104 in Embodiment 1 using image quality evaluation SEI information under scalable encoding conditions. Table 5 is a schematic table for writing the image quality information according to the embodiments of the present application into the media bitstream.

[0127] pqm_picture_type indicates the image type related to the SEI information. Refer to the description and explanation of pqm_picture_type in Embodiment 2.

[0128] pqm_measurement_method indicates the image quality evaluation method to be adopted. Refer to the description and explanation of pqm_measurement_method in Embodiment 2.

[0129] pqm_max_layers_minus1 indicates the number of encoding layers, and the value of pqm_max_layers_minus1 should be equal to the value of vps_max_layers_minus1 in the Video Parameter Set (VPS).

[0130] pqm_PSNR[i] indicates calculating the image quality evaluation score of the i-th layer using the PSNR quality evaluation method.

[0131] pqm_SSIM[i] indicates calculating the image quality evaluation score of the i-th layer using the SSIM quality evaluation method.

[0132] num_quality_regions indicates the number of image quality evaluation regions.

[0133] pqm_quality_region_info[j]() indicates the information of the j-th quality evaluation region. Refer to the description and explanation of pqm_quality_region_info[i]() in Embodiment 2.

[0134] pqm_PSNR[i][j] instructs to calculate the quality evaluation score of the j-th image quality evaluation region in the i-th layer using the PSNR quality evaluation method.

[0135] pqm_SSIM[i][j] instructs to obtain the quality evaluation score of the j-th image quality evaluation region in the i-th layer using the SSIM quality evaluation method.

[0136] In this embodiment, for the sake of simplicity of form and to emphasize the characteristics of multi-layer coding, pqm_single_component_flag is omitted in the given SEI information example, and it is assumed that pqm_single_component_flag = 1, that is, quality evaluation is performed only on the luminance component of the image. However, this form can support the case where pqm_single_component_flag = 0, that is, quality evaluation is performed on the luminance component and the chrominance components of the image respectively.

[0137]

Table 5

[0138] This form can also support a form that performs quality evaluation on different luminance and chrominance components for different coding layers. For example, quality evaluation is performed on the luminance component and the chrominance components of the base layer image respectively, and quality evaluation is performed only on the luminance component of the enhancement layer image. Or, quality evaluation is performed on the luminance component and the chrominance components of the highest enhancement layer image respectively, and quality evaluation such as only on the luminance component of the base layer image is performed.

[0139] In this embodiment, the quality evaluation area corresponding to the quality evaluation score is one single base layer image or enhancement layer image, and it may also be a certain quality evaluation area included in the single base layer image or enhancement layer image. In this case, the durability range of the image quality evaluation SEI information carrying the image quality evaluation information is the access unit (AU) including the image quality evaluation SEI information, indicating that it acts only on one (frame) of the image related to the image quality evaluation SEI information.

Example

[0140] This embodiment provides an example of a method for realizing "writing the image quality information into the SEI information in the media bit stream" in step S104 in Embodiment 1 for a plurality of frames of images. Table 6 is a schematic table for writing the image quality information according to the embodiment of the present application into the bit stream.

[0141]

Table 6

[0142] pqm_picture_type indicates the image type related to the SEI information. Refer to the description and explanation of pqm_picture_type in Embodiment 2.

[0143] pqm_measurement_method indicates the image quality evaluation method to be adopted. Refer to the description and explanation of pqm_measurement_method in Embodiment 2.

[0144] The pqm_sequence_info indicates the information of the image sequence related to the image quality evaluation SEI information. The information of the image sequence includes at least the identification numbers of the images of each frame in the image sequence, and the image information in the image sequence can be determined based on the identification numbers of the images. The identification number of the image may be the decoding order or the playback order of the image, or any other arbitrary method that can determine the image information.

[0145] The pqm_quality_of_sequence_flag indicates whether to calculate the quality evaluation score for the image sequence. When the value of pqm_quality_of_sequence_flag is 1, it means that only one overall quality evaluation score is calculated for all the images in the image sequence. When the value of pqm_quality_of_sequence_flag is 0, it means that the respective quality evaluation scores are calculated for each image in the image sequence.

[0146] In particular, this embodiment only gives an example of calculating the quality evaluation score of the image sequence. For example, the quality evaluation type of the image sequence, that is, the pqm_quality_of_sequence_type for indicating how to calculate the quality evaluation score of the image sequence, may be designed. Table 7 is a schematic table of the quality evaluation type of the image sequence according to the embodiment of the present application, and the value of pqm_quality_of_sequence_type may be as shown in the following table.

[0147]

Table 7

[0148] The pqm_sequence_PSNR indicates that the quality evaluation score of the image sequence is calculated using the quality evaluation method PSNR.

[0149] The pqm_num_of_picture indicates the number of images included in the image sequence.

[0150] pqm_PSNR[i] instructs to calculate the quality evaluation score of the i-th image in the image sequence using the quality evaluation method PSNR.

[0151] Referring to Example 2, it indicates whether to support performing different quality evaluation methods for different image quality evaluation regions using pqm_mixed_measurement_flag. In this example, in the actual implementation process, different more appropriate quality evaluation methods may be respectively adopted for a single image evaluation region and a series of image evaluation regions. For example, quality evaluation methods such as PSNR and SSIM can be adopted for a single image evaluation region, while quality evaluation methods such as VQM and wPSNR can be adopted for a series of image evaluation regions. Similarly, different quality evaluation methods may be adopted for different series of image evaluation regions.

[0152] num_quality_regions indicates the number of quality evaluation regions in each image in the image sequence.

[0153] pqm_quality_region_info[i]() indicates the information of the i-th series of quality evaluation regions. Refer to the description and explanation of pqm_quality_region_info[i]() in Example 2. Here, the i-th series of quality evaluation regions may be considered as a series formed by the i-th quality evaluation regions in each image. For example, in the case of multiple images to be evaluated, each image to be evaluated is divided into multiple regions, and the i-th quality evaluation regions in each image to be evaluated form a series of quality evaluation regions.

[0154] pqm_sequence_PSNR[i] instructs to calculate the quality evaluation score of the i-th series of quality evaluation regions using the quality evaluation method PSNR.

[0155] num_quality_regions[i] indicates the number of quality evaluation regions in the i-th image in the image sequence.

[0156] pqm_quality_region_info[i][j]() indicates the information of the j-th quality evaluation region series in the i-th image within the image sequence. Refer to the description of pqm_quality_region_info[i]() in Example 2.

[0157] pqm_PSNR[i][j] indicates that the quality evaluation score of the j-th image quality evaluation region in the i-th image is calculated using the quality evaluation method PSNR.

[0158] Similar to Example 3, in this example, the pqm_single_component_flag is omitted in the given example of SEI information, and it is assumed that pqm_single_component_flag = 1, that is, quality evaluation is performed only on the luminance component of the image or image sequence. However, this form can support the case where pqm_single_component_flag = 0, that is, quality evaluation is performed on the luminance component and color difference components of the image or image sequence respectively.

[0159] This form can also support adopting different quality evaluation forms for luminance and color difference components for image sequences and single images. For example, quality evaluation is performed on only the luminance component and color difference components of a single image respectively, and only quality evaluation on the luminance component is performed for the image sequence. Or, quality evaluation is performed on only the luminance component and color difference components of the image sequence respectively, and only quality evaluation such as on the luminance component is performed for a single image.

[0160] In this example, the quality evaluation region corresponding to the quality evaluation score is a certain quality evaluation region series included in one image sequence or image. In this case, the durability range of the image quality evaluation SEI information carrying the image quality evaluation information is the coded video sequence (CVS) or coded layer video sequence (CLVS) including the image quality evaluation SEI information.

Example

[0161] This embodiment provides an example of another SEI information method for realizing "writing the image quality information into the media bitstream" in step S104 of Embodiment 1. Table 8 is a schematic diagram showing writing the image quality information according to the embodiment of the present application into the media bitstream.

[0162]

Table 8

[0163] When the value of pqm_cancel_flag is equal to 1, it indicates that the image quality evaluation SEI information cancels the durability of any previous image quality evaluation SEI message and does not use the related SEI function. Conversely, when the value is equal to 0, it indicates that the immediately following is image quality evaluation information.

[0164] pqm_persistence_flag is used to indicate the durability of the related image quality evaluation SEI information. When pqm_persistence_flag = 0, it indicates that the image quality evaluation SEI information is applied only to the currently decoded image. When pqm_persistence_flag = 1, the image quality evaluation SEI information is applied to the currently decoded image and continues to be used for all subsequent output images until one or more of the following conditions are met: (1) the condition that the bitstream ends, (2) the condition that a new image sequence starts, and (3) the condition that the subsequently output image is associated with the image quality evaluation SEI information.

[0165] In the present technical solution, when pqm_persistence_flag = 0, it is used to indicate that the quality evaluation score of the related image quality evaluation SEI information is for the current decoded image or the image quality evaluation region in the current decoded image. When pqm_persistence_flag = 1, it may be understood that it is used to indicate that the quality evaluation score of the related image quality evaluation SEI information is for the image sequence composed of the current decoded image and subsequent output images, or for the image quality evaluation series in the image composed of the current decoded image and subsequent output images.

[0166] pqm_picture_type indicates the image type related to the SEI information. Refer to the description of pqm_picture_type in Embodiment 2.

[0167] pqm_measurement_method indicates the image quality evaluation method to be adopted. Refer to the description of pqm_measurement_method in Embodiment 2.

[0168] pqm_PSNR indicates the quality evaluation score of the current image calculated using the quality evaluation method PSNR or the image sequence composed of the current image and subsequent output images.

[0169] num_quality_regions indicates the number of quality evaluation regions in one image within the image sequence.

[0170] pqm_quality_region_info[i]() indicates the information of the i-th quality evaluation region. Refer to the description of pqm_quality_region_info[i]() in Embodiment 2.

[0171] pqm_PSNR[i] indicates calculating the quality evaluation score of the i-th image quality evaluation region in the current image or the i-th image quality evaluation region series in the current image and subsequent output images using the quality evaluation method PSNR.

[0172] In this embodiment, by using pqm_cancel_flag and pqm_persistence_flag, it is possible to calculate an image quality evaluation score for a single image quality evaluation region, a picture group (Group of Picture, GoP), one scene, or a series of image quality evaluation regions, etc. Here, a single image quality evaluation region includes a single image or one image quality evaluation region included in a single image. The series of image quality evaluation regions includes the entire image sequence or a series of image quality evaluation regions composed of image quality evaluation regions included in the same or different images.

Embodiment

[0173] This embodiment provides a further example of a method for realizing "writing the image quality information into the media bitstream" in step S104 in Embodiment 1.

[0174] JPEG0007701548000011.jpg108170

[0175] picture_type indicates the image type. When the value is 0, it indicates that the image is one image quality evaluation region. When the value is 1, it indicates that the image is divided into at least two image quality evaluation regions.

[0176] measurement_method indicates the image quality evaluation method to be adopted. Refer to the description of pqm_measurement_method in Embodiment 2.

[0177] quality_result indicates calculating the image quality evaluation result using the quality evaluation method indicated by measurement_method.

[0178] num_quality_regions indicates the number of image quality evaluation regions.

[0179] quality_region_info[i]() indicates the information of the i-th quality evaluation region. Refer to the description of pqm_quality_region_info[i]() in Embodiment 2.

[0180] measurement_method[i] indicates the image quality evaluation method adopted for the i-th quality evaluation region.

[0181] quality_result[i] indicates calculating the quality evaluation result of the i-th quality evaluation region using the quality evaluation method indicated by measurement_method[i].

[0182] JPEG0007701548000012.jpg48170

[0183] segment_top_left_x indicates the horizontal coordinate in the entire image of the top-left pixel point of the quality evaluation region.

[0184] segment_top_left_y indicates the vertical coordinate in the entire image of the top-left pixel point of the quality evaluation region.

[0185] segment_width[i] indicates the pixel width of the quality evaluation region.

[0186] segment_height[i] indicates the pixel height of the quality evaluation region.

[0187] Note that this embodiment only shows one simple example. Similar to it, the forms described in Embodiments 1 to 5 can all store image quality evaluation data in a file using the ISO base media file format, and the description here is omitted.

Embodiment

[0188] In step S701, obtain the media bit stream.

[0189] This technical solution does not limit the type and format of the acquired media bitstream, etc.

[0190] In step S702, analyze the media bitstream to obtain image quality evaluation information, that is, image quality information.

[0191] Analyze the media bitstream to obtain the image quality information carried in the media bitstream. The image quality information may be included in the SEI information in the media bitstream.

[0192] In addition, as described in step S104 of Embodiment 1, in addition to generating corresponding image quality evaluation SEI information based on the image quality evaluation information, the image quality evaluation area and the corresponding quality evaluation score are used as the level information of the image quality evaluation area, and if necessary, different parts of the media bitstream, for example, the media description part, may be written. Therefore, this embodiment does not limit the method of obtaining image quality evaluation information from the media bitstream either.

[0193] In step S703, based on the image quality evaluation information, determine the image quality evaluation area and the corresponding quality evaluation score.

[0194] Before determining the image quality evaluation area and the corresponding quality evaluation score based on the image quality evaluation information, it may further include determining the image quality evaluation area division instruction information corresponding to the image quality evaluation information and the number of the image quality evaluation areas based on the image quality evaluation information. Determining the image quality evaluation area and the corresponding quality evaluation score may further include determining the quality evaluation method corresponding to the quality evaluation score.

[0195] In one embodiment, the embodiments of the present application provide a bitstream processing device. FIG. 4 is a structural schematic diagram of the bitstream processing device according to the embodiments of the present application. The device can be integrated into a terminal device. As shown in FIG. 4, the device includes the following first determination module 41, second determination module 42, third determination module 43, and addition module 44.

[0196] The first determination module 41 is configured to determine an image quality evaluation region.

[0197] The second determination module 42 is configured to determine quality evaluation data of the quality evaluation region.

[0198] The third determination module 43 is configured to determine image quality information including the quality evaluation data.

[0199] The addition module 44 is configured to add the image quality information to a bitstream to obtain a processed bitstream.

[0200] The bitstream processing device according to this embodiment is used to implement the bitstream processing method of the embodiment shown in FIG. 1. The implementation principle and technical effects of the bitstream processing device according to this embodiment are similar to those of the bitstream processing method of the embodiment shown in FIG. 1, and the description is omitted here.

[0201] Based on the above embodiments, modified examples of the above embodiments are presented. For the sake of simplicity of description, only the differences between the modified examples and the above embodiments are described.

[0202] In one embodiment, the number of the image quality evaluation regions is at least one. The image quality evaluation region includes the entire image region to be evaluated or a plurality of image regions to be evaluated. Alternatively, the image quality evaluation region includes a part of the regions formed after the image to be evaluated is divided into regions. The image to be evaluated determines a division policy based on image features.

[0203] In one embodiment, the image quality information further includes the image quality evaluation method adopted to determine the quality evaluation data, the time stamp for generating the quality evaluation data, the device identifier for generating the quality evaluation data, the identifier of the quality evaluation data, the number of image quality evaluation regions, information of the original image, including the link of the original image or the statistical information of a part of the original image, information of the original image where the image to be evaluated is a reconstructed image generated after processing the original image, the mathematical model adopted to perform no-reference image quality evaluation, the image quality evaluation region division type indication information, the indication information of the commonality of the quality evaluation methods adopted for different image quality evaluation regions, the indication information of the components to be evaluated in the image quality evaluation region, information of the scalable coding layer, including the number of scalable coding layers, the scalable coding layer to which the image quality evaluation region belongs, the quality evaluation method adopted for the i-th layer image in the image to be evaluated, the quality evaluation method adopted for the n-th image quality evaluation region of the k-th layer image in the image to be evaluated, where i, k, and n are positive integers, information of the image sequence including a plurality of images to be evaluated corresponding to the image quality evaluation region, the number of images included in the image sequence, the number of quality evaluation regions in each image in the image sequence, the quality evaluation method adopted to determine the j-th (j is a positive integer) image quality evaluation region in the image sequence, information of the m-th (m is a positive integer) quality evaluation region, the quality evaluation method adopted to determine the quality evaluation data of the h-th (h is a positive integer) quality evaluation region, and one or more of the number of quality evaluation regions in the g-th (g is a positive integer) image to be evaluated in the image sequence.

[0204] In one embodiment, the image quality information further includes the division information of the image quality evaluation region and the indication information of whether the division information exists. The division information of the image quality evaluation region includes one or more of the identifier of the quality evaluation region, the position of the quality evaluation region, and the size of the quality evaluation region.

[0205] In one embodiment, the first determination module 41 is specifically configured to determine the image quality evaluation region of the image to be evaluated based on the image features of the image to be evaluated.

[0206] In one embodiment, adding the image quality information to the bitstream by the addition module 44 to obtain a processed bitstream includes generating supplementary enhancement information based on the image quality information, and adding the supplementary enhancement information to the bitstream to obtain a processed bitstream.

[0207] In one embodiment, adding the image quality information to the bitstream by the addition module 44 to obtain a processed bitstream includes writing the image quality information to a media file, or writing the image quality information to the media file as the level information of the corresponding image quality evaluation region, and adding the media file to the bitstream to obtain a processed bitstream.

[0208] In one embodiment, by the addition module 44, one image quality evaluation region corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting different quality evaluation methods for one image quality evaluation region.

[0209] In one embodiment, the quality evaluation methods adopted for different image quality evaluation regions by the addition module 44 may be the same or different.

[0210] In one embodiment, the embodiments of the present application further provide a bitstream processing device, which is integrated into a terminal device for the received and processed bitstream. FIG. 5 is a structural schematic diagram of the bitstream processing device according to the embodiments of the present application. As shown in FIG. 5, the device includes the following acquisition module 51, analysis module 52, and determination module 53.

[0211] The acquisition module 51 is configured to acquire the processed bitstream.

[0212] The analysis module 52 is configured to analyze the processed bitstream to obtain image quality information.

[0213] The determination module 53 is configured to determine an image quality evaluation area and corresponding quality evaluation data based on the image quality information.

[0214] The bitstream processing device according to this embodiment is used to implement the bitstream processing method of the embodiment shown in FIG. 2. The implementation principle and technical effects of the bitstream processing device according to this embodiment are similar to those of the bitstream processing method of the embodiment shown in FIG. 2, and the description is omitted here.

[0215] In one embodiment, the embodiments of the present application further provide a terminal device. FIG. 6 is a structural schematic diagram of the terminal device according to the embodiments of the present application. As shown in FIG. 6, the terminal device according to the embodiments of the present application includes one or more processors 61 and a storage device 62. The number of processors 61 in the terminal device may be one or more. In FIG. 6, one processor 61 is taken as an example. The storage device 62 is used to store one or more programs. When the one or more programs are executed by the one or more processors 61, the one or more processors 61 implement the bitstream processing method described in the embodiments of the present application.

[0216] The terminal device further includes a communication device 63, an input device 64, and an output device 65.

[0217] The processor 61, memory device 62, communication device 63, input device 64, and output device 65 in the terminal device can be connected by a bus or other means. In FIG. 6, it is exemplified that they are connected via a bus.

[0218] The input device 64 can receive the input numerical or character information and generate key signal inputs related to user settings and function control of the terminal device. The output device 65 may include a display device such as a display.

[0219] The communication device 63 may include a receiver and a transmitter. The communication device 63 is configured to perform information transmission and reception communication according to the control of the processor 61. The information includes, but is not limited to, the processed bit stream.

[0220] The memory device 62 can be used as a computer-readable storage medium to store software programs, computer-executable programs, and modules, for example, program instructions / modules corresponding to the bitstream processing method according to the embodiments of the present application (for example, the first determination module 41, the second determination module 42, the third determination module 43, and the addition module 44 in the bitstream processing device, or the acquisition module 51, the analysis module 52, and the determination module 53 in the bitstream processing device). The memory device 62 may include a program storage area and a data storage area. Here, the program storage area can store an operating system and application programs required for at least one function, and the data storage area can store data created based on the use of the terminal device. Further, the memory device 62 may include a high-speed random access memory and may further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory, or other non-volatile solid-state storage devices. In some embodiments, the memory device 62 can include a memory provided remotely with respect to the processor 61, and these remote memories can be connected to the device via a network. Examples of the above network may include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0221] The embodiments of the present application further provide a storage medium, on which a computer program is stored. When the computer program is executed by a processor, any method in the embodiments of the present application is realized. When a computer program is stored on the storage medium and the computer program is executed by a processor, any bitstream processing method in the embodiments of the present application is realized.

[0222] The bitstream processing method includes determining an image quality evaluation area, determining quality evaluation data for the quality evaluation area, determining image quality information including the quality evaluation data, and adding the image quality information to a bitstream to obtain a processed bitstream.

[0223] The bitstream processing method includes obtaining a processed bitstream, analyzing the processed bitstream to obtain image quality information, and determining an image quality evaluation area and corresponding quality evaluation data based on the image quality information.

[0224] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. Further specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may be any tangible medium that contains or stores a program that can be used in or in conjunction with an instruction execution system, apparatus, or device.

[0225] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code therein. Such propagated data signals can take various forms and may include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium can transmit, propagate, or convey a program used in or in conjunction with an instruction execution system, apparatus, or device.

[0226] The program code included in the computer-readable medium can be transmitted via any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), or any suitable combination of the foregoing.

[0227] Computer program code for performing the operations of the present application can be described in one or more programming languages or combinations thereof, and the programming languages include object-oriented programming languages such as Java (registered trademark), Smalltalk, and C++, and further include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially executed on the user's computer and partially on a remote computer, or entirely executed on a remote computer or a business server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, connected via the Internet using an Internet service provider).

[0228] The above are only multiple embodiments of the present application and do not limit the protection scope of the present application.

[0229] Those skilled in the art should understand that the term "terminal device" includes any suitable type of wireless user device, for example, including mobile phones, portable data processing devices, portable network browsers or in-vehicle mobile stations.

[0230] Generally, various embodiments of the present application can be realized by hardware or application-specific circuits, software, logic or any combination thereof. For example, some aspects can be realized by hardware, and other aspects can be realized by firmware or software executable by a controller, a microprocessor or other computing devices, and the present application is not limited thereto.

[0231] Embodiments of the present application can be implemented by executing computer program instructions by a data processor of a mobile device. For example, in an entity of a processor, it can be implemented by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or target code described in any combination of one or more programming languages.

[0232] Any block diagram of a logic flow in the drawings of the present application may represent a program step, or may represent logical circuits, modules and functions connected to each other, or may represent a combination of a program step and logical circuits, modules and functions. The computer program may be stored in a memory. The memory can have any type suitable for a local technical environment and can be implemented with any suitable data storage technology, for example, read-only memory (ROM), random access memory (RAM), optical storage devices and systems (such as digital versatile disc (DVD) or compact disc (CD)), etc., but is not limited thereto. The computer-readable medium may include a non-transitory storage medium. The data processor may be of any type suitable for a local technical environment, for example, a general-purpose computer, a dedicated computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a processor based on a multi-core processor architecture, but is not limited thereto.

[0233] Through a plurality of non-limiting examples, the above has provided a detailed description of a plurality of embodiments of the present application. However, considering in conjunction with the drawings and the claims, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art without departing from the scope of the present application. Therefore, the appropriate scope of the present application is determined based on the claims.

Claims

1. Determining an image quality evaluation area; Determining quality evaluation data for the quality evaluation area; Determining image quality information including the quality evaluation data; Adding the image quality information to a bitstream to obtain a processed bitstream, including: The quality evaluation data is data obtained after performing quality evaluation on the quality evaluation area using at least one quality evaluation method; One image quality evaluation area corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting different quality evaluation methods for one image quality evaluation area; A bitstream processing method.

2. The number of the image quality evaluation areas is at least one; The image quality evaluation area includes the entire image area to be evaluated or multiple image areas to be evaluated, or the image quality evaluation area includes some areas formed after dividing the image to be evaluated into areas; The image to be evaluated determines a segmentation policy based on image features. The method according to Claim 1.

3. The image quality information includes: The image quality evaluation method adopted to determine the quality evaluation data; The timestamp for generating the quality evaluation data; The device identifier for generating the quality evaluation data; The identifier of the quality evaluation data; Information of the original image, including: The link of the original image or statistical information of a part of the original image; Information of the original image when the image to be evaluated is a reconstructed image generated after processing the original image; The mathematical model adopted for non-reference image quality evaluation; One or more of the above; The method according to Claim 1.

4. The image quality information includes: The number of the image quality evaluation areas; Image quality evaluation area segmentation instruction information; Instruction information on the commonality of the quality evaluation methods adopted for different image quality evaluation areas; Instruction information on the components to be evaluated in the image quality evaluation area; Information of the scalable coding layer, including: including the number of the scalable encoding layers, the scalable encoding layer to which the image quality evaluation region belongs, the quality evaluation method adopted for the image of the i-th layer in the image to be evaluated, and the quality evaluation method adopted for the n-th image quality evaluation region of the image of the k-th layer in the image to be evaluated, further including one or more of the information of the scalable encoding layers where i, k, and n are positive integers, The method according to claim 1.

5. The image quality information includes the information of an image sequence including a plurality of images to be evaluated corresponding to the image quality evaluation region, the number of images included in the image sequence, the number of quality evaluation regions in each image within the image sequence, the quality evaluation method adopted to determine the j-th (j is a positive integer) image quality evaluation region within the image sequence, the information of the m-th (m is a positive integer) quality evaluation region, the quality evaluation method adopted to determine the quality evaluation data of the h-th (h is a positive integer) quality evaluation region, further including one or more of the number of quality evaluation regions in the g-th (g is a positive integer) image to be evaluated within the image sequence, The method according to claim 1.

6. The image quality information further includes the division information of the image quality evaluation region and the indication information indicating whether the division information exists, The division information of the image quality evaluation region includes one or more of the identifier of the quality evaluation region, the position of the quality evaluation region, and the size of the quality evaluation region, The method according to claim 1.

7. Determining the image quality evaluation region includes determining the image quality evaluation region of the image to be evaluated based on the image features of the image to be evaluated, The method according to claim 1.

8. Obtaining the processed bitstream by adding the image quality information to the bitstream includes generating supplementary enhancement information based on the image quality information, and obtaining the processed bitstream by adding the supplementary enhancement information to the bitstream, The method according to claim 1.

9. Obtaining the processed bitstream by adding the image quality information to the bitstream includes writing the image quality information to a media file, or writing the image quality information to the media file as the level information of the corresponding image quality evaluation region, and obtaining the processed bitstream by adding the media file to the bitstream, The method according to claim 1.

10. The quality evaluation methods adopted for different image quality evaluation regions are the same or different. The method according to claim 1.

11. Obtaining a processed bitstream; Analyzing the processed bitstream to obtain image quality information; Determining an image quality evaluation region and corresponding quality evaluation data based on the image quality information, The quality evaluation data is data obtained after performing quality evaluation on the quality evaluation region by at least one quality evaluation method. One image quality evaluation region corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting different quality evaluation methods for one image quality evaluation region. Bitstream processing method.

12. A first determination module configured to determine an image quality evaluation region; A second determination module configured to determine quality evaluation data for the quality evaluation region; A third determination module configured to determine image quality information including the quality evaluation data; An addition module configured to add the image quality information to a bitstream to obtain a processed bitstream. The quality evaluation data is data obtained after performing quality evaluation on the quality evaluation region by at least one quality evaluation method. One image quality evaluation region corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting different quality evaluation methods for one image quality evaluation region. Bitstream processing apparatus.

13. An acquisition module configured to acquire a processed bitstream; An analysis module configured to analyze the processed bitstream to obtain image quality information; A determination module configured to determine an image quality evaluation region and corresponding quality evaluation data based on the image quality information; The quality evaluation data is data obtained after performing quality evaluation on the quality evaluation region by at least one quality evaluation method; One image quality evaluation region corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting different quality evaluation methods for one image quality evaluation region; Bitstream processing device.

14. A terminal device comprising one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of Claims 1 to 11; Terminal device.

15. A computer program is stored, and when the computer program is executed by a processor, the method according to any one of Claims 1 to 11 is implemented; Storage medium. ​ ​

Images