Bitstream processing method and apparatus, terminal device, and storage medium

Abstract

To provide a bitstream processing method and apparatus, a terminal device, and a storage medium which improve flexibility of image quality evaluation.SOLUTION: The bitstream processing method includes: determining (S110) an image quality evaluation area; determining (S120) quality evaluation datum of the quality evaluation area; determining (S130) image quality information including the quality evaluation datum; and adding (S140) the image quality information to a bitstream to obtain a processed bitstream. The quality evaluation datum is data obtained after the quality evaluation area is subjected to quality evaluation by at least one quality evaluation method.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application is based on and claims the benefit of priority to a Chinese patent application bearing application number 202110832744.X and filed on July 22, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of communications technology, for example, to a bitstream processing method, device, terminal device and storage medium. [Background technology]

[0003] With the development of immersive media technology, more and more video applications can provide 360-degree panoramic videos for users to watch. In order to provide users with more flexible and suitable video image services, image quality assessment can be performed on the images.

[0004] In some cases, when performing quality assessment on an image, the image quality is calculated on a per-image basis, which requires a large amount of calculation. Therefore, how to improve the flexibility of image quality assessment is currently a technical problem that needs to be solved as soon as possible. Summary of the Invention [Problem to be solved by the invention]

[0005] The present application provides a bitstream processing method, apparatus, terminal device and storage medium. [Means for solving the problem]

[0006] In aspect 1, the present embodiment comprises: determining a picture quality assessment area; determining quality assessment data of the quality assessment area; determining picture quality information including the quality assessment data; and adding the picture quality information to a bitstream to obtain a processed bitstream. A bitstream processing method is provided.

[0007] In aspect 2, the present embodiment comprises: obtaining a processed bitstream; analyzing the processed bitstream to obtain image quality information; and determining an image quality assessment area and corresponding quality assessment data based on the image quality information. A bitstream processing method is provided.

[0008] In aspect 3, the present embodiment comprises: a first determining module configured to determine an image quality assessment area; a second determining module configured to determine quality assessment data of the quality assessment area; a third determining module configured to determine image quality information including the quality assessment data; and an appending module configured to append the image quality information to a bitstream to obtain a processed bitstream. A bitstream processing device is provided.

[0009] In aspect 4, the present embodiment comprises: an acquisition module configured to acquire a processed bitstream; an analysis module configured to analyze the processed bitstream to acquire image quality information; and a determination module configured to determine an image quality assessment area and corresponding quality assessment data based on the image quality information; A bitstream processing device is provided.

[0010] In aspect 5, the present embodiment comprises: A terminal device comprising one or more processors and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors realize a bitstream processing method according to an embodiment of the present application; Provide terminal equipment.

[0011] In aspect 6, the present embodiment comprises: A computer program is stored, and when the computer program is executed by a processor, the computer program realizes any one of the bitstream processing methods according to the embodiments of the present application. Provide a storage medium.

[0012] The above examples, other aspects and implementations of the present application are provided in more detail in the description of the drawings, detailed description, and claims. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a flowchart of a bitstream processing method according to an embodiment of the present application; [Figure 2] 4 is a flowchart of a further bitstream processing method according to an embodiment of the present application; [Figure 3] FIG. 2 is a schematic diagram of a media bitstream according to an embodiment of the present application; [Figure 4] 1 is a structural schematic diagram of a bitstream processing device according to an embodiment of the present application; [Figure 5] 1 is a structural schematic diagram of a bitstream processing device according to an embodiment of the present application; [Figure 6] 1 is a structural schematic diagram of a terminal device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0014] In order to clarify the purpose, technical solution and advantages of the present application, the following detailed description of the embodiments of the present application is given with reference to the accompanying drawings, in which the features of the embodiments and the features of the embodiments can be arbitrarily combined with each other unless inconsistent.

[0015] The steps depicted in the flowcharts of the figures may be implemented in a computer system as a group of computer-executable instructions, and although a logical order has been shown in the flowcharts, in some cases the steps may be performed in a different order than that shown or described herein.

[0016] In one embodiment, Figure 1 is a flowchart of a bitstream processing method according to an embodiment of the present application, which is applicable to processing bitstreams to improve the flexibility of image quality assessment, and which can be performed by a bitstream processing device, which can be implemented in software and / or hardware and integrated into a terminal device, which can be a sending device for transmitting the bitstream.

[0017] As shown in FIG. 1, the bitstream processing method according to the embodiment of the present application includes steps S110, S120, S130 and S140 as follows.

[0018] In S110, an image quality evaluation area is determined.

[0019] The image quality evaluation area can be understood as the entire image to be evaluated, or a plurality of images to be evaluated, or a partial area of ​​the entire image to be evaluated. Here, the partial area can be determined according to a specific scene, and the partial area can be an area formed after dividing the image to be evaluated.

[0020] The number of image quality evaluation areas may be one or more, and one quality evaluation area may correspond to one or more quality evaluation data.

[0021] This step does not limit how the image quality evaluation area is determined, and for example, the image quality evaluation area can be determined based on the image features of the image to be evaluated.

[0022] In S120, the quality evaluation data of the quality evaluation area is determined.

[0023] The quality assessment data may be considered to be data obtained after performing a quality assessment on a quality assessment region.

[0024] To determine the quality assessment data, an image quality assessment method can be adopted, including a full-reference quality assessment method, a reduced-reference quality assessment method, and a no-reference quality assessment method, where how the image assessment data is determined based on the image quality assessment method is not limited.

[0025] The quality evaluation data in the quality evaluation area may be one or more.

[0026] At S130, image quality information including the quality assessment data is determined.

[0027] Image quality information may be considered as information representative of image quality awaiting evaluation.

[0028] After the quality evaluation data is determined, the quality evaluation data can be added to the image quality information in this step. Here, there is no limitation on the content included in the image quality information.

[0029] In S140, the image quality information is added to the bitstream to obtain a processed bitstream.

[0030] After determining the image quality information, in this step, the image quality information can be added to the bitstream to obtain the processed bitstream, so that the terminal device receiving the processed bitstream can analyze the image quality information. The bitstream in this embodiment can be a media bitstream.

[0031] The bitstream processing method according to the embodiment of the present application determines image quality information including quality assessment data, and then adds the image quality information to the bitstream, thereby improving the flexibility of image quality assessment of the processed bitstream.

[0032] Based on the above embodiment, a modified version of the above embodiment is presented, and for the sake of simplicity, only the differences between the modified version and the above embodiment are described.

[0033] In one embodiment, the number of the image quality evaluation areas is at least one, and the image quality evaluation area includes the entire image area to be evaluated or multiple image areas to be evaluated, or the image quality evaluation area includes a portion of an area formed after area division of the image to be evaluated, and a division policy is determined for the image to be evaluated based on image features.

[0034] The image feature may be information representing the image feature, and the specific content of the image feature is not limited here. The image feature may be the image type or specific content in the image, etc.

[0035] The image quality evaluation area may be the entire image area to be evaluated, multiple image areas to be evaluated, or a partial area formed after segmentation of one or more images to be evaluated. The partial area may be considered to be one or more of the multiple areas formed after segmentation of the image to be evaluated. The number of areas included in the partial area is not limited, and the partial area may also be determined according to the actual scene.

[0036] In one embodiment, the image quality information includes an image quality evaluation method adopted to determine the quality evaluation data, a timestamp for generating the quality evaluation data, an identifier for a device for generating the quality evaluation data, an identifier for the quality evaluation data, i.e., the number of the quality evaluation data, the number of the image quality evaluation areas, information on the original image, including a link to the original image or statistical information on a part of the original image, and the image to be evaluated is a reconstructed image generated after processing the original image, a mathematical model adopted to perform no-reference image quality evaluation, image quality evaluation area division instruction information such as pqm_picture_division_idc, which may be an image feature of the image to be evaluated such as pqm_picture_type, and a commonality of the quality evaluation methods adopted for different image quality evaluation areas such as pqm_mixed_measurement_flag for indicating whether the quality evaluation methods adopted for different image quality evaluation areas are the same or not. information indicating the quality of the image, information indicating the component to be evaluated by the image quality evaluation area, information on scalable coding layers, including the number of scalable coding layers, the scalable coding layer to which the image quality evaluation area belongs, a quality evaluation method adopted for the i-th layer image in the images to be evaluated such as pqm_PSNR[i], and a quality evaluation method adopted for the n-th image quality evaluation area of ​​the k-th layer image in the images to be evaluated such as pqm_PSNR[i][j], where i, k, and n are positive integers; The quality evaluation information further includes one or more of information on an image sequence including a plurality of images awaiting evaluation corresponding to the quality evaluation area, the number of images included in the image sequence, the number of quality evaluation areas in each image in the image sequence, the quality evaluation method used to determine the jth (j is a positive integer) image quality evaluation area in the image sequence, information on the mth (m is a positive integer) quality evaluation area, the quality evaluation method used to determine the quality evaluation data of the hth (h is a positive integer) quality evaluation area, and the number of quality evaluation areas in the gth (g is a positive integer) image awaiting evaluation in the image sequence.

[0037] The image quality evaluation region division instruction information can be used to instruct the number of image quality evaluation regions and the calculation method of quality evaluation data.

[0038] The present application can calculate a quality assessment score, i.e., quality assessment data, for one image (region), and may also calculate quality assessment scores for multiple images (regions) awaiting evaluation, where multiple images awaiting evaluation are, i.e., a sequence of images.

[0039] The multiple quality assessment regions correspond to one group of quality assessment data. The multiple quality assessment regions may be multiple quality assessment regions in one image to be assessed, or may be quality assessment regions in multiple images to be assessed. The multiple quality assessment regions may be considered as a series of quality assessment regions.

[0040] This embodiment does not limit the combination method of the contents included in the quality information, and those skilled in the art can determine it according to the actual application scenario.

[0041] In one embodiment, the image quality information further includes division information of the image quality evaluation area and indication information as to whether the division information exists, and the division information of the image quality evaluation area includes one or more of an identifier of the quality evaluation area, a position of the quality evaluation area, and a size of the quality evaluation area.

[0042] In one embodiment, determining the image quality assessment area includes determining the image quality assessment area of ​​the image to be evaluated based on image characteristics of the image to be evaluated.

[0043] Different image features correspond to different image quality assessment regions, and the specific manner of determining the image quality assessment region is determined based on the selected image features and is not limited here.

[0044] In one embodiment, adding the image quality information to a bitstream to obtain a processed bitstream comprises the steps of:

[0045] Supplemental enhancement information is generated based on the image quality information.

[0046] The supplemental enhancement information is added to the bitstream to obtain a processed bitstream.

[0047] When generating the supplemental enhancement information based on the image quality information, the image quality information can be added to the supplemental enhancement information.

[0048] In one embodiment, adding the image quality information to a bitstream to obtain a processed bitstream comprises the steps of:

[0049] The image quality information is written to a media file, or the image quality information is written to a media file as level information of the corresponding image quality evaluation area.

[0050] The media file is appended to the bitstream to obtain a processed bitstream.

[0051] In one embodiment, one image quality evaluation area corresponds to one or more quality evaluation data, and the multiple quality evaluation data are determined by performing quality evaluation on different coding layers of the image to be evaluated with different components, or the multiple quality evaluation data are determined by performing quality evaluation on an image sequence with different components, or the multiple quality evaluation data are determined by adopting the same or different quality evaluation methods for one image quality evaluation area.

[0052] In one embodiment, the quality assessment methods employed in different image quality assessment areas may be the same or different.

[0053] In one embodiment, FIG. 2 is a flowchart of a further bitstream processing method according to an embodiment of the present application, which can be applied to improve the flexibility of image quality assessment, and which can be performed by a bitstream processing device, which can be implemented in software and / or hardware and integrated into a terminal device that receives the processed bitstream.

[0054] As shown in FIG. 2, the bitstream processing method according to the embodiment of the present application includes steps S210, S220 and S230 as follows.

[0055] In S210, the processed bitstream is obtained.

[0056] In S220, the processed bitstream is analyzed to obtain image quality information.

[0057] Here, the specific means of analysis is not limited, and the specific means of analysis is determined based on the specific means by which the terminal device that transmits the processed bitstream generates the processed bitstream, for example, obtaining image quality information from the media file of the processed bitstream or obtaining image quality information from the supplemental enhancement information.

[0058] In S230, an image quality evaluation area and corresponding quality evaluation data are determined based on the image quality information.

[0059] This step can extract the contents contained in the image quality information to determine the image quality evaluation area and corresponding quality evaluation data.For example, the image quality evaluation area and quality evaluation data can be determined by obtaining contents such as image quality evaluation area division instruction information, the number of image quality evaluation areas, and the quality evaluation method to be adopted.

[0060] For the contents not explained in detail in this embodiment, the above-mentioned embodiment can be referred to, and the explanation will be omitted here.

[0061] The bitstream processing method according to the embodiment of the present application obtains a processed bitstream containing image quality information, then analyzes the processed bitstream to obtain corresponding image quality evaluation areas and quality evaluation data, thereby effectively improving the flexibility of image quality evaluation.

[0062] The present application will be described below with reference to examples. A bitstream processing method according to an embodiment of the present application may be considered as a processing method for adding image quality information to a media bitstream, in which the present application determines image quality metric areas, i.e., image quality evaluation areas, based on image features, calculates quality evaluation scores, i.e., quality evaluation data, for each image quality metric area, and writes image quality information including the quality evaluation scores corresponding to the image quality metric areas into the media bitstream. The image quality information corresponding to the image quality evaluation areas is provided in the media bitstream so that an application receiving the media bitstream can provide more flexible and appropriate video image services based on the image quality metric areas (i.e., image quality evaluation areas) provided by the encoding side.

[0063] With the development of immersive media technology, more and more video applications can provide users with 360-degree panoramic videos. However, because the human viewpoint is typically around 120 degrees, the video screen a user actually sees at any given time is not a complete 360-degree panoramic video screen, but rather a portion of the screen corresponding to the user's current viewpoint. Furthermore, different users' perceptions of the screen are not identical, and they may be interested in different areas, target objects, or colors on the screen. Therefore, when viewing immersive videos, if a more flexible viewing or processing method could be provided based on the scene characteristics or pre-defined settings according to the user's needs and preferences, the user would be able to see a video screen with a more suitable image quality and enjoy a better user experience. ISO / IEC 23090-2 OMAF specifies quality-related quality levels, which indicate the relative quality order of the quality level range. The ranking value can be derived from a specific quality metric. ISO / IEC 23090-6 also uses quality rankings to measure viewpoint switching delays in immersive media metrics.

[0064] Image quality assessment is used to measure various distortions of images during the acquisition, processing, compression, storage, transmission, and playback processes. Objective image quality assessment methods use a mathematical model to provide results based on digital calculations. Based on whether image information needs to be referenced when calculating image quality, objective image quality assessment methods can be divided into three types: full-reference quality assessment, partial-reference quality assessment, and no-reference quality assessment. Full-reference and partial-reference quality assessment mainly analyze image features and calculate the quality of the reconstructed image by quantifying the difference between the original image and the reconstructed image. Currently, in the field of video encoding and decoding, full-reference image quality assessment methods are often used to evaluate the quality of video images. Commonly used full-reference image quality objective assessment methods include Peak Signal-to-Noise Ratio (PSNR), structural similarity (SSIM), Multi-Scale Structural SIMilarity (MS-SSIM), Video Quality Metric (VQM), and Mean Squared Error (MSE), etc. Based on the above image quality assessment methods, a series of improved quality assessment methods have been further derived, such as weighted Peak Signal-to-noise Ratio (wPSNR), weighted spherical Peak Signal-to-noise Ratio (WS-PSNR), and spherical Peak Signal-to-noise Ratio (S-PSNR), etc. Additionally, ISO / IEC 23001-10 specifies how to use media quality metrics. ITU-T J.143 specifies user needs for objective quality assessment of video in digital cable television. ITU-TP.914 specifies display requirements for 3D video quality assessment.ITU-T P.1204 specifies that video quality assessment should be carried out for highly reliable transmission streaming services with 4K resolution.

[0065] Currently, video coding is no longer limited to satisfying user viewing needs. As user needs continue to grow and change, post-transmission analysis of video coding and other visual applications are becoming more abundant. Objective evaluation results of image quality based on such references can only be obtained by calculation on the encoder side. Carrying this information in the media bitstream can provide application programs with coded image quality evaluation information, allowing application programs to provide more flexible and appropriate video services and further research related to quality evaluation based on the quality evaluation information. For example, a complete reference quality indicator can provide quality-oriented application programs with improved fidelity measurement accuracy. Because these application programs typically cannot obtain a complete reference, most can only evaluate quality based on parameters such as resolution, bit rate, quantization parameter, or quality ranking. Furthermore, user-generated content from mobile devices or drones is usually pre-compressed and the capture quality is unknown. Therefore, it is very necessary and meaningful to convey the original quality indicators of user-generated content to clients and services. Meanwhile, in video communications, one very practical scenario is converting an existing compressed video stream to another format / bit rate. After decoding, the video is re-encoded, and during this process, compression and scaling processes may be increased. During the video transcoding process, additional pixel processing such as noise reduction, contrast / brightness adjustment, etc. may also be performed. In the above scenarios, when performing the corresponding operations, the desired goal is to improve the perceived video quality rather than reducing the quality. Therefore, providing image quality assessment information in the media bitstream, i.e., the original quality assessment information and the subsequently generated quality assessment information, is also meaningful for the evaluation of the entire system and service.

[0066] At the JVET conference, a related proposal was put forward in the form of image quality metric Supplemental Enhancement Information (SEI) information, in which the image quality metric SEI information only includes a metric type (i.e., an objective evaluation method of image quality adopted, e.g., PSNR, SSIM, etc.) and quality metric values ​​of the luminance component and / or chrominance component in the metric type.

[0067] In some cases, objective quality assessment of an image is only considered by calculating the objective quality of the image per image. In a panoramic video, if a user only views a certain region of the image screen, the quality value calculated for the entire image is obviously meaningless. Therefore, it is necessary to determine quality metric regions based on image features and calculate a quality assessment score for each quality metric region. With the daily development of video technology, there are more and more application scenarios, such as panoramic videos and freeview videos, in which a video image screen corresponding to the user's viewpoint is played back according to the user's viewpoint. Therefore, calculating image quality assessment scores according to regions is also more meaningful. While objective evaluation results of image quality based on references can usually only be obtained by calculation on the encoder side, the SEI information carrying this information can provide relevant information to application programs, enabling them to provide more flexible and appropriate video services based on quality information.

[0068] In the present invention, bitstream processing is realized by the following embodiment. [Example]

[0069] An embodiment of the present application provides a method for processing a media bitstream, which determines an image quality assessment area based on an image to be quality assessed, i.e., image features of the image to be assessed, calculates a quality assessment score corresponding to the quality assessment area, and writes it into the media bitstream.

[0070] This embodiment includes the following steps S101, S102, S103, S104, and S105.

[0071] In step S101, an image awaiting quality evaluation is acquired.

[0072] The quality evaluation-awaiting image is a reconstructed image generated after the original image has undergone the encoding and decoding process. The original image refers to an image before it has undergone encoding and compression. Unless otherwise specified, the image may be a single still image or image data of one frame in a video.

[0073] In step S102, an image quality evaluation area is determined based on image features.

[0074] The image feature may be an image type, and the image type may be a normal image and a non-normal image, and the non-normal image includes at least a 360-degree panoramic video image, a free-view video image, and other multi-perspective video images, etc.

[0075] If the image is a normal image, the image quality evaluation area is the entire image area, and if the image is a non-normal image, the image can be divided into different areas, each of which corresponds to one image quality evaluation area. Usually, the division of the image quality evaluation area corresponds to a viewpoint area that a user can select and view, but this embodiment does not limit the division method of the image quality evaluation area.

[0076] In this embodiment, the division of image types into normal and non-normal images is merely an example, and the classification may be based on the actual type of image, or may follow other image classification methods that can be used to divide image viewpoint regions.

[0077] The image feature may also be the image screen content, and the image quality evaluation area may be divided depending on whether the image contains screen content such as a person, a vehicle, etc. Alternatively, the image quality evaluation area may be divided depending on whether the image screen content has movement or other changes.

[0078] In the H.265 / High Efficiency Video Coding (HEVC) standard and the Versatile Video Coding (VVC) standard, an image can be divided into tiles and slices, and VVC also supports dividing an image into subpictures. Therefore, one quality assessment region in this embodiment may correspond to one image or may be one subpicture, or one quality assessment region may include one or more tiles, or one or more slices.

[0079] In step S103, a quality evaluation score corresponding to the image quality evaluation area is calculated to determine image quality information.

[0080] To calculate the quality assessment score corresponding to the image quality assessment region, the image quality assessment method to be adopted must first be determined. If the image quality assessment method to be adopted is a full-reference quality assessment method, the image quality assessment region and the pixel values ​​of the region corresponding to the image quality assessment region in the original image are used to calculate the quality assessment score corresponding to the image quality assessment region according to a quality assessment mathematical model provided by the full-reference image quality assessment method. If the image quality assessment method to be adopted is a partial-reference quality assessment method, the image quality assessment region and some statistical information extracted from the original image required by the partial-reference image quality assessment method are used to calculate the quality assessment score corresponding to the image quality assessment region according to a quality assessment mathematical model provided by the method. For example, the digital watermark in the image or the visual sensitivity coefficient in the image can be extracted as some of the statistical information. If the image quality assessment method to be adopted is a no-reference quality assessment method, the original image does not need to be used, and the quality assessment score corresponding to the image quality assessment region is directly calculated according to a quality assessment mathematical model provided by the no-reference quality assessment method. If the no-reference quality assessment method is based on a neural network, the mathematical model should at least include information such as a neural network model and network parameters, and may further provide a dataset or training set adopted by the neural network model.

[0081] The quality evaluation score may correspond to one quality evaluation area or to multiple quality evaluation areas, and the multiple quality evaluation areas may be quality evaluation areas at different positions in the same image or may be quality evaluation areas in multiple images.

[0082] The image quality information should include a quality assessment score corresponding to at least one image quality assessment area.

[0083] The image quality information may include an image quality assessment method employed corresponding to a quality assessment score corresponding to at least one image quality assessment area.

[0084] If there are two or more image quality evaluation areas, the image quality information should include the number of image quality evaluation areas.

[0085] If there are two or more image quality evaluation areas, the image quality information may include image quality evaluation area division information, and the image quality evaluation area division information may include one or more of attributes such as the number of the image quality evaluation area, the position of the image quality evaluation area, the size of the image quality evaluation area, etc.

[0086] If the media bitstream is a scalable coding bitstream, the image quality information may include information on scalable coding layers, and the information on scalable coding layers may include the number of scalable coding layers and the scalable coding layer to which the image quality assessment area belongs.

[0087] In some embodiments, when the image quality assessment method included in the image quality information is a full reference or partial reference image quality assessment method, the image quality information may include information about the original image, and the information about the original image may be a link to obtain the original image or statistical information about part of the original image.

[0088] If the image quality assessment method included in the image quality information is a no-reference image quality assessment method, the image quality information may include a mathematical model adopted by the no-reference image quality assessment method. If the no-reference image quality assessment method is based on a neural network, the mathematical model may include information such as a neural network model, network parameters, and / or data set information adopted by the neural network model.

[0089] As described in the background art, if a video bitstream may undergo a transcoding operation or a process of decoding and then re-encoding during transmission, one or more quality evaluation scores may exist in one quality evaluation area. If two or more quality evaluation scores correspond to the same quality evaluation area, the image quality information should include the image quality evaluation methods adopted by the multiple quality evaluation scores corresponding to the one image quality evaluation area, and the image quality evaluation methods may be of different evaluation types, including full reference, partial reference, and no reference. The image quality information may also include one or more of the following information: a timestamp corresponding to the generation of each quality evaluation score, a device ID generating each quality evaluation score, a number generating each quality evaluation score, etc.

[0090] Different quality evaluation areas can adopt different image quality evaluation methods, and the selection of the image quality evaluation methods can be determined based on the image features included in the image quality evaluation areas. For example, if the image quality evaluation area is divided based on the image screen content as the image feature, it can be divided into an image quality evaluation area including important people and a background image quality evaluation area not including important content, and different image quality evaluation methods can be adopted for each area.

[0091] Different image quality assessment methods usually have their own specific assessment scales, for example, PSNR usually varies within a range of 20 to 60 dB (decibels), and SSIM varies within a range of 0 to 1. The quality assessment score in this embodiment may be a score originally calculated by the image quality assessment method, or may be a value converted by a mathematical formula, for example, by performing an integer rounding operation on the original score or by performing level classification on the original score.

[0092] In step S104, the image quality assessment information, ie, the image quality information, is written into the media bitstream.

[0093] In one implementation, corresponding image quality evaluation SEI information is generated based on the image quality evaluation information and written into the media bitstream, where image quality evaluation SEI information can be understood as SEI information including image quality information.

[0094] In addition to generating corresponding image quality evaluation SEI information based on the image quality evaluation information, the image quality evaluation area and the corresponding quality evaluation score may be used as level information of the image quality evaluation area, and may be written into different parts of the media bitstream, such as the media description part, as needed. The image quality evaluation information may also be written into corresponding parameter sets as needed, and the present application does not limit the manner of writing the image quality evaluation information into the media bitstream.

[0095] In step S105, the media bitstream is transmitted or stored.

[0096] A media bitstream of the image quality evaluation area and the corresponding quality evaluation score is transmitted or stored. When the image quality evaluation area and the corresponding quality evaluation score are used as level information of the image quality evaluation area, the media bitstream may be transmitted or stored based on different attributes of the level information instead of being transmitted or stored in scan order. [Example]

[0097] This embodiment provides an example of a form for realizing "writing the image quality information into a media bitstream" in step S104 in the first embodiment using image quality evaluation SEI information.

[0098] In one implementation, the image quality information can be written into supplemental enhancement information in the video bitstream, and Table 1 shows a schematic table for writing image quality information into supplemental enhancement information according to an embodiment of the present application, and a specific example is the structure shown in the table below.

[0099] [Table 1]

[0100] The SEI information includes a payload type (payloadType) and a payload size (payloadSize), where payloadType is used to indicate the type of SEI information, for example, if payloadType is PIC_QUALITY_MEASURE_INFO, it indicates that the SEI information is image quality evaluation SEI information, and payloadSize contains the image quality information actually carried. Table 2 shows a schematic table of payload sizes carrying image quality information according to an embodiment of the present application, as shown in the table below.

[0101] [Table 2] JPEG2025131889000004.jpg137162

[0102] pqm_picture_type indicates the type of image associated with the SEI information. As described in steps S102 and S103 in the first embodiment, when pqm_picture_type=0, it indicates that the image associated with the SEI information is a normal image and the image quality evaluation region is the entire image region, and when pqm_picture_type=1, it indicates that the image associated with the SEI information is a non-normal image and the image quality evaluation region may be a different quality evaluation region divided from the entire image or a different quality evaluation region divided from a sub-image.

[0103] Note that only one example of using pqm_picture_type as the image feature is shown here. According to the first embodiment, the image feature may be based on the content of the image screen, and whether to divide the image into quality evaluation areas is determined based on the content of the screen, and the image quality evaluation area division instruction information is determined. Table 3 is a schematic table of the image quality evaluation area division instruction information according to the embodiment of the present application. The following table shows one possible example.

[0104] [Table 3]

[0105] Furthermore, when it is necessary to divide an image into multiple image quality evaluation areas and perform quality evaluation on each of them, the sum of the multiple image quality evaluation areas may be a complete image or may be only a partial area of ​​a single image.

[0106] pqm_mixed_measurement_flag indicates whether or not to support using different quality assessment methods for different image quality assessment areas. If pqm_mixed_measurement_flag=0, it indicates that the same quality assessment method is used for all image quality assessment areas. If pqm_mixed_measurement_flag=1, it indicates that the same quality assessment method is used for all image quality assessment areas.

[0107] As described in step S102 in the first embodiment, when the image quality evaluation area is divided into an image quality evaluation area including important people and a background image quality evaluation area not including important content based on the image screen content as an image feature, different image quality evaluation methods can be adopted for each. Although not specified in the following embodiments, pqm_mixed_measurement_flag=1 is supported by default, that is, the same quality evaluation method is used for all image quality evaluation areas.

[0108] pqm_measurement_method indicates the image quality evaluation method to be adopted, and Table 4 is a schematic table for indicating the image quality evaluation method according to the embodiment of the present application. For specific indication methods, refer to the table below.

[0109] [Table 4]

[0110] This example only provides one example of the value of the image quality assessment method, and any quality assessment method for assessing image quality is acceptable, including employing two or more image quality assessment methods simultaneously, or using an external quality assessment method registration mechanism to identify different quality assessment methods, etc.

[0111] pqm_single_component_flag indicates whether to perform quality evaluation for only a single component of the image quality evaluation area. When pqm_single_component_flag=1, quality evaluation is performed for only a single component in the color space of the image quality evaluation area. When pqm_single_component_flag=0, quality evaluation must be performed for all components in the color space of the image quality evaluation area. It is supported to adopt different policies for multiple quality evaluation areas; that is, some quality evaluation areas perform quality evaluation for only a single component, and some quality evaluation areas perform quality evaluation for all components.

[0112] There are several ways to represent color spaces. For example, the commonly seen RGB color space describes color using three primary colors, with component R representing red, component G representing green, and component B representing blue. YUV color space describes color using luminance and chromaticity, with the luminance component being Y and the color difference components being composed of two independent signals, usually represented as U, V or Cb, Cr, etc. In the following examples, YUV will be used as an example, and unless otherwise specified, quality evaluation will be performed by default on only the luminance component Y.

[0113] pqm_PSNR[i] indicates that the quality assessment score of the image luminance component Y is calculated using the quality assessment method PSNR, or that the quality assessment scores of the image chrominance components U, V are calculated using the quality assessment method PSNR.

[0114] pqm_SSIM[i] indicates that the quality assessment score of the image luminance component Y is obtained using the quality assessment method SSIM, or that the quality assessment scores of the image color difference components U and V are calculated using the quality assessment method SSIM.

[0115] pqm_VQM[i] indicates that the quality assessment score of the image luminance component Y is calculated using the quality assessment method VQM, or that the quality assessment scores of the image color difference components U, V are calculated using the quality assessment method VQM.

[0116] pqm_MSSSIM[i] indicates that the quality assessment score of the image luminance component Y is to be calculated using the quality assessment method MS-SSIM, or that the quality assessment scores of the image chrominance components U and V are to be calculated using the quality assessment method MS-SSIM.

[0117] num_quality_regions indicates the number of image quality assessment regions.

[0118] pqm_quality_region_info_present_flag indicates whether quality evaluation region segmentation information is present. When pqm_quality_region_info_present_flag=1, it indicates that quality evaluation region segmentation information is present, and when pqm_quality_region_info_present_flag=0, it indicates that quality evaluation region segmentation information is not present. In this case, the region segmentation information in the panoramic image information may be used by default, or the quality evaluation region segmentation information may be transmitted separately in the media bitstream.

[0119] pqm_quality_region_info[i]() indicates information about the ith quality evaluation region, i.e., division information about the quality evaluation region. The quality evaluation region information may further include one or more of attributes such as the number of the image quality evaluation region, the position of the image quality evaluation region, and the size of the image quality evaluation region, where the position of the image quality evaluation region may include the horizontal position of the image quality evaluation region in the entire image and the vertical position of the image quality evaluation region in the entire image, or the position of the image quality evaluation region may include the horizontal coordinate of the upper left pixel point of the image quality evaluation region in the entire image and the vertical coordinate of the upper left pixel point of the image quality evaluation region in the entire image, and the size of the image quality evaluation region may include the width of the image quality evaluation region and the height of the image quality evaluation region.

[0120] pqm_PSNR[i][j] indicates that the quality evaluation score of the luminance component Y in the i-th image quality evaluation region is calculated using the quality evaluation method PSNR, or that the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region are calculated using the quality evaluation method PSNR.

[0121] pqm_SSIM[i][j] indicates that the quality evaluation score of the luminance component Y in the i-th image quality evaluation region is to be obtained using the quality evaluation method SSIM, or that the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region are to be calculated using the quality evaluation method SSIM.

[0122] pqm_VQM[i][j] indicates that the quality evaluation score of the luminance component Y in the i-th image quality evaluation region is calculated using the quality evaluation method VQM, or that the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region are calculated using the quality evaluation method VQM.

[0123] pqm_MSSSIM[i][j] indicates that the quality evaluation score of the luminance component Y in the i-th image quality evaluation region is to be calculated using the quality evaluation method MS-SSIM, or that the quality evaluation scores of the color difference components U and V in the i-th image quality evaluation region are to be calculated using the quality evaluation method MS-SSIM.

[0124] In this embodiment, the quality evaluation area corresponding to the quality evaluation score is one single image, or a quality evaluation area contained in a single image. In this case, the durability range of the image quality evaluation SEI information carrying the image quality evaluation information is the picture unit (PU) containing the image quality evaluation SEI information, which indicates that the image quality evaluation SEI information affects only one (frame) image to which it is related. [Example]

[0125] The High Efficiency Video Coding (H.265 / HEVC) standard and the Versatile Video Coding (H.266 / VVC) standard, both developed by the Joint Video Expert Teams (JVET) under ISO / IEC and ITU, support the concept of "layered video coding / scalable video coding." FIG. 3 is a schematic diagram of a media bitstream according to an embodiment of the present application. As shown in FIG. 3, a media bitstream may include one base layer and at least one enhancement layer to provide video image screens with different qualities. In this case, taking image 1 as an example, one quality assessment region in image 1 may have multiple quality assessment scores, which correspond to the base layer, the first enhancement layer, and the second enhancement layer, respectively.

[0126] This embodiment provides an example of a method for realizing "writing the image quality information into a media bitstream" in step S104 in embodiment 1 using image quality evaluation SEI information under scalable coding conditions. Table 5 is a schematic table for writing image quality information into a media bitstream according to an embodiment of the present application.

[0127] pqm_picture_type indicates the image type associated with the SEI information. See the description of pqm_picture_type in Example 2.

[0128] pqm_measurement_method indicates the image quality evaluation method to be adopted, and see the description of pqm_measurement_method in the second embodiment.

[0129] pqm_max_layers_minus1 indicates the number of coding layers, and the value of pqm_max_layers_minus1 should be equal to the value of vps_max_layers_minus1 in the video parameter set VPS (Video Parameter Set).

[0130] pqm_PSNR[i] indicates that the quality assessment score of the image in the i-th layer is calculated using the quality assessment method PSNR.

[0131] pqm_SSIM[i] indicates that the quality assessment score of the image in the i-th layer is calculated using the quality assessment method SSIM.

[0132] num_quality_regions indicates the number of image quality assessment regions.

[0133] pqm_quality_region_info[j]() indicates information on the j-th quality evaluation region. See the description of pqm_quality_region_info[i]() in the second embodiment.

[0134] pqm_PSNR[i][j] indicates that the quality assessment score of the j-th image quality assessment area in the i-th layer is calculated using the quality assessment method PSNR.

[0135] pqm_SSIM[i][j] indicates that the quality evaluation score of the j-th image quality evaluation area in the i-th layer is to be obtained using the quality evaluation method SSIM.

[0136] In this embodiment, in order to emphasize the characteristics of multi-layer coding for the sake of simplicity, the pqm_single_component_flag is omitted in the given example SEI information, and it is assumed that pqm_single_component_flag=1, i.e., quality evaluation is performed only on the luminance component of the image. However, this embodiment can support the case where pqm_single_component_flag=0, i.e., quality evaluation is performed separately on the luminance component and chrominance component of the image.

[0137] [Table 5]

[0138] This embodiment can also support a form in which different quality assessments of luminance and chrominance components are performed for different coding layers. For example, quality assessment is performed for only the luminance component and chrominance component of a base layer image, and quality assessment is performed for only the luminance component of an enhancement layer image. Alternatively, quality assessment is performed for only the luminance component and chrominance component of the highest enhancement layer image, and quality assessment is performed for only the luminance component of a base layer image, etc.

[0139] In this embodiment, the quality evaluation area corresponding to the quality evaluation score is one single base layer image or enhancement layer image, and may also be a quality evaluation area contained in a single base layer image or enhancement layer image. In this case, the durability range of the image quality evaluation SEI information carrying the image quality evaluation information is the access unit (AU) containing the image quality evaluation SEI information, which indicates that the image quality evaluation SEI information affects only one (frame) image to which it is related. [Example]

[0140] This embodiment provides an example of a method for realizing "writing the image quality information into SEI information in the media bitstream" in step S104 in embodiment 1 for images of multiple frames. Table 6 is a schematic table for writing image quality information into a bitstream according to an embodiment of the present application.

[0141] [Table 6] JPEG2025131889000009.jpg210164

[0142] pqm_picture_type indicates the image type associated with the SEI information. See the description of pqm_picture_type in Example 2.

[0143] pqm_measurement_method indicates the image quality evaluation method to be adopted, and see the description of pqm_measurement_method in the second embodiment.

[0144] pqm_sequence_info indicates information about an image sequence related to the image quality assessment SEI information. The information about the image sequence includes at least an image identification number for each frame in the image sequence, and image information in the image sequence can be determined based on the image identification number. The image identification number may be the decoding order or playback order of the image, or any other method that can determine image information.

[0145] pqm_quality_of_sequence_flag indicates whether a quality assessment score should be calculated for the image sequence. A value of 1 in pqm_quality_of_sequence_flag indicates that only one overall quality assessment score should be calculated for all images in the image sequence, whereas a value of 0 in pqm_quality_of_sequence_flag indicates that an individual quality assessment score should be calculated for each image in the image sequence.

[0146] In particular, this embodiment only provides an example of calculating the quality assessment score of an image sequence. For example, a quality assessment type of an image sequence, i.e., pqm_quality_of_sequence_type, may be designed to indicate how to calculate the quality assessment score of the image sequence. Table 7 is a schematic table of the quality assessment types of image sequences according to an embodiment of the present application, and the values ​​of pqm_quality_of_sequence_type may be as shown in the following table:

[0147] [Table 7]

[0148] pqm_sequence_PSNR indicates that the quality assessment score of an image sequence is calculated using the quality assessment method PSNR.

[0149] pqm_num_of_picture indicates the number of pictures in the picture sequence.

[0150] pqm_PSNR[i] indicates that the quality assessment score of the i-th image in the image sequence is calculated using the quality assessment method PSNR.

[0151] Referring to Example 2, pqm_mixed_measurement_flag is used to indicate whether different quality assessment methods are supported for different image quality assessment regions. In this example, in actual implementation, different quality assessment methods that are more suitable for a single image assessment region and a sequence of image assessment regions may be adopted. For example, a quality assessment method such as PSNR or SSIM may be adopted for a single image assessment region, while a quality assessment method such as VQM or wPSNR may be adopted for a sequence of image assessment regions. Similarly, different quality assessment methods may be adopted for different sequences of image assessment regions.

[0152] num_quality_regions indicates the number of quality assessment regions in each image in the image sequence.

[0153] pqm_quality_region_info[i]() indicates information about the i-th quality evaluation region sequence. See the description of pqm_quality_region_info[i]() in the second embodiment. Here, the i-th quality evaluation region sequence may be considered as a sequence formed by the i-th quality evaluation region in each image. For example, in the case of multiple images to be evaluated, each image to be evaluated is divided into multiple regions, and the i-th quality evaluation region in each image to be evaluated forms the quality evaluation region sequence.

[0154] pqm_sequence_PSNR[i] indicates that the quality evaluation score of the i-th quality evaluation region sequence is calculated using the quality evaluation method PSNR.

[0155] num_quality_regions[i] indicates the number of quality assessment regions in the i-th image in the image sequence.

[0156] pqm_quality_region_info[i][j]() indicates information on the j-th quality assessment region sequence in the i-th image in the image sequence. See the description of pqm_quality_region_info[i]() in the second embodiment.

[0157] pqm_PSNR[i][j] indicates that the quality assessment score of the j-th image quality assessment region in the i-th image is calculated using the quality assessment method PSNR.

[0158] Similar to Example 3, this example omits pqm_single_component_flag for the given example SEI information and assumes pqm_single_component_flag=1, i.e., performs quality evaluation on only the luminance component of the image or image sequence. However, this example can support the case where pqm_single_component_flag=0, i.e., performs quality evaluation on both the luminance and chrominance components of the image or image sequence, respectively.

[0159] This embodiment can also support adopting different luminance and chrominance component quality assessment forms for an image sequence and a single image, for example, performing quality assessment only on the luminance component and chrominance component of a single image, respectively, and performing quality assessment only on the luminance component for an image sequence, or performing quality assessment only on the luminance component and chrominance component of an image sequence, respectively, and performing quality assessment only on the luminance component for a single image, etc.

[0160] In this embodiment, the quality evaluation area corresponding to the quality evaluation score is a series of quality evaluation areas contained in one image sequence or image, and in this case, the durability range of the image quality evaluation SEI information carrying the image quality evaluation information is a coded video sequence (CVS) or coded layer video sequence (CLVS) containing the image quality evaluation SEI information. [Example]

[0161] This embodiment provides an example of another SEI information method for realizing "writing the image quality information into a media bitstream" in step S104 in embodiment 1. Table 8 is a schematic table for writing image quality information into a media bitstream according to an embodiment of the present application.

[0162] [Table 8]

[0163] pqm_cancel_flag, when set to 1, indicates that the image quality assessment SEI information cancels the durability of any previous image quality assessment SEI message and does not use the associated SEI function. Conversely, when set to 0, it indicates that the next message is image quality assessment information.

[0164] pqm_persistence_flag is used to indicate the persistence of the associated picture quality assessment SEI information. When pqm_persistence_flag=0, it indicates that the picture quality assessment SEI information applies only to the currently decoded picture. When pqm_persistence_flag=1, it indicates that the picture quality assessment SEI information applies to the currently decoded picture and continues to be used for all subsequent output pictures until one or more of the following conditions are met: (1) the bitstream ends; (2) a new picture sequence begins; or (3) the subsequently output picture is associated with picture quality assessment SEI information.

[0165] In this technical proposal, when pqm_persistence_flag=0, it may be understood that it is used to indicate that the quality assessment score of the associated image quality assessment SEI information is for the current decoded image or an image quality assessment area in the current decoded image, and when pqm_persistence_flag=1, it is used to indicate that the quality assessment score of the associated image quality assessment SEI information is for an image sequence consisting of the current decoded image and a subsequent output image, or an image quality assessment series in an image consisting of the current decoded image and a subsequent output image.

[0166] pqm_picture_type indicates the image type associated with the SEI information. See the description of pqm_picture_type in Example 2.

[0167] pqm_measurement_method indicates the image quality evaluation method to be adopted, and see the description of pqm_measurement_method in the second embodiment.

[0168] pqm_PSNR indicates the quality assessment score of the current image or of the image sequence consisting of the current image and the subsequent output image, calculated using the quality assessment method PSNR.

[0169] num_quality_regions indicates the number of quality assessment regions in an image in the image sequence.

[0170] pqm_quality_region_info[i]() indicates information on the i-th quality evaluation region, and see the description of pqm_quality_region_info[i]() in the second embodiment.

[0171] pqm_PSNR[i] indicates that the quality assessment method PSNR is used to calculate the quality assessment score of the i-th image quality assessment region in the current image or the i-th sequence of image quality assessment regions in the current image and the subsequent output image.

[0172] In this embodiment, pqm_cancel_flag and pqm_persistence_flag can be used to calculate a picture quality assessment score for a single picture quality assessment region, a Group of Pictures (GoP), one scene, or a series of picture quality assessment regions, etc. Here, a single picture quality assessment region includes a single image or one picture quality assessment region included in a single image. A series of picture quality assessment regions includes an entire image sequence, or a series of picture quality assessment regions composed of picture quality assessment regions included in the same or different images. [Example]

[0173] This embodiment provides a further example of a scheme for realizing "writing the image quality information into a media bitstream" in step S104 in the first embodiment.

[0174] JPEG2025131889000012.jpg99170

[0175] picture_type indicates the image type, where a value of 0 indicates that the image is one image quality evaluation area, and a value of 1 indicates that the image is divided into at least two image quality evaluation areas.

[0176] Measurement_method indicates the image quality evaluation method to be adopted, and see the description of pqm_measurement_method in the second embodiment.

[0177] quality_result indicates that the quality assessment result of the image should be calculated using the quality assessment method indicated by measurement_method.

[0178] num_quality_regions indicates the number of image quality assessment regions.

[0179] quality_region_info[i]() indicates information on the i-th quality evaluation region. See the description of pqm_quality_region_info[i]() in the second embodiment.

[0180] measurement_method[i] indicates the image quality assessment method to be adopted for the i-th quality assessment area.

[0181] quality_result[i] indicates that the quality evaluation result of the i-th quality evaluation area is to be calculated using the quality evaluation method indicated by measurement_method[i].

[0182] JPEG2025131889000013.jpg51170

[0183] The segment_top_left_x indicates the horizontal coordinate of the top left pixel point of the quality assessment region in the entire image.

[0184] The segment_top_left_y indicates the vertical coordinate of the top left pixel point of the quality assessment region in the entire image.

[0185] segment_width[i] indicates the pixel width of the quality assessment region.

[0186] segment_height[i] indicates the pixel height of the quality assessment region.

[0187] This embodiment shows only one simple example. Similarly, the embodiments described in the first to fifth embodiments can store image quality evaluation data in a file using the ISO base media file format, and the description thereof will be omitted here. [Example]

[0188] In step S701, a media bitstream is obtained.

[0189] The present technical solution does not limit the type and manner of the acquired media bitstream.

[0190] In step S702, the media bitstream is analyzed to obtain image quality evaluation information, ie, image quality information.

[0191] The media bitstream is analyzed to obtain image quality information carried in the media bitstream, which may be included in SEI information in the media bitstream.

[0192] In addition to generating corresponding image quality evaluation SEI information based on image quality evaluation information as described in step S104 of the first embodiment, the image quality evaluation area and the corresponding quality evaluation score may be used as level information of the image quality evaluation area and written into a different part of the media bitstream, for example, the media description part, as needed. Therefore, the present embodiment does not limit the method of obtaining image quality evaluation information from the media bitstream.

[0193] In step S703, an image quality evaluation area and a corresponding quality evaluation score are determined based on the image quality evaluation information.

[0194] Before determining the image quality evaluation areas and the corresponding quality evaluation scores based on the image quality evaluation information, the method may further include determining, based on the image quality evaluation information, image quality evaluation area division instruction information corresponding to the image quality evaluation information and the number of the image quality evaluation areas. Determining the image quality evaluation areas and the corresponding quality evaluation scores may further include determining a quality evaluation method corresponding to the quality evaluation scores.

[0195] In one embodiment, an embodiment of the present application provides a bitstream processing device, and FIG. 4 is a structural schematic diagram of a bitstream processing device according to an embodiment of the present application, which can be integrated into a terminal device. As shown in FIG. 4, the device includes a first determination module 41, a second determination module 42, a third determination module 43 and an additional module 44 as follows:

[0196] The first determining module 41 is configured to determine an image quality assessment area.

[0197] The second determining module 42 is configured to determine quality assessment data of the quality assessment area.

[0198] The third determining module 43 is configured to determine image quality information including said quality assessment data.

[0199] The appending module 44 is configured to append said image quality information to the bitstream to obtain a processed bitstream.

[0200] The bitstream processing device of this embodiment is used to realize the bitstream processing method of the embodiment shown in Figure 1, and the realization principle and technical effect of the bitstream processing device of this embodiment are similar to those of the bitstream processing method of the embodiment shown in Figure 1, so the description will be omitted here.

[0201] Based on the above embodiment, a modified version of the above embodiment is presented, and for the sake of simplicity, only the differences between the modified version and the above embodiment are described.

[0202] In one embodiment, the number of the image quality evaluation areas is at least one, and the image quality evaluation area includes the entire image area to be evaluated or multiple image areas to be evaluated, or the image quality evaluation area includes a portion of an area formed after area division of the image to be evaluated, and a division policy is determined for the image to be evaluated based on image features.

[0203] In one embodiment, the image quality information includes an image quality evaluation method adopted to determine the quality evaluation data, a timestamp for generating the quality evaluation data, an identifier for the device for generating the quality evaluation data, an identifier for the quality evaluation data, the number of the image quality evaluation areas, information on an original image, including a link to the original image or statistical information on a part of the original image, and the image to be evaluated is a reconstructed image generated after processing the original image, a mathematical model adopted to perform no-reference image quality evaluation, image quality evaluation area division type indication information, indication information on the commonality of the quality evaluation methods adopted for different image quality evaluation areas, indication information on the components to be evaluated by the image quality evaluation area, and information on scalable coding layers, including the number of scalable coding layers, the image quality, The information further includes one or more of the following: information on the scalable coding layer to which the quality evaluation area belongs, a quality evaluation method adopted for the i-th layer image in the images awaiting evaluation, and a quality evaluation method adopted for the n-th image quality evaluation area of ​​the k-th layer image in the images awaiting evaluation, where i, k, and n are positive integers; information on an image sequence including multiple images awaiting evaluation that correspond to the image quality evaluation area; the number of images included in the image sequence; the number of quality evaluation areas in each image in the image sequence; the quality evaluation method adopted to determine the j-th image quality evaluation area (j is a positive integer) in the image sequence; information on the m-th quality evaluation area (m is a positive integer); the quality evaluation method adopted to determine the quality evaluation data of the h-th quality evaluation area (h is a positive integer); and the number of quality evaluation areas in the g-th image awaiting evaluation (g is a positive integer) in the image sequence.

[0204] In one embodiment, the image quality information further includes division information of the image quality evaluation area and indication information as to whether the division information exists, and the division information of the image quality evaluation area includes one or more of an identifier of the quality evaluation area, a position of the quality evaluation area, and a size of the quality evaluation area.

[0205] In one embodiment, the first determining module 41 is specifically configured to determine the image quality evaluation area of ​​the image to be evaluated based on the image features of the image to be evaluated.

[0206] In one embodiment, adding the image quality information to the bitstream to obtain a processed bitstream by the adding module 44 includes generating supplemental enhancement information based on the image quality information, and adding the supplemental enhancement information to the bitstream to obtain a processed bitstream.

[0207] In one embodiment, adding the image quality information to the bitstream by the adding module 44 to obtain a processed bitstream includes writing the image quality information to a media file, or writing the image quality information to the media file as level information of the corresponding image quality evaluation area, and adding the media file to the bitstream to obtain a processed bitstream.

[0208] In one embodiment, the additional module 44 determines that one image quality assessment area corresponds to one or more quality assessment data, and the multiple quality assessment data are determined by performing quality assessment on different coding layers of the image to be assessed with different components, or by performing quality assessment on an image sequence with different components, or by employing different quality assessment methods for one image quality assessment area.

[0209] In one embodiment, the quality assessment methods employed by the addition module 44 for different image quality assessment areas may be the same or different.

[0210] In one embodiment, the present invention further provides a bitstream processing device, which is integrated into a terminal device for receiving and processing the bitstream. Figure 5 is a structural diagram of the bitstream processing device according to the present invention. As shown in Figure 5, the device includes an acquisition module 51, an analysis module 52 and a determination module 53 as follows:

[0211] The acquisition module 51 is configured to acquire the processed bitstream.

[0212] An analysis module 52 is configured to analyze the processed bitstream to obtain image quality information.

[0213] The determining module 53 is configured to determine an image quality assessment area and corresponding quality assessment data based on the image quality information.

[0214] The bitstream processing device of this embodiment is used to realize the bitstream processing method of the embodiment shown in Figure 2, and the realization principle and technical effect of the bitstream processing device of this embodiment are similar to those of the bitstream processing method of the embodiment shown in Figure 2, so the description will be omitted here.

[0215] In one embodiment, the embodiment of the present application further provides a terminal device, and Figure 6 is a structural schematic diagram of the terminal device of the embodiment of the present application. As shown in Figure 6, the terminal device of the embodiment of the present application includes one or more processors 61 and a storage device 62. The processor 61 in the terminal device may be one or more. In Figure 6, one processor 61 is taken as an example, and the storage device 62 is used to store one or more programs. When the one or more programs are executed by the one or more processors 61, the one or more processors 61 realize the bitstream processing method described in the embodiment of the present application.

[0216] The terminal device further comprises a communication device 63 , an input device 64 and an output device 65 .

[0217] The processor 61, storage device 62, communication device 63, input device 64 and output device 65 in the terminal device can be connected via a bus or other methods, and FIG. 6 shows an example in which they are connected via a bus.

[0218] The input device 64 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the terminal equipment. The output device 65 can include a display device such as a display.

[0219] The communication device 63 may include a receiver and a transmitter. The communication device 63 is configured to transmit and receive information under the control of the processor 61. The information includes, but is not limited to, a processed bitstream.

[0220] The storage device 62 can be used as a computer-readable storage medium to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the bitstream processing method according to an embodiment of the present application (e.g., the first determination module 41, the second determination module 42, the third determination module 43, and the addition module 44 in the bitstream processing device, or the acquisition module 51, the analysis module 52, and the determination module 53 in the bitstream processing device). The storage device 62 may include a program storage area and a data storage area, where the program storage area can store an operating system and application programs required for at least one function, and the data storage area can store data generated based on the use of the terminal device. The storage device 62 may also include high-speed random access memory and may further include non-volatile memory such as at least one magnetic disk storage device, flash memory, or other non-volatile solid-state storage device. In some embodiments, the storage device 62 can include memory located remotely from the processor 61, and these remote memories can be connected to the device via a network. Examples of such networks may include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0221] An embodiment of the present application further provides a storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, it realizes any of the methods in the embodiments of the present application, and when the computer program is stored in the storage medium, and when the computer program is executed by a processor, it realizes any of the bitstream processing methods in the embodiments of the present application.

[0222] The bitstream processing method includes determining an image quality evaluation area, determining quality evaluation data for the quality evaluation area, determining image quality information including the quality evaluation data, and adding the image quality information to a bitstream to obtain a processed bitstream.

[0223] The bitstream processing method includes obtaining a processed bitstream, analyzing the processed bitstream to obtain image quality information, and determining an image quality evaluation area and corresponding quality evaluation data based on the image quality information.

[0224] The computer storage medium of the present application may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. Further specific examples (a non-exhaustive list) of computer-readable storage media include an electrical connection having one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may be any tangible medium containing or storing a program that can be used in or in connection with an instruction execution system, apparatus, or device.

[0225] A computer-readable signal medium may include a propagated data signal, either in baseband or as part of a carrier wave, having computer-readable program code carried therein. Such a propagated data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which is capable of transmitting, propagating, or transporting a program for use in or in connection with an instruction execution system, apparatus, or device.

[0226] The program code contained in the computer readable medium may be transmitted over any suitable medium, including, but not limited to, electrical wire, optical cable, radio frequency (RF), or the like, or any suitable combination of the above.

[0227] Computer program code for carrying out the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or a business server. When referring to a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider).

[0228] The above are merely a number of embodiments of the present application, and are not intended to limit the scope of protection of the present application.

[0229] Those skilled in the art will appreciate that the term terminal equipment includes any suitable type of wireless user equipment, including, for example, a mobile phone, a portable data processing device, a portable network browser, or a mobile station mounted on a vehicle.

[0230] In general, various embodiments of the present application may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware while other aspects may be implemented in firmware or software executable by a controller, microprocessor, or other computing device, and the present application is not limited thereto.

[0231] Embodiments of the present application may be implemented by execution of computer program instructions by a data processor of a mobile device, for example in a processor entity, by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or target code written in any combination of one or more programming languages.

[0232] Any logic flow block diagrams in the drawings herein may represent program steps, interconnected logic circuits, modules, and functions, or combinations of program steps and logic circuits, modules, and functions. Computer programs may be stored in memory. The memory may be of any type suitable for the local technology environment and may be implemented with any suitable data storage technology, such as, but not limited to, read-only memory (ROM), random access memory (RAM), optical storage devices and systems (Digital Versatile Disc (DVD) or Compact Disc (CD)), etc. Computer-readable media may include non-transitory storage media. The data processor may be of any type suitable for the local technology environment, such as, but not limited to, a general purpose computer, a special purpose computer, a microprocessor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), and a processor based on a multi-core processor architecture.

[0233] The above provides a detailed description of several embodiments of the present application by way of several non-limiting examples. However, when considered in conjunction with the drawings and claims, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art without departing from the scope of the present application. Therefore, the appropriate scope of the present application is determined based on the claims.

Claims

1. determining an image quality assessment region; determining quality assessment data of the quality assessment area; determining image quality information including said quality assessment data; adding the image quality information to a bitstream to obtain a processed bitstream; The quality evaluation data is data obtained after performing quality evaluation on the quality evaluation area using at least one quality evaluation method. Bitstream processing methods.

2. the number of the image quality assessment areas is at least one; The one image quality evaluation area is One image, One sub-image, A region of the image that contains at least one tile; a region in which the image includes at least one slice; The method of claim 1.

3. One image quality assessment area corresponds to one or more quality assessment data, and the multiple quality assessment data are determined by performing quality assessment on different coding layers of the image to be assessed with different components, or the multiple quality assessment data are determined by performing quality assessment on image sequences with different components, or the multiple quality assessment data are determined by adopting different quality assessment methods for one image quality assessment area; The method of claim 1.

4. The quality assessment methods adopted for different image quality assessment areas may be the same or different. The method of claim 1.

5. The image quality information is an image quality assessment method employed to determine said quality assessment data; a timestamp for generating the quality assessment data; a device identifier for generating the quality evaluation data; an identifier of the quality assessment data; Information on the original image, information about the original image, including a link to the original image or statistical information of a part of the original image, and the image to be evaluated is a reconstructed image generated after processing the original image; and a mathematical model employed to perform no-reference image quality assessment. The method of claim 1.

6. The image quality information is the number of image quality assessment areas; Image quality evaluation area division instruction information; Indication of commonality of quality assessment methods adopted for different image quality assessment areas; Indication information of the component to be evaluated in the image quality evaluation area; Scalable coding layer information, the number of scalable coding layers, the scalable coding layer to which the image quality evaluation area belongs, the quality evaluation method adopted for the i-th layer image in the images to be evaluated, and the quality evaluation method adopted for the n-th image quality evaluation area in the k-th layer image in the images to be evaluated, and scalable coding layer information, where i, k, and n are positive integers. The method of claim 1.

7. The image quality information is information on an image sequence including a plurality of images to be evaluated, which corresponds to the image quality evaluation area; The number of images in the image sequence, the number of quality assessment regions in each image in the sequence; a quality assessment method employed to determine the j-th image quality assessment region (j is a positive integer) in the image sequence; Information on the mth (m is a positive integer) quality evaluation area; a quality evaluation method adopted to determine the quality evaluation data of the h-th quality evaluation area (h is a positive integer); the number of quality assessment regions in the gth image (g is a positive integer) awaiting assessment in the image sequence; The method of claim 1.

8. obtaining the processed bitstream; analyzing the processed bitstream to obtain image quality information; determining an image quality evaluation area and corresponding quality evaluation data based on the image quality information; The quality evaluation data is data obtained after performing quality evaluation on the quality evaluation area using at least one quality evaluation method. Bitstream processing methods.

9. the number of the image quality assessment areas is at least one; the image quality evaluation area includes the entire image area to be evaluated or a plurality of image areas to be evaluated, or the image quality evaluation area includes a part of an area formed after dividing the image to be evaluated; The image to be evaluated is subjected to a segmentation policy based on image characteristics. The method of claim 8.

10. the number of the image quality assessment areas is at least one; The one image quality evaluation area is One image, One sub-image, A region of the image that contains at least one tile; a region in which the image includes at least one slice; The method of claim 8.

11. One image quality assessment area corresponds to one or more quality assessment data, and the multiple quality assessment data are determined by performing quality assessment on different coding layers of the image to be assessed with different components, or the multiple quality assessment data are determined by performing quality assessment on image sequences with different components, or the multiple quality assessment data are determined by adopting different quality assessment methods for one image quality assessment area; The method of claim 8.

12. The quality assessment methods adopted for different image quality assessment areas may be the same or different. The method of claim 8.

13. The image quality information is an image quality assessment method employed to determine said quality assessment data; a timestamp for generating the quality assessment data; a device identifier for generating the quality evaluation data; an identifier of the quality assessment data; information of an original image, the information including a link to the original image or statistical information of a part of the original image, and the image to be evaluated is a reconstructed image generated after processing the original image; and a mathematical model employed to perform no-reference image quality assessment. The method of claim 8.

14. The image quality information is the number of image quality assessment areas; Image quality evaluation area division instruction information; Indication of commonality of quality assessment methods adopted for different image quality assessment areas; Indication information of the component to be evaluated in the image quality evaluation area; Scalable coding layer information, and information on the scalable coding layer, including the number of scalable coding layers, the scalable coding layer to which the image quality evaluation area belongs, the quality evaluation method adopted for the i-th layer image in the images to be evaluated, and the quality evaluation method adopted for the n-th image quality evaluation area in the k-th layer image in the images to be evaluated, where i, k, and n are positive integers. The method of claim 8.

15. The image quality information is information on an image sequence including a plurality of images to be evaluated, which corresponds to the image quality evaluation area; The number of images in the image sequence, the number of quality assessment regions in each image in the sequence; a quality assessment method employed to determine the j-th image quality assessment region (j is a positive integer) in the image sequence; Information on the mth (m is a positive integer) quality evaluation area; a quality evaluation method adopted to determine the quality evaluation data of the h-th quality evaluation area (h is a positive integer); the number of quality assessment regions in the gth image (g is a positive integer) awaiting assessment in the image sequence; The method of claim 8.

16. The image quality information is further including division information of the image quality evaluation area and indication information indicating whether the division information exists; The division information of the image quality evaluation area is including one or more of an identifier of the quality assessment area, a location of the quality assessment area, and a size of the quality assessment area; The method of claim 8.

17. one or more processors; A terminal device comprising a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 16. Terminal equipment.

18. A computer program is stored therein, The computer program, when executed by a processor, implements the method of any one of claims 1 to 16. storage medium.

Images