Quality assessment method and system for face images oriented to face semantic encoding and decoding
By performing face image reconstruction and key point detection on the decoding end, and calculating shape and identity offset index values, the problem that existing QoE technology cannot evaluate the quality of face semantic codec images is solved, and the face semantic codec images are realized Accurate quality assessment improves user experience.
Patent Information
- Application Number
- CN202310964759.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-08-02
AI Technical Summary
The existing quality of user experience (QoE) evaluation technology is not suitable for semantic codec, and it is impossible to accurately measure the semantic gap between the content of the video image generated by face semantic codec and the original video image, resulting in the inability to effectively evaluate the face semantic codec process. The quality of face image in it.
The decoding end uses the image reconstruction based on the reference face image and the key points of the current original face image sent by the encoded end to generate the current reconstructed face image, and obtain the key points through the key point detection model to calculate the face shape offset index value and identity offset index value, combined with preset index value for quality evaluation, to achieve evaluation of the quality of face semantic codec images.
It provides more accurate evaluation of face image quality, and can evaluate the quality of reconstructed face images from the perspectives of shape and identity, fills the gap that traditional QoE technology cannot evaluate in face semantic codec, and improves the semantic codec process User experience in.
Smart Images

Figure CN117292245B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image quality assessment, and in particular to a method and system for assessing the quality of facial images oriented to facial semantic encoding and decoding. Background Art
[0002] Semantic communication technology is a new generation of communication technology that draws on the human brain's ultra-high image and video compression performance mechanism and integrates the human brain's visual perception and cognitive mechanism into the communication process, achieving efficient semantic representation and high-definition and high-smooth reconstruction of videos at extremely low bit rates. Face semantic communication is a special case of semantic communication technology. It has received more attention due to its wide range of application scenarios and more mature technology.
[0003] However, the Quality of Experience (QoE) evaluation of communication systems is an evaluation technology for traditional coding and decoding methods (such as H.264, etc.), which is not applicable to semantic coding and decoding, and cannot be directly used to measure the semantic gap between the video image generated by face semantic coding and decoding and the original video image content.
[0004] Therefore, it is necessary to develop a quality assessment method and system for face images oriented to face semantic encoding and decoding, so as to realize the quality assessment of face images oriented to face semantic encoding and decoding. Summary of the invention
[0005] In view of the above problems, the embodiments of the present application provide a method and system for quality assessment of facial images for face semantic coding and decoding, so as to overcome the above problems or at least partially solve the above problems.
[0006] A first aspect of an embodiment of the present application provides a method for assessing the quality of a face image for face semantic coding and decoding, the method comprising:
[0007] The decoding end reconstructs the image according to the key points of the reference face image sent by the encoding end and the current original face image to generate the current reconstructed face image;
[0008] The decoding end obtains the key points of the current reconstructed face image through a key point detection model;
[0009] The decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain a face shape deviation index value of the current reconstructed face image;
[0010] The decoding end compares the similarity between the current reconstructed face image and the reference face image to obtain a face identity offset index value of the current reconstructed face image;
[0011] The decoding end obtains a quality assessment result of the current reconstructed face image based on the face identity offset index value, the face shape offset index value and a preset index value, wherein the preset index value includes at least one of the following: a device parameter value, a network performance parameter value, a player application parameter value, and a video source parameter value.
[0012] In an optional implementation manner, the decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape deviation index value of the current reconstructed face image, which at least includes:
[0013] The decoding end calculates the distance between each key point of the current reconstructed face image and the corresponding key point in the current original face image, and calculates the average key point distance based on the number of key points. The average key point distance is used to characterize the overall offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image; two corresponding key points are used to characterize the same position point on the face.
[0014] In an optional implementation, the calculation formula for the average key point distance is:
[0015]
[0016] Wherein, AKD represents the average key point distance, N represents the number of key points, and p i represents the i-th key point in the current original face image, represents the i-th key point of the currently reconstructed face image, where i is greater than or equal to 1 and less than or equal to N.
[0017] In an optional implementation manner, the decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape deviation index value of the current reconstructed face image, which at least includes:
[0018] The decoding end classifies the key points of the current reconstructed face image and the key points of the current original face image according to the parts to which they belong, and the parts include at least five parts: eyebrows, eyes, nose, lips and facial contour;
[0019] The decoding end calculates the distance between each key point and the center point of the part to which it belongs by using the Laplace operator L for the key points of the current reconstructed face image and the key points of the current original face image, and determines the calculated distance as the relative position of the key point relative to the center point of the part to which it belongs;
[0020] The decoding end calculates the offset between the relative position corresponding to the key point in the current reconstructed face image and the relative position of the corresponding key point in the current original face image, and sums the calculated offsets to obtain a Laplacian coordinate difference GLCD. The Laplacian coordinate difference GLCD is used to characterize the local offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image, and two corresponding key points are used to characterize the same position point on the face.
[0021] In an optional implementation, the calculation formula for the relative position is:
[0022]
[0023] Among them, p i represents the i-th key point of the current reconstructed face image or the i-th key point of the current original face image, represents the relative position of the ith key point of the current reconstructed face image or the ith key point of the current original face image, U(p i ) represents the set of key points belonging to the same part as the i-th key point, |U(p i )| represents the number of key points of the part to which the i-th key point belongs, p j Indicates p i Any key point of the part.
[0024] In an optional implementation, the calculation formula of the graph Laplacian coordinate difference GLCD is:
[0025]
[0026] Among them, p i represents the i-th key point in the current original face image, represents the i-th key point of the currently reconstructed face image, represents the relative position of the i-th key point of the current original face image, represents the relative position of the i-th key point of the currently reconstructed face image, GLCD represents the graph Laplacian coordinate difference GLCD, N represents the number of key points, and i is greater than or equal to 1 and less than or equal to N.
[0027] In an optional implementation manner, the decoding end compares the similarity between the current reconstructed face image and the reference face image to obtain a face identity offset index value of the current reconstructed face image, including:
[0028] The decoding end obtains a reconstructed face vector of the current reconstructed face image through a face recognition model, and obtains a reference face vector of the current reference face image through the face recognition model;
[0029] The decoding end calculates the cosine similarity between the reconstructed face vector and the reference face vector, and determines the cosine similarity as the face similarity.
[0030] In an optional implementation, the reconstructed face vector or the reference face vector is obtained by extracting a vector output by the last layer of the face recognition model;
[0031] The key point detection model used to obtain the key points of the current reconstructed face image is the same as the key point detection model used by the encoding end to obtain the key points of the current original face image.
[0032] In an optional embodiment, the decoding end calculates the face shape offset index value of the currently reconstructed face image generated every N frames, and calculates the face identity offset index value of the currently reconstructed face image generated every M frames, where N is an integer greater than or equal to zero, and M is an integer greater than N.
[0033] In an optional embodiment, the method further includes:
[0034] According to the quality assessment result of the current reconstructed face image, the encoder parameters used by the decoding end for face semantic encoding and the decoder parameters used by the decoding end for face semantic decoding are adjusted.
[0035] A second aspect of the embodiment of the present application further provides a quality assessment system for face images oriented to face semantic coding and decoding, comprising: an encoding end and a decoding end for point-to-point communication;
[0036] The encoding end is used to generate and send key points of a reference face image and a current original face image to the decoding end;
[0037] The decoding end is used to generate a preset index value of the current reconstructed face image, and to execute the face semantic encoding and decoding oriented face image quality assessment method described in any one of the first aspects of the embodiments of the present application, and to obtain the quality assessment result of the current reconstructed face image based on the face identity offset index value, face shape offset index value and preset index value of the current reconstructed face image.
[0038] A third aspect of the embodiment of the present application further provides a face image quality assessment system for face semantic encoding and decoding, comprising: an encoding end, a decoding end, and a cloud server;
[0039] The encoding end is used to generate and send key points of a reference face image and a current original face image to the decoding end and the cloud server respectively;
[0040] The cloud server is used to execute the face image quality assessment method for face semantic coding and decoding described in any one of the first aspects of the embodiments of the present application, obtain the face identity offset index value and the face shape offset index value of the current reconstructed face image and send them to the decoding end;
[0041] The decoding end is used to generate a preset index value of the current reconstructed face image, and obtain a quality assessment result of the current reconstructed face image based on the face identity offset index value, face shape offset index value and preset index value of the current reconstructed face image.
[0042] The fourth aspect of this embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the method for quality assessment of facial images for facial semantic coding and decoding as described in any one of the first aspects of the embodiments of the present application are implemented.
[0043] The fifth aspect of the embodiments of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the quality assessment method of a facial image for facial semantic encoding and decoding as described in any one of the first aspect of the embodiments of the present application are implemented.
[0044] A sixth aspect of an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any one of the methods for quality assessment of facial images for facial semantic encoding and decoding described in the first aspect.
[0045] The embodiment of the present application provides a method and system for assessing the quality of facial images for facial semantic coding and decoding, the method comprising: a decoding end reconstructing an image according to key points of a reference facial image and a current original facial image sent by an encoding end, to generate a current reconstructed facial image; the decoding end obtains the key points of the current reconstructed facial image through a key point detection model; the decoding end compares the key points of the current reconstructed facial image with the key points of the current original facial image to obtain a facial shape offset index value of the current reconstructed facial image; the decoding end compares the similarity of the current reconstructed facial image with the reference facial image to obtain a facial identity offset index value of the current reconstructed facial image; the decoding end obtains a quality assessment result of the current reconstructed facial image based on the facial identity offset index value, the facial shape offset index value and a preset index value, wherein the preset index value includes at least one of the following: a device parameter value, a network performance parameter value, a player application parameter value, and a video source parameter value. The embodiment of the present application compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape offset index value of the current reconstructed face image on the one hand, and compares the similarity of the current reconstructed face image with the reference face image on the other hand to obtain the face identity offset index value of the current reconstructed face image. The quality of the reconstructed face image is evaluated from the two perspectives of face shape offset and face identity offset combined with preset index values, and a targeted QoE factor is proposed for face semantic encoding and decoding, thereby achieving more accurate face image quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0047] Figure 1 It is a flowchart of the steps of a method for quality assessment of a face image for face semantic encoding and decoding provided by an embodiment of the present application;
[0048] Figure 2 is a schematic diagram of local deformation of a human face provided in an embodiment of the present application;
[0049] Figure 3 This is a schematic diagram of a facial feature vector extraction process provided by an embodiment of the present application;
[0050] Figure 4 is a schematic diagram of an inter-frame calculation provided in an embodiment of the present application;
[0051] Figure 5It is a schematic diagram of the calculation process of a method for assessing the quality of a face image for face semantic encoding and decoding provided by an embodiment of the present application;
[0052] Figure 6 It is a structural schematic diagram of a face image quality assessment system for face semantic encoding and decoding provided by an embodiment of the present application;
[0053] Figure 7 It is a structural schematic diagram of another face image quality assessment system for face semantic encoding and decoding provided by an embodiment of the present application;
[0054] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.
[0056] Semantic communication refers to the use of semantic information of the transmitted content for encoding and decoding, which can remove redundant data, reduce the amount of transmitted data, and achieve more efficient communication technology. Facial semantic communication is a branch of semantic communication. It refers to the communication process of using facial images as transmission content, analyzing the semantic information in the image, encoding and decoding the facial image, and reconstructing the facial image.
[0057] In the related technologies, the user experience quality (QoE) evaluation of the communication system is an evaluation technology for traditional coding and decoding methods (such as H.264, etc.), which is not applicable to semantic coding and decoding, and cannot be directly used to measure the semantic gap between the video image generated by face semantic coding and decoding and the original video image content. Therefore, directly using the original QoE technology to evaluate the quality of the reconstructed face image generated in the face semantic coding and decoding process cannot obtain accurate evaluation results.
[0058] In view of the above problems, the embodiment of the present application proposes a quality assessment method and system for face images for face semantic coding and decoding to solve the problem that the above-mentioned original QoE technology is not suitable for evaluating the quality of face images. The quality assessment method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through some embodiments and their application scenarios.
[0059] This embodiment proposes a quality assessment method for face images oriented to face semantic encoding and decoding. Figure 1 , Figure 1 A flow chart of the steps of a method for assessing the quality of a face image for face semantic encoding and decoding is shown in FIG. Figure 1 As shown, the method includes:
[0060] Step S101, the decoding end performs image reconstruction according to the reference face image sent by the encoding end and the key points of the current original face image to generate a current reconstructed face image.
[0061] In this embodiment, the execution of face semantic communication mainly includes an encoding end and a decoding end, wherein the encoding end and the decoding end can be any terminal with communication function such as a computer, a mobile phone, a tablet, a PC, etc. In the process of face semantic communication, the encoding end sends a reference face image to the decoding end, the encoding end extracts key points from the current original face image, and sends the key points to the decoding end, and finally the decoding end reconstructs the image based on the reference face image and the key points using the generation network to obtain the current reconstructed face image. In this process, the encoding end only needs to send the reference face image and the key points of the current original face image, that is, the face image reconstruction can be realized at the decoding end, so as to achieve the purpose of face semantic communication, greatly reduce the transmitted image bit rate, and improve communication efficiency. In the embodiment of the present application, the image decoding method can adopt any image decoding algorithm or model, such as an image driving method, which is not limited in the embodiment of the present application.
[0062] In actual applications, under the semantic coding and decoding framework, the video / image obtained by the decoding end is generated by a generative network, and there is a semantic gap in the content between the generated video / image and the original video / image. Especially in the case of high compression rate, taking face semantic communication as an example, the decoding end only reconstructs the face through face key points and reference face images. Due to the limited morphological information contained in the key points, it is easy to cause the reconstructed face image to be deformed overall or partially. Since traditional QoE technology is not suitable for face coding and decoding, problems such as deformation in the reconstructed face image generated during the face coding and decoding process cannot be discovered or accurately evaluated by traditional QoE technology evaluation.
[0063] Step S102: the decoding end obtains the key points of the current reconstructed face image through a key point detection model.
[0064] In order to evaluate the image quality of the current reconstructed face image generated by the decoding end, in the embodiment of the present application, a key point detection model is used to extract multiple key points of the image from the current reconstructed face image. The key points represent the feature points of the face image, such as contour feature points, facial feature points, etc. In the present embodiment, the method for selecting the key points is not limited.
[0065] In an optional implementation, the key point detection model used to obtain the key points of the current reconstructed facial image is the same as the key point detection model used by the encoding end to obtain the key points of the current original facial image.
[0066] In actual application, there are many different algorithms for extracting or detecting key points. Different key point detection algorithms obtain different numbers of key points, and the key points correspond to different facial positions. Exemplarily, the number of key points can be 5 points, 68 points or 103 points, etc., which is mainly determined by the key point extraction algorithm. In this embodiment, in order to facilitate the subsequent comparison of the key points of the current reconstructed face image extracted with the key points of the current original face image, the key point detection model used by the encoding end to obtain the key points of the current original face image is selected, so that the encoding end extracts 68 key points through the key point detection model, and the decoding end can also extract 68 key points through the key point detection model, so as to facilitate the subsequent comparative analysis of the key points. In this embodiment, the selection of the key point detection algorithm or model is not restricted. Exemplarily, the encoding and decoding end can select a lightweight model such as a practical facial key point detection model (PFLD) as a key point detection model.
[0067] Step S103: the decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain a face shape deviation index value of the current reconstructed face image.
[0068] The face shape offset index value is used to characterize the face shape in the current reconstructed face image. Compared with the offset degree of the face shape in the current original face image, the larger the face shape offset index value, the larger the offset degree, and the lower the quality of the current reconstructed face image; the smaller the face shape offset index value, the smaller the offset degree, and the higher the quality of the current reconstructed face image. In this embodiment, the offset degree of the face shape in the two images is determined by comparing the key points of the current reconstructed face image with the key points of the current original face image. Specifically, it can be determined by the relative displacement relationship between the corresponding key points, for example, the relative displacement relationship between the key point "nose tip" of the current reconstructed face image and the key point "nose tip" in the current original face image.
[0069] In an optional implementation, in step S103, the decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape deviation index value of the current reconstructed face image, which at least includes:
[0070] The decoding end calculates the distance between each key point of the current reconstructed face image and the corresponding key point in the current original face image, and calculates the average key point distance based on the number of key points. The average key point distance is used to characterize the overall offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image; two corresponding key points are used to characterize the same position point on the face.
[0071] In this embodiment, two corresponding key points are used to represent the same position point on the face. The decoding end uses the key points extracted from the current reconstructed face image to determine the key points in the current original face image corresponding to the key points one by one. There is a one-to-one correspondence between the key points on the two images. For example, the "left corner of the eye" key point in the current reconstructed face image corresponds to the "left corner of the eye" key point in the current original face image. The two corresponding key points are used to represent the same position point on the face, that is, the characteristic information of the corner of the left eye.
[0072] The decoding end calculates the distance between each key point of the current reconstructed face image and the corresponding key point in the current original face image. In this embodiment, since the two corresponding key points represent the same position point on the face, under ideal conditions, the position of the key point of the current reconstructed face image should coincide with the position of the corresponding key point in the current original face image. There is a possibility of positional offset of the face in the image after semantic coding and decoding reconstruction, so the embodiment of the present application can determine the degree of offset of the key point position by calculating the distance between the two corresponding key points. The greater the distance between the two key points, the greater the degree of offset of the face at the corresponding position in the current reconstructed face image. Exemplarily, if the distance between the "left corner of the eye" key point in the current reconstructed face image and the "left corner of the eye" key point in the current original face image is too large, it means that the degree of positional offset of the left corner of the eye of the face in the current reconstructed face image is too large.
[0073] According to the number of key points, the average key point distance is calculated, and the average key point distance is used to characterize the overall offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image. In this embodiment, for each pair of relative key points, the distance between the key points is calculated, and then the average value is calculated according to the number of key points to obtain the average key point distance. Exemplarily, the number of key points is 5, and the distance between the key point A in the current reconstructed face image and the key point a in the current original face image is calculated to be 2mm, the distance between the key point B and the key point b is 1mm, the distance between the key point C and the key point c is 3mm, the distance between the key point D and the key point d is 4mm, and the distance between the key point F and the key point f is 2mm. According to the number of key points, the average key point distance is calculated to be 2.4mm, which characterizes the overall offset degree of the current reconstructed face image relative to the current original face image.
[0074] In an optional implementation, the calculation formula for the average key point distance is:
[0075]
[0076] Wherein, AKD represents the average key point distance, N represents the number of key points, and p i represents the i-th key point in the current original face image, represents the i-th key point of the current reconstructed face image, i being greater than or equal to 1 and less than or equal to N. In this embodiment, the degree of deviation of the overall face position of the current reconstructed face image relative to the current original face image is obtained by calculating the average Euclidean distance of all key points.
[0077] In an optional implementation manner, the decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape deviation index value of the current reconstructed face image, which at least includes:
[0078] The decoding end classifies the key points of the current reconstructed face image and the key points of the current original face image according to the parts to which they belong, and the parts include at least five parts: eyebrows, eyes, nose, lips and facial contour.
[0079] The decoding end calculates the distance from each key point to the center point of the part to which it belongs through the Laplace operator L for the key points of the current reconstructed face image and the key points of the current original face image, and determines the calculated distance as the relative position of the key point relative to the center point of the part to which it belongs.
[0080] The decoding end calculates the offset between the relative position corresponding to the key point in the current reconstructed face image and the relative position of the corresponding key point in the current original face image, and sums the calculated offsets to obtain a Laplacian coordinate difference GLCD. The Laplacian coordinate difference GLCD is used to characterize the local offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image, and two corresponding key points are used to characterize the same position point on the face.
[0081] In an embodiment of the present application, on the one hand, the overall deviation degree of the face of the current reconstructed face image can be determined based on the key points of the current reconstructed face image and the key points of the current original face image; on the other hand, the local deviation degree of the face of the current reconstructed face image can also be determined based on the key points of the current reconstructed face image and the key points of the current original face image.
[0082] The embodiment of the present application classifies the positions of each key point according to the parts to which it belongs, and the parts include at least five parts: eyebrows, eyes, nose, lips and facial contour. According to the positions of the key points, the key points in the face image are classified to determine the parts to which each key point belongs, and all the key points are divided into eyebrow key points, glasses key points, nose key points and facial contour key points. The key points of each part are analyzed separately to determine the degree of deviation of each part.
[0083] The distance from each key point to the center point of the part to which it belongs is calculated by the Laplace operator L, and the calculated distance is determined as the relative position of the key point relative to the center point of the part to which it belongs. Specifically, for the five classified parts: eyebrows, eyes, nose, lips and facial contour, the center point of each part is determined, for example, the center point of the nose is determined, and then for the nose key points belonging to the nose part, the distance from each nose key point to the nose center point is calculated by the Laplace operator L.
[0084] In an optional implementation, the calculation formula for the relative position is:
[0085]
[0086] Among them, p i represents the i-th key point of the current reconstructed face image or the i-th key point of the current original face image, represents the relative position of the ith key point of the current reconstructed face image or the ith key point of the current original face image, U(p i ) represents the set of key points belonging to the same part as the i-th key point, |U(p i )| represents the number of key points of the part to which the i-th key point belongs, pj Indicates p i Any key point of the part.
[0087] In this embodiment, the offset between the relative position corresponding to the key point in the current reconstructed face image and the relative position of the corresponding key point in the current original face image is calculated. Exemplarily, the distance from the nose key point A in the current reconstructed face image to the center of the nose is 1 cm, and the distance from the corresponding nose key point a to the center of the nose in the current original face image is 1.2 cm. The two corresponding key points are used to represent the same position point on the face, that is, the nose key point A and the nose key point a are in a one-to-one correspondence (for example, both points represent the position of the nose tip of the face). In this example, the offset of the relative position is 2 mm.
[0088] This embodiment sums the calculated offsets to obtain the Laplacian coordinate difference GLCD, which is used to characterize the local offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image. Specifically, the offsets of all key points are calculated, summed, and the total value is used as the Laplacian coordinate difference GLCD. The total value of the offsets of all key points is used as the Laplacian coordinate difference GLCD. If the value is larger, it means that the deformation of the part in the current reconstructed face image is larger; if the value is smaller, it means that the deformation of the part in the current reconstructed face image is smaller.
[0089] In an optional implementation, the calculation formula of the graph Laplacian coordinate difference GLCD is:
[0090]
[0091] Among them, p i represents the i-th key point in the current original face image, represents the i-th key point of the currently reconstructed face image, represents the relative position of the i-th key point of the current original face image, represents the relative position of the i-th key point of the currently reconstructed face image, GLCD represents the graph Laplacian coordinate difference GLCD, N represents the number of key points, and i is greater than or equal to 1 and less than or equal to N.
[0092] Reference Figure 2 , Figure 2 A schematic diagram of local deformation of a human face is shown, such as Figure 2As shown, the embodiment of the present application divides the face into five parts: eyebrows, eyes, nose, lips and facial contour, and then for each part, calculates the relative position of the key point belonging to the part to the center point of the part. Figure (a) represents the current original face image, and Figure (b) represents the current reconstructed face image. The offset between the relative position corresponding to the key point in the current reconstructed face image in Figure (b) and the relative position of the corresponding key point in the current original face image in Figure (a) is calculated, and the total value of the offset is used as the Laplacian coordinate difference GLCD of the part. The value of the Laplacian coordinate difference GLCD represents the degree of deformation of the five parts of eyebrows, eyes, nose, lips and facial contour.
[0093] Step S104: the decoding end compares the similarity between the current reconstructed face image and the reference face image to obtain a face identity offset index value of the current reconstructed face image.
[0094] In an embodiment of the present application, the current reconstructed face image and the reference face image are compared for similarity, and the similarity indicates the degree of similarity between the contents of the two images, that is, the face similarity, which can be used as a face identity shift index value in the embodiment of the present application to characterize the degree of semantic identity shift of the face.
[0095] In an optional implementation manner, the decoding end compares the similarity between the current reconstructed face image and the reference face image to obtain a face identity offset index value of the current reconstructed face image, including:
[0096] The decoding end obtains a reconstructed face vector of the current reconstructed face image through a face recognition model, and obtains a reference face vector of the current reference face image through the face recognition model;
[0097] The decoding end calculates the cosine similarity between the reconstructed face vector and the reference face vector, and determines the cosine similarity as the face similarity.
[0098] Reference Figure 3 , Figure 3 A schematic diagram of the extraction process of a facial feature vector is shown in FIG. Figure 3As shown, the current reconstructed face image and the current reference face image are respectively input into the face recognition model to obtain the reconstructed face vector of the current reconstructed face image and the reference face vector of the current reference face image, wherein the reconstructed face vector is used to characterize the image features of the current reconstructed face image, and the reference face vector is used to characterize the image features of the current reference face image. The embodiment of the present application measures the similarity of face semantics by comparing and calculating the cosine distance (i.e., cosine similarity) between the two vectors, and uses it as the face similarity between the reference frame and the current reconstructed frame, reflecting the offset of the face semantic identity. When the face similarity between the current reference face image and the current reconstructed face image is low, it means that the reconstruction effect is poor, or the interval between the time point of shooting the current original face image and the time point of shooting the current reference face image is too far.
[0099] In an optional implementation, the reconstructed face vector or the reference face vector is obtained by extracting a vector output by the last layer of the face recognition model;
[0100] In actual application, there are many different face recognition models, and different face recognition models output different vectors. In this embodiment, in order to facilitate the comparison of the extracted reconstructed face vector of the current reconstructed face image with the reference face vector of the current reference face image and calculate the similarity, the same face recognition model is selected and the vector output by the last layer of the face recognition model is consistently used as the reconstructed face vector or the reference face vector. This ensures that a feature vector with consistent extraction standards is obtained to facilitate the subsequent calculation of cosine similarity for the vector. In this embodiment, the choice of face recognition model is not restricted. For example, a lightweight model such as MobilefaceNet can be selected as a face recognition model.
[0101] Step S105, the decoding end obtains a quality assessment result of the current reconstructed face image based on the face identity offset index value, the face shape offset index value and a preset index value, wherein the preset index value includes at least one of the following: a device parameter value, a network performance parameter value, a player application parameter value, and a video source parameter value.
[0102] The embodiment of the present application determines the overall displacement of the generated face image according to the average key point distance, determines the local shape offset of the generated face image according to the Laplacian coordinate difference GLCD, and obtains the face shape offset index value, and determines the degree of semantic identity offset of the face according to the face similarity, and obtains the face identity offset index value. The two are combined and compared with the preset index values (device parameter values, network performance parameter values, player application parameter values, and video source parameter values) for analysis, which can well reflect the shape of the face image generated in the face semantic coding and decoding, and the degree of semantic offset, thereby making up for the defects of traditional QoE in face semantic coding and decoding.
[0103] In an optional embodiment, the decoding end calculates the face shape offset index value of the currently reconstructed face image generated every N frames, and calculates the face identity offset index value of the currently reconstructed face image generated every M frames, where N is an integer greater than or equal to zero, and M is an integer greater than N.
[0104] Reference Figure 4 , Figure 4 A schematic diagram of frame-by-frame calculation is shown, such as Figure 4 As shown in , for a face image, shape information (average key point distance, graph Laplacian coordinate difference GLCD) is variable with respect to time, so high-frequency evaluation is required. The decoder can calculate the face shape offset index value of the current reconstructed face image generated every N frames, such as Figure 4 The average key point distance and / or the Laplacian coordinate difference GLCD are calculated for every 2 video frames in the video. Optionally, the face shape offset index value can be calculated every 2-3 frames. The semantic identity information (face similarity) is relatively stable with respect to time changes. Therefore, only a low-frequency evaluation is required. The face identity offset index value of the generated current reconstructed face image is calculated every M frames, so that M is an integer greater than N, such as Figure 4 The face similarity is calculated every 4 frames. Optionally, the face identity offset index value can be calculated every 5-10 frames. The embodiment of the present application sets a corresponding frame skipping mechanism according to the sensitivity of different QoE factors (face shape offset index value and face identity offset index value) to time, thereby alleviating the computing pressure on the decoding end.
[0105] In an optional embodiment, the method further includes:
[0106] According to the quality assessment result of the current reconstructed face image, the encoder parameters used by the decoding end for face semantic encoding and the decoder parameters used by the decoding end for face semantic decoding are adjusted.
[0107] In specific implementation, the encoder parameters and decoder parameters can be tuned according to the quality evaluation results, or the calculation frequency of the interval frames can be adjusted. For example, when the quality of the current reconstructed face image generated by using 5 key points is low, the model parameters of the codec can be adjusted to increase the number of key points to 10, thereby improving the image reconstruction quality by increasing the key points. Alternatively, when the quality of the current reconstructed face image is low, the lightweight model used by the encoding end is replaced with a larger model to generate a better reconstructed face image.
[0108] Figure 5 A schematic diagram of the calculation process of a face image quality assessment method for face semantic encoding and decoding is shown, as Figure 5 As shown, the decoder reconstructs the image according to the key points of the reference face image sent by the encoder and the current original face image to obtain the current reconstructed face image. Then, the key points of the current reconstructed face image are extracted to obtain the key points of the current reconstructed face image. The face shape offset index value (including the face shape offset index value of the current reconstructed face image) is obtained by comparing the key points of the current reconstructed face image with the key points of the current original face image. Figure 5 The average key point distance and Laplacian coordinate difference GLCD in the image are obtained. In addition, the face vectors of the current reconstructed face image and the reference face image are extracted respectively to obtain the reconstructed face vector of the current reconstructed face image and the reference face vector of the current reference face image. According to the reconstructed face vector and the reference face vector, the face identity offset index value of the current reconstructed face image is obtained ( Figure 5 The embodiment of the present application innovatively implements the application of average key point distance calculation, graph Laplacian coordinate difference GLCD calculation, and face similarity calculation in face semantic coding and decoding QoE calculation, fills the gap of traditional QoE in face semantic coding and decoding, proposes targeted QoE factors for face semantic coding and decoding, and can make more accurate image quality evaluation.
[0109] The second aspect of the present application embodiment provides a face image quality assessment system for face semantic coding and decoding, referring to Figure 6 , Figure 6 A schematic diagram of the structure of a face image quality assessment system for face semantic encoding and decoding is shown in FIG. Figure 6 As shown, the system includes: an encoding end and a decoding end for point-to-point communication;
[0110] The encoding end is used to generate and send key points of a reference face image and a current original face image to the decoding end;
[0111] The decoding end is used to generate a preset index value of the current reconstructed face image, and to execute the face semantic encoding and decoding oriented face image quality assessment method described in any one of the first aspects of the embodiments of the present application, and to obtain the quality assessment result of the current reconstructed face image based on the face identity offset index value, face shape offset index value and preset index value of the current reconstructed face image.
[0112] In this embodiment, a module for executing the quality assessment method for reconstructing a facial image can be set at the decoding end, and calculations can be performed at the decoding end. This places high requirements on the computing power of the decoding end and is suitable for application scenarios of point-to-point communication such as video communication and video conferencing.
[0113] A fourth aspect of the present application provides a face image quality assessment system for face semantic coding and decoding, referring to Figure 7 , Figure 7 FIG. 4 shows a schematic diagram of another face image quality assessment system for face semantic encoding and decoding. Figure 7 As shown, the system includes: an encoding end, a decoding end and a cloud server;
[0114] The encoding end is used to generate and send key points of a reference face image and a current original face image to the decoding end and the cloud server respectively;
[0115] The cloud server is used to execute the face image quality assessment method for face semantic coding and decoding described in any one of the first aspects of the embodiments of the present application, obtain the face identity offset index value and the face shape offset index value of the current reconstructed face image and send them to the decoding end;
[0116] The decoding end is used to generate a preset index value of the current reconstructed face image, and obtain a quality assessment result of the current reconstructed face image based on the face identity offset index value, face shape offset index value and preset index value of the current reconstructed face image.
[0117] In an embodiment of the present application, a functional module for performing quality assessment of reconstructed faces is set on a cloud server, thereby reducing the computing power requirements on the decoding end, which is suitable for one-to-many application scenarios such as security.
[0118] The present application also provides an electronic device, referring to Figure 8 , Figure 8 Schematic diagram of the structure of the electronic device proposed in the embodiment of the present application. Figure 8As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in a face image quality assessment method for face semantic encoding and decoding disclosed in an embodiment of the present application.
[0119] An embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in a method for quality assessment of facial images for facial semantic encoding and decoding disclosed in an embodiment of the present application are implemented.
[0120] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in a method for quality assessment of facial images for facial semantic encoding and decoding disclosed in an embodiment of the present application.
[0121] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0122] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0123] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0124] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0125] The above is a detailed introduction to the quality assessment method and system of face images for face semantic encoding and decoding provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for assessing the quality of face images for face semantic coding and decoding, characterized in that: Applied to video communication, for evaluating the semantic gap between the video generated by a decoding end and the original video in terms of content, the method comprises: The decoding end reconstructs the image according to the key points of the reference face image sent by the encoding end and the current original face image to generate the current reconstructed face image; The decoding end obtains the key points of the current reconstructed face image through a key point detection model; The decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape offset index value of the current reconstructed face image, which at least includes: the decoding end calculates the offset between the relative position corresponding to the key point in the current reconstructed face image and the relative position of the corresponding key point in the current original face image, and sums the calculated offset to obtain the Laplacian coordinate difference GLCD, the Laplacian coordinate difference GLCD is used to characterize the local offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image, and two corresponding key points are used to characterize the same position point on the face; The decoding end compares the similarity between the current reconstructed face image and the reference face image to obtain a face identity offset index value of the current reconstructed face image; the decoding end obtains a quality assessment result of the current reconstructed face image based on the face identity offset index value, the face shape offset index value and a preset index value, wherein the preset index value includes at least one of the following: a device parameter value, a network performance parameter value, a player application parameter value, and a video source parameter value; The decoding end adopts a corresponding frame skipping mechanism for the generated video according to the sensitivity of the face shape offset index value and the face identity offset index value to time, specifically including: calculating the face shape offset index value of the currently reconstructed face image generated every N frames of the generated video, and calculating the face identity offset index value of the currently reconstructed face image generated every M frames of the generated video, where N is an integer greater than or equal to zero, and M is an integer greater than N.
2. The quality assessment method according to claim 1, characterized in that: The decoding end compares the key points of the current reconstructed face image with the key points of the current original face image to obtain the face shape deviation index value of the current reconstructed face image, which at least includes: The decoding end calculates the distance between each key point of the current reconstructed face image and the corresponding key point in the current original face image, and calculates the average key point distance based on the number of key points. The average key point distance is used to characterize the overall offset of the reconstructed face in the current reconstructed face image relative to the original face in the current original face image; two corresponding key points are used to characterize the same position point on the face.
3. The quality assessment method according to claim 2, characterized in that: The calculation formula of the average key point distance is: Wherein, AKD represents the average key point distance, N represents the number of key points, and p i represents the i-th key point in the current original face image, represents the i-th key point of the currently reconstructed face image, where i is greater than or equal to 1 and less than or equal to N.
4. The quality assessment method according to claim 1 or 2, characterized in that: Also includes: The decoding end classifies the key points of the current reconstructed face image and the key points of the current original face image according to the parts to which they belong, and the parts include at least five parts: eyebrows, eyes, nose, lips and facial contour; The decoding end calculates the distance from each key point to the center point of the part to which it belongs through the Laplace operator L for the key points of the current reconstructed face image and the key points of the current original face image, and determines the calculated distance as the relative position of the key point relative to the center point of the part to which it belongs.
5. The quality assessment method according to claim 4, characterized in that: The calculation formula of the relative position is: Among them, p i represents the i-th key point of the current reconstructed face image or the i-th key point of the current original face image, represents the relative position of the ith key point of the current reconstructed face image or the ith key point of the current original face image, U(p i ) represents the set of key points belonging to the same part as the i-th key point, |U(p i )| represents the number of key points of the part to which the i-th key point belongs, p j Indicates p i Any key point of the part.
6. The quality assessment method according to claim 4, characterized in that: The calculation formula of the Laplacian coordinate difference GLCD is: Among them, p i represents the i-th key point in the current original face image, represents the i-th key point of the currently reconstructed face image, represents the relative position of the i-th key point of the current original face image, represents the relative position of the i-th key point of the currently reconstructed face image, GLCD represents the graph Laplacian coordinate difference GLCD, N represents the number of key points, and i is greater than or equal to 1 and less than or equal to N.
7. The quality assessment method according to claim 1, characterized in that: The decoding end compares the similarity between the current reconstructed face image and the reference face image to obtain a face identity offset index value of the current reconstructed face image, including: The decoding end obtains a reconstructed face vector of the current reconstructed face image through a face recognition model, and obtains a reference face vector of the current reference face image through the face recognition model; The decoding end calculates the cosine similarity between the reconstructed face vector and the reference face vector, and determines the cosine similarity as the face similarity.
8. The quality assessment method according to claim 7, characterized in that: The reconstructed face vector or the reference face vector is obtained by extracting the vector output by the last layer of the face recognition model; The key point detection model used to obtain the key points of the current reconstructed face image is the same as the key point detection model used by the encoding end to obtain the key points of the current original face image.
9. A face image quality assessment system for face semantic encoding and decoding, characterized in that: Applied to video communication, it is used to evaluate the semantic gap between the video generated by the decoding end and the original video in terms of content, including: the encoding end and the decoding end of point-to-point communication; The encoding end is used to generate and send key points of a reference face image and a current original face image to the decoding end; The decoding end is used to generate a preset index value of the current reconstructed facial image, and to execute the quality assessment method for reconstructed facial images for facial semantic encoding and decoding described in any one of claims 1-8, and to obtain a quality assessment result of the current reconstructed facial image based on the face identity offset index value, face shape offset index value and preset index value of the current reconstructed facial image.
10. A face image quality assessment system for face semantic encoding and decoding, characterized in that: Applied to video communication, it is used to evaluate the semantic gap between the video generated by the decoding end and the original video in terms of content, including: the encoding end, the decoding end, and the cloud server; The encoding end is used to generate and send key points of a reference face image and a current original face image to the decoding end and the cloud server respectively; The cloud server is used to execute the quality assessment method for reconstructing a face image for face semantic coding and decoding according to any one of claims 1 to 8, obtain the face identity offset index value and the face shape offset index value of the current reconstructed face image and send them to the decoding end; The decoding end is used to generate a preset index value of the current reconstructed face image, and obtain a quality assessment result of the current reconstructed face image based on the face identity offset index value, face shape offset index value and preset index value of the current reconstructed face image.
Citation Information
Patent Citations
Method for automatically generating face image according to visual attribute description
CN110598719A
Face image screening method and device
CN111639517A
Stretchable face image coding method and system for man-machine mixed vision
CN115880762A
Model training method, face image quality evaluation method, equipment and medium
CN115908260A
Image quality evaluation method and device, electronic equipment and computer program product
CN116052234A
