Image synthesis method, video conference method, device, and storage medium

Abstract

The one or more embodiments of the specification provide an image synthesis method, a video conference method, a device and a storage medium. The image synthesis method comprises: acquiring at least two frames of images to be synthesized; the at least two frames of images are divided into first type images and second type images according to image quality, and the image quality of the first type images is higher than that of the second type images; performing compression processing on the first type images to obtain compressed first type images; and synthesizing the compressed first type images in the at least two frames of images and the second type images in the at least two frames of images. It is beneficial to improve the harmony of the synthesized image.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and more particularly to an image synthesis method, a video conferencing method, an apparatus, and a storage medium. Background Technology

[0002] One type of image synthesis technique in related technologies refers to extracting at least a portion of image regions from at least two frames of images and synthesizing them to obtain a synthesized image. However, the image quality of the at least two frames of images may be different, resulting in a rather abrupt and disharmonious synthesized image.

[0003] For example, the surge in demand for video communication in recent years, coupled with the widespread adoption of mobile devices and high-speed networks, has led to the widespread use of image / video compositing technology in video conferencing, video calls, and live streaming platforms to protect privacy and enhance immersion and entertainment. This technology involves combining the human figure in an image captured by a camera with a pre-set virtual background. However, the image captured by the camera is limited by factors such as compression during network transmission, resulting in low image quality. The difference in quality between the captured image and the virtual background leads to inconsistent quality in different areas of the composite image, creating a jarring and discordant visual effect. Summary of the Invention

[0004] In view of the above, this specification provides one or more embodiments of an image synthesis method, a video conferencing method, an apparatus, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, an image synthesis method is proposed, comprising:

[0007] At least two frames of images to be synthesized are obtained; the at least two frames of images are divided into a first type of image and a second type of image according to their image quality, and the image quality of the first type of image is higher than that of the second type of image;

[0008] The first type of image is compressed to obtain the compressed first type of image;

[0009] The compressed first type of image from the at least two frames is combined with the second type of image from the at least two frames.

[0010] According to a second aspect of one or more embodiments of this specification, an image synthesis method is provided, comprising:

[0011] At least two frames of images to be synthesized are obtained; the at least two frames of images are divided into a first type of image and a second type of image according to their image quality, and the image quality of the first type of image is higher than that of the second type of image;

[0012] For any first type of image in the at least two frames, determine processing parameters for reducing the image quality of the first type of image, and process the first type of image using the processing parameters;

[0013] The processing result of the first type of image in the at least two images is combined with the second type of image in the at least two images to form a composite.

[0014] According to a third aspect of one or more embodiments of this specification, a video conferencing method is provided, comprising:

[0015] The system receives a foreground video frame sent by a client participating in a video conference, and acquires a background image synthesized with the foreground video frame; the image quality of the foreground video frame is lower than that of the background image.

[0016] The background image is compressed to obtain a compressed background image;

[0017] The compressed background image is combined with the foreground video frame to obtain a composite video frame that is sent to the client; the composite video frame is used to display on the client.

[0018] According to a fourth aspect of one or more embodiments of this specification, a video conferencing method is provided, comprising:

[0019] Receive video frames sent by at least two clients participating in the video conference; the at least two video frames are divided into a first type of video frame and a second type of video frame according to their image quality, and the video frame quality of the first type of video frame is higher than that of the second type of video frame.

[0020] The first type of video frames are compressed to obtain compressed first type of video frames;

[0021] The compressed first type of video frame from the at least two video frames is combined with the second type of video frame from the at least two video frames to generate a composite video frame that is sent to each of the clients; the composite video frame is used to display on each of the clients.

[0022] According to a fifth aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0023] processor;

[0024] Memory used to store processor-executable instructions;

[0025] The processor implements the method as described in any one of the first, second, third, or fourth aspects by running the executable instructions.

[0026] According to a sixth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in any one of the first, second, third, or fourth aspects.

[0027] The embodiments in this specification have the following beneficial effects:

[0028] At least two frames of images are acquired, which can be classified into a first category and a second category based on image quality, with the first category images having higher image quality than the second category images. Then, for any first category image in the at least two frames, the first category image is compressed to reduce its image quality, resulting in a compressed first category image. This compressed first category image has an image quality close to that of the second category image. The compressed first category image in the at least two frames is then combined with the second category image in the at least two frames to improve the harmony of the combined image, making the combined image natural and not abrupt. Attached Figure Description

[0029] Figure 1 This is a flowchart of the synthesis of two images in related technologies.

[0030] Figure 2A and Figure 2B This is a schematic diagram of different processes of an image synthesis method provided in an exemplary embodiment.

[0031] Figure 3 This is a schematic diagram of the structure of the U-net model provided in an exemplary embodiment.

[0032] Figure 4 This is a schematic diagram of the structure of a compression parameter estimation model provided in an exemplary embodiment.

[0033] Figure 5A This is a schematic diagram of the structure of a video conferencing system provided in an exemplary embodiment.

[0034] Figure 5B This is a flowchart illustrating a video conferencing method provided in an exemplary embodiment.

[0035] Figure 5C This is a schematic diagram illustrating the process of compositing a foreground video and a virtual background, provided in an exemplary embodiment.

[0036] Figure 6A This is a schematic diagram illustrating the effect of combining two images in a related technology.

[0037] Figure 6B This is a schematic diagram illustrating the effect of combining two frames of images in an exemplary embodiment.

[0038] Figure 7 This is a flowchart illustrating another video conferencing method provided in an exemplary embodiment.

[0039] Figure 8 This is a schematic flowchart of another image synthesis method provided in an exemplary embodiment.

[0040] Figure 9 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0041] Figure 10 This is a block diagram of an image synthesis apparatus provided in an exemplary embodiment. Detailed Implementation

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0043] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0044] One type of image synthesis technique in related technologies refers to extracting at least a portion of image regions from at least two images and synthesizing them to obtain a composite image, such as... Figure 1 Taking the synthesis of two frames as an example, the image quality of at least two frames may be different, resulting in different quality in different areas of the synthesized image, presenting a rather abrupt and disharmonious visual effect.

[0045] This section uses the synthesis of two images as an example: Image fusion based on virtual backgrounds is widely used to protect privacy or provide a more immersive and engaging experience. In scenarios such as video conferencing, video calls, or live streaming platforms, the image of a person in the foreground video captured by the camera device is synthesized with a virtual background to generate a composite video. Figure 1 One of the two frames in the image can be any frame from the foreground video, and the other frame can be a preset virtual background. Specifically, a portrait matting method is used to process each frame of the foreground video to obtain a composite texture map of the portrait in each frame. This composite texture map is used to extract the portrait from the input foreground video. Simultaneously, the position of the portrait in the virtual background is estimated to obtain positional information, such as a user selecting a position within the virtual background. Then, for each frame of the foreground video, the composite texture map and positional information of each frame are combined with the virtual background to output a composite video. However, the foreground video may undergo one or more lossy compression processes during acquisition and output. Due to compression distortion, the image quality of each frame in the foreground video degrades, while the virtual background does not suffer from this problem. This results in a difference in image quality between the foreground video and the virtual background captured by the camera, leading to a jarring and discordant visual effect in the composite image.

[0046] Of course, in practical applications, it is not limited to the synthesis of two frames of images. It can also be the synthesis of more than two frames of images. For example, synthesizing two or more frames of images into a panoramic image. If there are differences in the image quality between the images, the synthesized panoramic image will present a rather abrupt and disharmonious visual effect.

[0047] For example, in scenarios such as video conferencing, video calls, or live streaming platforms, in addition to compositing foreground video and virtual background, it can also be compositing video frames from multiple sources. When multiple parties are involved, the backend server needs to composite multiple video streams into a single video for display on user devices. In one example, in a multi-party video conference, the backend server needs to composite multiple video streams into a single video for display on multiple user devices. If there are differences in image quality between the video frames from the multiple sources, the composite video will present a rather abrupt and discordant visual effect. In another example, in a multi-party live streaming scenario, the backend server needs to composite multiple live stream videos into a single video so that viewers entering the live stream can see the combined live stream of multiple hosts. Again, if there are differences in image quality between the video frames from the multiple live streams, the composite video will present a rather abrupt and discordant visual effect.

[0048] For questions regarding the relevant technologies, please refer to Figure 2AThis embodiment obtains at least two frames of images, which are divided into a first type of image and a second type of image according to their image quality. The image quality of the first type of image is higher than that of the second type of image. Then, the first type of image is compressed to reduce its image quality, resulting in a compressed first type of image. This compressed first type of image has an image quality close to that of the second type of image. The compressed first type of image in the at least two frames is then combined with the second type of image in the at least two frames to improve the harmony of the combined image, making the combined image natural and not abrupt.

[0049] The image synthesis method provided in the embodiments of this specification can be executed by an electronic device, including but not limited to a server, personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0050] For example, the electronic device may integrate a computer program product, which, when executed by the processor in the electronic device, is used to implement the image synthesis method provided in the embodiments described herein.

[0051] Please see Figure 2B , Figure 2B This is a schematic flowchart of an image synthesis method provided in an embodiment of this specification. The method can be executed by an electronic device and includes:

[0052] In step S101, at least two frames of images to be synthesized are obtained; the at least two frames of images are divided into a first type of image and a second type of image according to image quality, and the image quality of the first type of image is higher than that of the second type of image.

[0053] In step S102, the first type of image is compressed to obtain a compressed first type of image.

[0054] In step S103, the compressed first type image from the at least two frames of images is combined with the second type image from the at least two frames of images.

[0055] It is understood that this embodiment does not impose any restrictions on the source of the at least two frames of images to be synthesized, and the specific selection can be made according to the actual application scenario.

[0056] For example, the second type of image in at least two frames is an image that has undergone lossy compression; the first type of image in at least two frames has not undergone lossy compression, or the number of times lossy compression is performed on the first type of image is less than the number of times lossy compression is performed on the second type of image; thus, there is a difference in image quality between at least two frames.

[0057] For example, one of the at least two image frames is a foreground video frame sent by the client participating in the video conference, and the other frame is a preset background image; the electronic device needs to composite the foreground human image in the foreground video frame with the background image. Since the foreground video frame sent by the client needs to undergo lossy compression processing, resulting in the image quality of the foreground video frame being lower than that of the background image, it is necessary to first compress the background image in step S102 to reduce the image quality of the background image before compositing.

[0058] For example, the at least two image frames are video frames sent by at least two clients participating in the video conference. The electronic device needs to combine the video frames sent by the at least two clients into a single video frame and send it to each client participating in the video conference for easy display on each client, ensuring that all users participating in the video conference can see everyone's image. Since the network environments of the at least two clients are different, the lossy compression levels of the video frames sent by the at least two clients also differ, resulting in image quality differences between the at least two image frames. Therefore, step S102 is required to first compress the video frames with lower image quality before combining them.

[0059] For example, the at least two image frames are video frames sent by at least two live streaming clients participating in the live stream interaction. The electronic device needs to combine the video frames sent by the at least two live streaming clients into a single video frame and send it to all clients in the same live streaming room so that both the viewer clients and the streamer clients in that room can see the live stream of multiple streamers interacting. Since the network environments of the at least two streamer clients are different, the lossy compression levels of the video frames sent by the at least two streamer clients also differ, resulting in image quality differences between the at least two image frames. Therefore, step S102 is needed to first compress the video frames with lower image quality before combining them.

[0060] In one possible implementation, a compression parameter can be preset according to the actual application scenario, and the first type of image can be compressed using the preset compression parameter to reduce the image quality of the first type of image, so as to obtain the compressed first type of image. This makes the image quality of the compressed first type of image close to that of the second type of image, which is beneficial to improving the harmony of the synthesis result and making the synthesis result natural and not abrupt.

[0061] In one possible implementation, when the image quality differs but not significantly, and the human eye's ability to distinguish images already meets a certain image harmonization standard, a compression parameter that meets the image harmonization condition can be determined using the image quality of the second type of image as a reference. This image harmonization condition indicates that the difference between the image quality of the compressed first type of image and the image quality of the second type of image is no greater than a preset difference. This preset difference can be specifically set according to the actual application scenario. Then, the first type of image is compressed using this compression parameter, which reduces its image quality to a certain extent, making the image quality of the compressed first type of image closer to that of the second type of image. This helps to improve the harmony of the synthesis result, making the synthesis result natural and not abrupt.

[0062] In one possible implementation, for any of the at least two images of the first type, the first type image can be compressed with the goal of reducing its image quality to meet the image quality of the second type image, so that the image quality of the compressed first type image is basically the same as that of the second type image, thereby further improving the harmony of the synthesis result.

[0063] In some embodiments, the electronic device can determine the image quality corresponding to at least two frames of images respectively, and then classify the at least two frames of images into two types, a first type and a second type, according to the image quality. For example, the image with lower image quality is identified as the second type of image, and the image with higher image quality is identified as the first type of image. That is to say, the image quality of the first type of image is higher than that of the second type of image; in other words, the first type of image is the image that needs to have its image quality reduced, and the second type of image is the image that does not need to have its image quality reduced.

[0064] For example, if there are 3 frames of images to be synthesized, the frame with the lowest image quality can be designated as the second type of image, and the other two frames can be designated as the first type of image. For example, if there are 10 frames of images to be synthesized, the image quality can be sorted from highest to lowest, and the 3 lowest-quality frames can be designated as the second type of image, and the other seven frames as the first type of image. This embodiment does not impose any limitation on the number of divisions between the first and second types of images.

[0065] It is understandable that image quality reflects the richness of image information in a frame; higher image quality means richer image information, and vice versa. Image information includes, but is not limited to, texture information, color information, shape information, gradient information, and spatial relationship information, etc.

[0066] In some possible implementations, the image quality corresponding to the at least two frames can be determined based on the user's quality evaluation information of the image content, such as the user scoring the quality of each frame, with a lower score indicating lower image quality.

[0067] In some possible implementations, the image quality corresponding to the at least two frames can be determined based on the difference between the image and the reference image, with a larger difference resulting in lower image quality.

[0068] In some possible implementations, to reduce user operations and further improve the accuracy of image quality assessment, for any one of the at least two frames, the electronic device can process the image based on a pre-trained image quality assessment model to obtain the image quality. For example, the image quality assessment model is used to extract features from the input image and use the extracted features to make predictions, thereby outputting a prediction result. The prediction result includes a quality matrix of the input image, where the number of values ​​in the quality matrix is ​​the same as the number of pixels in the input image, and the two correspond one-to-one. The values ​​in the quality matrix indicate the degree of loss of the pixel value of the corresponding pixel in the input image.

[0069] In one example, a smaller value in the quality matrix indicates a greater degree of pixel value loss for the corresponding pixel in the input image; conversely, a larger value in the quality matrix indicates a smaller degree of pixel value loss for the corresponding pixel in the input image. Of course, it is also possible that a larger value in the quality matrix indicates a greater degree of pixel value loss for the corresponding pixel in the input image. This embodiment does not impose any limitations on this.

[0070] The image quality assessment model is obtained through supervised learning using a first training sample set. Each training sample in the first training sample set includes a distorted image and a quality matrix label. The distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image.

[0071] For example, algorithms such as SSIM (Structural Similarity), PSNR (Peak Signal-to-Noise Ratio), and / or LPIPS (Learned Perceptual Image Patch Similarity) can be used to measure the difference between the first training image and the distorted image to determine the quality matrix label.

[0072] Supervised learning is a machine learning task that infers functions from labeled training datasets. It uses samples with known characteristics as training sets to build a mathematical model (such as a discriminant model in pattern recognition, a weight model in artificial neural networks, etc.), and then uses the established model to predict unknown samples.

[0073] The training process of the image quality assessment model is illustrated here: The following process is executed iteratively for the first training sample set until a loop termination condition is met: The current image quality assessment model is used to obtain the predicted quality matrix of each frame of distorted images in the training sample set. If the loop termination condition is not met, the model parameters of the current image quality assessment model are adjusted based on the predicted quality matrix and the true quality matrix labels of each frame of distorted images to obtain an adjusted current image quality assessment model. This adjusted current image quality assessment model serves as the current image quality assessment model for the next iteration. It is understood that the loop termination condition includes, but is not limited to, reaching a preset number of iterations, or the difference between the predicted quality matrix and the true quality matrix labels of each frame of distorted images being less than a preset difference, but is not limited to these conditions.

[0074] For example, during training, the electronic device can calculate a loss function based on the predicted quality matrix and the true quality matrix labels of each frame of distorted image, and then adjust the model parameters of the current image quality assessment model in reverse based on the calculated loss value. The loss function includes, but is not limited to, the mean squared error loss function, the cross-entropy loss function, or the mean absolute error loss function, etc., and this embodiment does not impose any limitations on it.

[0075] It is understood that the training process of the image quality assessment model and the image synthesis process (steps S101 to S103) described above can be executed by the same electronic device; or, based on actual business requirements, they can be executed by different electronic devices. This embodiment does not impose any restrictions on this. For example, the training process of the image quality assessment model can be executed by a first electronic device, and the image synthesis process (steps S101 to S103) described above can be executed by a second electronic device. After the first electronic device completes the training of the image quality assessment model, it can port the image quality assessment model to the second electronic device.

[0076] For example, considering that the size of the input image may vary, to avoid the image scale change potentially affecting the final prediction result, a target size for the image input to the image quality assessment model can be preset. For instance, during the training phase described above, if the size of the distorted image is larger than the target size, the distorted image can be segmented to obtain at least two image blocks that conform to the target size, and these blocks can be input into the current image quality assessment model. Then, the prediction quality matrices corresponding to the at least two image blocks output by the current image quality assessment model are synthesized to obtain the prediction quality matrix of the distorted image. If the size of the distorted image is smaller than the target size, the edges of the distorted image can be padded to obtain an input image that conforms to the target size. Then, the prediction quality matrix belonging to the distorted image can be extracted from the prediction quality matrix corresponding to the input image output by the current image quality assessment model.

[0077] In one example, if the target size is 128*128, and the distorted image is 256*256, the distorted image can be divided into four 128*128 image blocks without overlap; if the distorted image is 64*64, a mirror filling operation can be performed on the distorted image, for example, by adding pixels with a pixel value of 0 to the image edges of the distorted image to obtain a 128*128 input image.

[0078] It is understood that the embodiments in this specification do not impose any restrictions on the specific structure of the image quality assessment model, and can be specifically set according to the actual application scenario. The structure of the image quality assessment model can be a convolutional neural network structure, a long short-term memory network (LSMT) structure, an autoencoder (transformer) structure, or a U-shaped convolutional neural network (U-net) model structure, etc.

[0079] For example, please refer to Figure 3Taking the U-shaped convolutional neural network (U-net) model with a target size of 128*128 as an example, the input image is first divided into 128*128 image blocks and fed into the U-net model. Then, the U-net model estimates the prediction quality matrix for each image block, and finally, the prediction quality matrices of each image block are combined to form the prediction quality matrix of the input image. The U-Net model is an encoder-decoder structure. The compressed channel is an encoder used to extract features from the input image layer by layer. It repeatedly uses a structure of two convolutional layers and one pooling layer, doubling the dimension of the feature map after each pooling operation. The extended channel is a decoder used to restore the positional information of the input image. It first performs a deconvolution operation to halve the dimension of the feature map, then concatenates the feature maps cropped from the compressed channel to form a new feature map twice the size. Two more convolutional layers are then used for feature extraction, and this structure is repeated. Each hidden layer of the U-Net model has a relatively high feature dimension, which is beneficial for the model to learn more diverse and comprehensive features. The "U-shaped" structure of the U-Net model makes the cropping and stitching process more intuitive and reasonable. The stitching of high-level feature maps and low-level feature maps, as well as the repeated and continuous operation of convolution, enable the model to obtain more accurate output feature maps by combining contextual and detailed information.

[0080] In some embodiments, the electronic device inputs the at least two frames of images to be synthesized into the image quality assessment model to obtain quality matrices corresponding to the at least two frames of images respectively.

[0081] For example, the electronic device can classify the at least two frames of images into two types, a first type and a second type, based on the quality matrices corresponding to the at least two frames, where the image quality of the first type of images is higher than that of the second type of images. Then, it can statistically analyze the quality matrix of the second type of images in the at least two frames to determine the quality feature values ​​of the second type of images. In one example, the number of values ​​in the image's quality matrix is ​​the same as the number of pixels in the image, and the two correspond one-to-one. The values ​​in the quality matrix indicate the degree of loss of the pixel value of the corresponding pixel in the image. The quality feature values ​​of the second type of images may include, but are not limited to, statistical values ​​of the values ​​in the quality matrix of the second type of images in the at least two frames. These statistical values ​​include, but are not limited to, the average, median, maximum, weighted sum of the median and maximum, or any value between the median and maximum. This embodiment does not impose any limitations on this.

[0082] In some exemplary embodiments, one way to reduce image quality is to perform lossy compression on the image. After obtaining the quality feature values ​​of the second type of image, the electronic device performs compression processing on any first type of image in the at least two frames, aiming to reduce the image quality to meet the quality feature values, so that the image quality of the compressed first type of image will further approach the image quality of the second type of image. For example, for any first type of image in the at least two frames, the electronic device can determine the compression parameters corresponding to the first type of image based on the first type of image and the quality feature values, and perform compression processing on the first type of image using a preset compression algorithm and the compression parameters.

[0083] In one example, considering that the step in the image compression process that typically results in the loss of image information is the quantization step, while other compression steps generally do not cause the loss of image information, the compression parameters may include quantization parameters (QP) or quantization matrices. For example, the image information may indicate pixel values, gradient information, and / or texture information in the image, etc., and this embodiment does not impose any limitations on this.

[0084] In this case, considering that under the same compression algorithm and parameters, for any two frames of the first type of image, if one frame of the first type of image has a higher image quality, reflecting more image information of the first type of image, then more image information will be lost after compression using the same compression algorithm and parameters. That is to say, under the same objective of reducing image quality to meet the aforementioned quality characteristic values, if the image quality of any two frames of the first type of image is different, then the compression parameters for these two frames of the first type of image will be different.

[0085] Therefore, for any first-type image in the at least two frames, suitable compression parameters can be determined based on the image quality of the first-type image and the quality feature value of the second-type image, thereby further improving the harmony of the final synthesis result. The degree of image quality degradation indicated by the compression parameters is positively correlated with the quality feature value of the second-type image; that is, the larger the quality feature value, the greater the loss of image information, and the greater the degree of image quality degradation indicated by the final determined compression parameters, and vice versa. The degree of image quality degradation indicated by the compression parameters is negatively correlated with the image quality of the first-type image; that is, the higher the image quality of the first-type image, the richer the image information in the first-type image, and the lower the degree of image quality degradation indicated by the compression parameters of the first-type image, and vice versa.

[0086] In some possible implementations, a mapping relationship can be predetermined with image quality and quality feature values ​​as independent variables and compression parameters as dependent variables. Then, in the actual process of determining the compression parameters of the first type of image, the compression parameters of the first type of image can be determined based on the image quality of the first type of image, the quality feature values, and the mapping relationship.

[0087] In other possible implementations, a compression parameter estimation model can be pre-trained to learn the relationship between image quality and quality feature values ​​and compression parameters, respectively. Then, for any first-type image in the at least two frames, the electronic device can use the pre-trained compression parameter estimation model to process the first-type image and the quality feature values ​​to obtain the compression parameters for the first-type image.

[0088] The compression parameter estimation model is obtained through supervised learning using a second training sample set. Each training sample in the second training sample set includes a second training image, compression parameter labels, and quality feature values ​​determined based on the second training image and the compression parameter labels. When determining the quality feature values ​​for any training sample, the electronic device can use a preset compression algorithm and the compression parameter labels in that training sample to compress the second training image in that training sample. Then, based on the compression result of the second training image and the difference between the two images, a quality matrix is ​​determined, and the values ​​in the quality matrix are statistically processed to obtain the quality feature values.

[0089] The training process of the compression parameter estimation model is illustrated here: The following process is executed iteratively for the second training sample set until a loop termination condition is met: The current compression parameter estimation model is used to obtain the compression parameter prediction results required to reduce the image quality of each second training image in the training sample set to meet the quality feature values. If the loop termination condition is not met, the model parameters of the current compression parameter estimation model are adjusted according to the required compression parameter prediction results and the true compression parameter labels for each frame of the second training image, to obtain an adjusted current compression parameter estimation model. This adjusted current compression parameter estimation model serves as the current compression parameter estimation model for the next loop. It is understood that the loop termination condition includes, but is not limited to, reaching a preset number of loops, or the difference between the required compression parameter prediction results and the true compression parameter labels for each frame of the second training image being less than a preset difference.

[0090] For example, the electronic device can calculate a loss function based on the compression parameter prediction results required to reduce the image quality of each frame of the second training image to meet the quality feature value and the preset compression parameters, and then adjust the model parameters of the current compression parameter estimation model in reverse according to the calculated loss value. The loss function includes, but is not limited to, the mean squared error loss function, the cross-entropy loss function, or the mean absolute error loss function, etc., and this embodiment does not impose any limitations on it.

[0091] It is understood that the first training image in the training process of the compression parameter estimation model and the second training image in the training process of the image quality assessment model can be different, or partially or completely identical, and this embodiment does not impose any restrictions on this. The training process of the compression parameter estimation model and the above-mentioned image synthesis process (steps S101 to S103) can be executed by the same electronic device; or, based on the actual business division requirements, they can be executed by different electronic devices, and this embodiment does not impose any restrictions on this.

[0092] It is understood that the embodiments in this specification do not impose any restrictions on the specific structure of the model with the preset structure, and can be specifically set according to the actual application scenario. The structure of the compressed parameter estimation model can be a convolutional neural network, a long short-term memory network (LSMT), or an autoencoder (transformer) model, etc.

[0093] For example, please refer to Figure 4 The compression parameter estimation model may include a residual network 10, a pooling layer 20, a stitching layer 30, and at least one fully connected layer 40. The residual network 10 is used to extract image features from a first type of image; exemplarily, the residual network 10 includes an additive layer and at least one convolutional layer. The convolutional layer is used to extract features from the first type of image, and the additive layer is used to add the features output by the convolutional layer to the first type of image to output the image features. The pooling layer 20 is used to perform dimensionality reduction processing on the image features output by the residual network. The stitching layer 30 is used to stitch the quality feature values ​​and the dimensionality-reduced image features output by the pooling layer 20 to obtain stitched features. The fully connected layer 40 is used to predict the compression parameters of the first type of image based on the stitched features output by the stitching layer 30.

[0094] In some embodiments, for any first-type image in the at least two frames, after determining the compression parameters required to reduce the image quality of the first-type image to the quality feature value, the electronic device can compress the first-type image using the compression parameters. After compressing all first-type images in the at least two frames, the electronic device can synthesize the compressed results of the first-type images in the at least two frames with the second-type images in the at least two frames, which helps to improve the harmony of the synthesized result and makes the synthesized result natural and not abrupt.

[0095] In one exemplary embodiment, the image synthesis method provided in this specification can be applied to video conferencing scenarios. Please refer to [link / reference]. Figure 5A , Figure 5A A schematic diagram of a video conferencing system is shown. The system includes a client and a server. The client is an application capable of online conferencing and can be installed on different users' devices, including but not limited to mobile phones, computers, tablets, watches, or wristbands. For example, users can log in to the client using a unique personal account. The server can be a physical server or a cloud server; this embodiment does not impose any limitations. The server provides background services for the client, such as background services for video conferencing.

[0096] In one example, when users A, B, and C participate in the same video conference through clients installed on their own devices, the server is responsible for processing and forwarding the relevant content of the video conference. For instance, if user A enables the virtual background replacement function, the server needs to composite the foreground object (such as user A's image) in the foreground video frame uploaded by user A's client with the virtual background, and then return the composite video to user A's client, user B's client, and user C's client. Alternatively, to allow each client to simultaneously display the images of users A, B, and C, the server needs to composite the three video streams uploaded by user A's client, user B's client, and user C's client into one video stream and distribute it to user A's client, user B's client, and user C's client.

[0097] Please see Figure 5B To address the need for users to change the background during video conferencing, this specification provides a video conferencing method, which can be executed by a server, and includes:

[0098] In step S201, a foreground video frame sent by a client participating in the video conference is received, and a background image synthesized with the foreground video frame is obtained; the image quality of the foreground video frame is lower than that of the background image.

[0099] In step S202, the background image is compressed to obtain a compressed background image.

[0100] In step S203, the compressed background image is combined with the foreground video frame to obtain a composite video frame sent to the client; the composite video frame is used to display on the client.

[0101] In this embodiment, in scenarios such as video conferencing, video calls, or live streaming platforms, when users need to replace the background, considering that the image quality of the foreground video frame is reduced after one or more compressions, while the background image is compressed and has a higher image quality, the background image can be compressed to improve the harmony of the two in the synthesis. This allows the image quality of the compressed background image to be close to that of the foreground video frame. Using the compressed background image and the foreground video frame for synthesis helps to improve the harmony of the synthesis result, making the synthesis result natural and not abrupt, thus improving the display effect.

[0102] For example, a compression parameter can be preset according to the actual application scenario, and the background image can be compressed using the compression parameter.

[0103] For example, a compression parameter that satisfies the image harmonization condition can be determined based on the image quality of the foreground video frame, and the background image can be compressed using the compression parameter; the image harmonization condition indicates that the difference between the image quality of the compressed background image and the image quality of the foreground video frame is not greater than a preset difference.

[0104] For example, the background image can be compressed with the goal of reducing the image quality to meet the image quality of the foreground video frame, so that the image quality of the compressed background image is substantially the same as that of the foreground video frame.

[0105] In one example, after applying the image synthesis method mentioned in the embodiments of this specification, please refer to... Figure 5C A portrait matting method is used to process each frame of the foreground video to obtain a portrait blending texture for each frame. This texture is then used to extract the portrait from the input foreground video. Simultaneously, the position of the portrait within a virtual background is estimated to obtain location information, such as if the user has selected a location within the virtual background. For each frame of the foreground video, the video frame I... in Input a pre-trained image quality assessment model to obtain the quality matrix I D Then, statistical processing is performed on the values ​​in the quality matrix to obtain the quality feature value D of the video frame. I Next, the quality feature values ​​of the video frame and the virtual background I are... bgThe compression parameters QP are obtained by inputting the pre-trained compression parameter estimation model, and then used to compress the virtual background. It is understood that if the compression result of the virtual background differs in format from the video frame of the foreground video, the compression result of the virtual background needs to be format-converted to ensure that the formats of both components are identical.

[0106] Then, for any video frame in the foreground video, the compressed result of that video frame and the virtual background are synthesized based on the corresponding human portrait blending texture and position information to output a composite video. Please refer to [link to relevant documentation]. Figure 6A and Figure 6B , Figure 6A It shows Figure 1 The resulting composite image, obtained through the processing flow, has a blurred human figure and a relatively clear background, resulting in a jarring and disharmonious combination. Figure 6B It shows Figure 5C The resulting composite image, due to the lossy compression of the virtual background, has a more similar appearance between the human figure and the background, resulting in a more harmonious composite image.

[0107] For details on the specific implementation of this embodiment, please refer to the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0108] Please see Figure 7 For scenarios involving the synthesis of multiple video streams in video conferencing, embodiments of this specification provide another video conferencing method. This method can be executed by a server and includes:

[0109] In step S301, video frames sent by at least two clients participating in the video conference are received; the at least two video frames are divided into a first type of video frame and a second type of video frame according to their image quality, and the video frame quality of the first type of video frame is higher than that of the second type of video frame.

[0110] In step S302, for any first type of video frame among the at least two video frames, compression processing is performed on the first type of video frame with the goal of reducing the image quality to meet the image quality of the second type of video frame.

[0111] In step S303, the compression result of the first type of video frame in the at least two video frames is combined with the second type of video frame in the at least two video frames to generate a composite video frame that is sent to each of the clients; the composite video frame is used to be displayed in each of the clients.

[0112] In this embodiment, considering the differences in image quality between video frames in a video conference scenario, the server, after receiving video frames from at least two clients participating in the video conference, categorizes them into a first type of video frame and a second type based on image quality. The first type of video frames are then compressed to ensure their image quality is close to that of the foreground video frames. Combining these compressed first type of video frames with the second type of video frames improves the harmony of the composite result, making it natural and less jarring, thus enhancing the presentation.

[0113] For example, a compression parameter can be preset according to the actual application scenario, and the compression parameter can be used to compress the first type of video frames.

[0114] For example, a compression parameter that satisfies the image harmonization condition can be determined based on the image quality of the second type of video frame, and the first type of video frame can be compressed using the compression parameter; the image harmonization condition indicates that the difference between the image quality of the compressed first type of video frame and the image quality of the second type of video frame is not greater than a preset difference.

[0115] For example, the first type of video frames can be compressed with the goal of reducing the image quality to meet the image quality of the second type of video frames, so that the image quality of the compressed first type of video frames is substantially the same as that of the second type of video frames.

[0116] For details on the specific implementation of this embodiment, please refer to the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0117] In some embodiments, considering that image processing for reducing high-quality images is not limited to compression, other processing methods can also be used, such as blurring the high-quality image using a preset blurring algorithm. Therefore, please refer to [link to relevant documentation]. Figure 8 This specification also provides an image synthesis method, including:

[0118] In step S401, at least two frames of images to be synthesized are obtained; the at least two frames of images are divided into a first type of image and a second type of image according to image quality, and the image quality of the first type of image is higher than that of the second type of image.

[0119] In step S402, processing parameters for reducing the image quality of the first type of images are determined, and the first type of images are processed using the processing parameters.

[0120] In step S403, the processing result of the first type of image in the at least two images is combined with the second type of image in the at least two images.

[0121] In this embodiment, processing parameters for reducing the image quality of the first type of image are determined, and the first type of image is processed using the processing parameters so that the image quality of the processed first type of image can be close to that of the second type of image. The processing result of the first type of image in the at least two images is combined with the second type of image in the at least two images to improve the harmony of the combined image, making the combined image natural and not abrupt.

[0122] In some embodiments, the processing parameters include blur parameters, and processing the first type of image using the processing parameters includes: blurring the first type of image using the blur parameters; for example, blur algorithms include, but are not limited to, Gaussian Blur, Box Blur, Dual Blur, Bokeh Blur, Tilt Shift Blur, Iris Blur, Grainy Blur, Radial Blur, or Directional Blur, etc.

[0123] In some embodiments, the processing parameters include compression parameters, and processing the first type of image using the processing parameters includes: compressing the first type of image using the compression parameters. For example, lossy compression algorithms include, but are not limited to, JPEG (Joint Photographic Experts Group) compression algorithm, H.261 compression algorithm, and / or MPEG (Moving Picture Experts Group) compression algorithm, etc.

[0124] In some embodiments, the method further includes: statistically analyzing the image quality of a second type of image among the at least two images to determine a quality feature value for the second type of image. The processing parameters are those required to reduce the image quality of the first type of image to meet the quality feature value.

[0125] In some embodiments, the image quality of an image is represented by a quality matrix, wherein the values ​​in the quality matrix are used to indicate the degree of loss of pixel values ​​of corresponding pixels in the image; the quality feature values ​​of the second type of image include: statistical values ​​of the values ​​in the quality matrix of the second type of image among the at least two images, wherein the statistical values ​​include at least one of the mean, median and maximum value.

[0126] In some embodiments, the method further includes: for any one of the at least two frames of images, processing the image based on a pre-trained image quality assessment model to obtain the image quality of the image. The image quality assessment model is obtained through supervised learning using a first training sample set; each training sample in the first training sample set includes a distorted image and a quality matrix label; the distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image.

[0127] In some embodiments, considering the different characteristics of images in different fields, targeted processing algorithms may be used. The required processing algorithm can be determined based on the field to which the first type of image belongs; for example, Gaussian blur might be used for images in the medical field, while radial blur might be used for images in the architectural field. In one example, a mapping relationship between fields and processing algorithms can be pre-stored. During image synthesis, the required processing algorithm is determined based on the field to which the first type of image belongs and this pre-stored mapping relationship.

[0128] The processing parameters are determined based on the image quality of the first type of image and the quality feature value. For example, the degree of image quality degradation indicated by the processing parameters is negatively correlated with the image quality of the first type of image and positively correlated with the quality feature value. For example, the processing parameters corresponding to the first type of image are obtained by inputting the first type of image and the quality feature value into a pre-trained processing parameter estimation model. The processing parameter estimation model is obtained through supervised learning using a preset training sample set. Each training sample in the preset training sample set includes a training image, a processing parameter label, and a quality feature value determined based on the training image and the processing parameter label.

[0129] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. Therefore, the arbitrary combination of the various technical features in the above embodiments is also within the scope of this specification.

[0130] Figure 9 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 9At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 503, and non-volatile memory 510, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into memory 503 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0131] Please refer to Figure 10 Image synthesis devices can be applied to, for example Figure 9 The device shown is used to implement the technical solution of this specification. The image synthesis apparatus may include:

[0132] The image acquisition module 601 is used to acquire at least two frames of images to be synthesized; the at least two frames of images are divided into a first type of image and a second type of image according to image quality, and the image quality of the first type of image is higher than that of the second type of image;

[0133] Compression module 602 compresses the first type of image to obtain a compressed first type of image;

[0134] The compositing module 603 is used to compose a first type of image from the at least two frames of images with a second type of image from the at least two frames of images.

[0135] In some embodiments, the system further includes: an image quality statistics module, configured to statistically analyze the image quality of the second type of images among the at least two images, and determine the quality characteristic values ​​of the second type of images. The compression module 602 is specifically configured to: compress the first type of images with the goal of reducing the image quality to meet the quality characteristic values.

[0136] In some embodiments, the image quality of an image is represented by a quality matrix, wherein the values ​​in the quality matrix are used to indicate the degree of loss of pixel values ​​of corresponding pixels in the image; the quality feature values ​​of the second type of image include: statistical values ​​of the values ​​in the quality matrix of the second type of image among the at least two images, wherein the statistical values ​​include at least one of the mean, median and maximum value.

[0137] In some embodiments, the system further includes: an image quality acquisition module, configured to process any image in the at least two frames based on a pre-trained image quality assessment model to obtain the image quality of the image; wherein the image quality assessment model is obtained through supervised learning using a first training sample set; each training sample in the first training sample set includes a distorted image and a quality matrix label; the distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image.

[0138] In some embodiments, the compression module is specifically used to determine the compression parameters corresponding to any first type image among the at least two images, based on the first type image and the quality feature value, and to compress the first type image using a preset compression algorithm and the compression parameters; wherein the degree of image quality degradation indicated by the compression parameters is negatively correlated with the image quality of the first type image and positively correlated with the quality feature value.

[0139] In some embodiments, the compression module is specifically used to process the first type of image and the quality feature value based on a pre-trained compression parameter estimation model to obtain the compression parameters corresponding to the first type of image; wherein, the compression parameter estimation model is obtained by supervised learning using a second training sample set; each training sample in the second training sample set includes a second training image, a compression parameter label, and a quality feature value determined based on the second training image and the compression parameter label.

[0140] In some embodiments, the at least two frames include at least one of the following: one frame of the at least two frames is a foreground video frame sent by a client participating in a video conference, and the other frame is a preset background image; the at least two frames are video frames sent by at least two clients participating in a video conference respectively; the at least two frames are video frames sent by at least two host clients participating in a live broadcast.

[0141] Accordingly, embodiments of this specification also provide another image synthesis apparatus, including:

[0142] An image acquisition module is used to acquire at least two frames of images to be synthesized; the at least two frames of images are divided into a first type of image and a second type of image according to image quality, and the image quality of the first type of image is higher than that of the second type of image;

[0143] A processing module is configured to determine processing parameters for reducing the image quality of the first type of images, and to process the first type of images using the processing parameters;

[0144] A synthesis module is used to synthesize a first type of image from the at least two images with a second type of image from the at least two images.

[0145] In some embodiments, the system further includes: an image quality statistics module, configured to statistically analyze the image quality of a second type of image among the at least two images, and determine the quality characteristic value of the second type of image. The processing parameters are those required to reduce the image quality of the first type of image to meet the quality characteristic value.

[0146] In some embodiments, the processing parameters include blurring parameters, and the processing module is specifically used to blur the first type of image using the blurring parameters. And / or, the processing parameters include compression parameters, and the processing module is specifically used to compress the first type of image using the compression parameters.

[0147] Accordingly, embodiments of this specification also provide a video conferencing device, including:

[0148] The image acquisition module is used to receive a foreground video frame sent by a client participating in a video conference, and to acquire a background image synthesized with the foreground video frame; the image quality of the foreground video frame is lower than that of the background image.

[0149] A compression module is used to compress the background image to obtain a compressed background image;

[0150] The compositing module is used to combine the compressed background image with the foreground video frame to obtain a composite video frame sent to the client; the composite video frame is used to display on the client.

[0151] Accordingly, embodiments of this specification also provide another video conferencing device, including:

[0152] The image acquisition module is used to receive video frames sent by at least two clients participating in the video conference; the at least two video frames are divided into a first type of video frame and a second type of video frame according to their image quality, and the video frame quality of the first type of video frame is higher than that of the second type of video frame.

[0153] The compression module is used to compress the first type of video frames to obtain compressed first type of video frames;

[0154] A compositing module is used to combine compressed first-type video frames from the at least two video frames with second-type video frames from the at least two video frames to generate a composite video frame that is sent to each of the clients; the composite video frame is used to be displayed on each of the clients.

[0155] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0156] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0157] Accordingly, embodiments of this specification also provide an electronic device, including:

[0158] processor;

[0159] Memory used to store processor-executable instructions;

[0160] The processor implements the method as described in any of the above by running the executable instructions.

[0161] The processor includes, but is not limited to, a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0162] For example, the memory may include at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc.

[0163] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of any of the methods described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0164] Accordingly, embodiments of this specification also provide a computer program product, including a computer program that, when executed by a processor, is used to implement the above-described method.

[0165] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0166] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0168] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0169] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0170] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0171] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0172] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0173] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. An image synthesis method, comprising: Obtain at least two frames of images to be synthesized; For any one of the at least two frames, the image is processed based on a pre-trained image quality assessment model to obtain the image quality. The image quality is represented by a quality matrix, where the number of values ​​in the quality matrix is ​​the same as the number of pixels in the image input to the image quality assessment model, and the two correspond one-to-one. The values ​​in the quality matrix indicate the degree of loss of the corresponding pixel value in the image input to the image quality assessment model. The at least two frames are divided into a first category of images and a second category of images based on their image quality, with the first category of images having higher image quality than the second category of images. The image quality assessment model is obtained through supervised learning using a first training sample set. Each training sample in the first training sample set includes a distorted image and a quality matrix label. The distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image. The first type of image is compressed to obtain the compressed first type of image; The compressed first type of image from the at least two frames is combined with the second type of image from the at least two frames.

2. The method according to claim 1, further comprising: The image quality of the second type of images among the at least two images is statistically analyzed to determine the quality feature value of the second type of images; The compression process for the first type of image includes: The first type of image is compressed with the goal of reducing the image quality to meet the quality characteristic value.

3. The method according to claim 2, wherein the quality feature values ​​of the second type of image include: The statistical values ​​of the values ​​in the quality matrix of the second type of images in the at least two images, the statistical values ​​including at least one of the mean, median and maximum.

4. The method according to claim 2 or 3, wherein the compression processing of the first type of image with the goal of reducing the image quality to meet the quality characteristic value includes: For any of the first type images among the at least two images, the compression parameters corresponding to the first type image are determined based on the first type image and the quality feature value, and the first type image is compressed using a preset compression algorithm and the compression parameters; The degree of image quality degradation indicated by the compression parameters is negatively correlated with the image quality of the first type of image and positively correlated with the quality feature value.

5. The method according to claim 4, wherein determining the compression parameters corresponding to the first type of image based on the first type of image and the quality feature value includes: The compression parameters corresponding to the first type of image are obtained by processing the first type of image and the quality feature value based on the pre-trained compression parameter estimation model. The compression parameter estimation model is obtained through supervised learning using a second training sample set. Each training sample in the second training sample set includes a second training image, a compression parameter label, and a quality feature value determined based on the second training image and the compression parameter label.

6. The method according to claim 1, wherein the at least two frames comprise at least one of the following: One of the at least two frames is a foreground video frame sent by the client participating in the video conference, and the other frame is a preset background image; The at least two frames are video frames sent by at least two clients participating in the video conference. The at least two frames are video frames sent by at least two host clients participating in the live broadcast.

7. An image synthesis method, comprising: Obtain at least two frames of images to be synthesized; For any one of the at least two frames, the image is processed based on a pre-trained image quality assessment model to obtain the image quality. The image quality is represented by a quality matrix, where the number of values ​​in the quality matrix is ​​the same as the number of pixels in the image input to the image quality assessment model, and the two correspond one-to-one. The values ​​in the quality matrix indicate the degree of loss of the corresponding pixel value in the image input to the image quality assessment model. The at least two frames are divided into a first category of images and a second category of images based on their image quality, with the first category of images having higher image quality than the second category of images. The image quality assessment model is obtained through supervised learning using a first training sample set. Each training sample in the first training sample set includes a distorted image and a quality matrix label. The distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image. Determine processing parameters for reducing the image quality of the first type of images, and process the first type of images using the processing parameters; The processing result of the first type of image in the at least two images is combined with the second type of image in the at least two images to form a composite.

8. The method according to claim 7, further comprising: The image quality of the second type of images among the at least two images is statistically analyzed to determine the quality feature value of the second type of images; The processing parameters are the processing parameters required to reduce the image quality of the first type of image to meet the quality characteristic value.

9. The method according to claim 7, wherein the processing parameters include blur parameters, and the processing of the first type of image using the processing parameters includes: The first type of image is blurred using the blurring parameters; And / or, The processing parameters include compression parameters, and the processing of the first type of image using the processing parameters includes: The first type of image is compressed using the compression parameters.

10. A video conferencing method, comprising: Receive foreground video frames sent by clients participating in a video conference, and acquire a background image synthesized with the foreground video frames; The image quality of the foreground video frame is lower than that of the background image. The image quality of both the foreground and background images can be evaluated using a pre-trained image quality assessment model. The image quality is represented by a quality matrix, where the number of values ​​in the quality matrix is ​​the same as the number of pixels in the image input to the image quality assessment model, and the two correspond one-to-one. The values ​​in the quality matrix indicate the degree of loss of the corresponding pixel value in the image input to the image quality assessment model. The image quality assessment model is obtained through supervised learning using a first training sample set. Each training sample in the first training sample set includes a distorted image and a quality matrix label. The distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image. The background image is compressed to obtain a compressed background image; The compressed background image is combined with the foreground video frame to obtain a composite video frame that is sent to the client; the composite video frame is used to display on the client.

11. A video conferencing method, comprising: Receive video frames sent by at least two clients participating in the video conference; For any one of at least two video frames, the video frame is processed based on a pre-trained image quality assessment model to obtain the image quality of the video frame. The image quality of the video frame is represented by a quality matrix, and the number of values ​​in the quality matrix is ​​the same as the number of pixels in the video frame input to the image quality assessment model, and the two correspond one-to-one. The values ​​in the quality matrix are used to indicate the degree of loss of the pixel value of the corresponding pixel in the video frame input to the image quality assessment model. At least two video frames are divided into a first category of video frames and a second category of video frames according to their image quality. The video frame quality of the first category of video frames is higher than that of the second category of video frames. The image quality assessment model is obtained by supervised learning using a first training sample set. Each training sample in the first training sample set includes a distorted image and a quality matrix label. The distorted image is obtained by reducing the image quality of the first training image, and the quality matrix label of the distorted image is determined based on the difference between the first training image and the distorted image. The first type of video frames are compressed to obtain compressed first type of video frames; The compressed first type of video frame from the at least two video frames is combined with the second type of video frame from the at least two video frames to generate a composite video frame that is sent to each of the clients; the composite video frame is used to display on each of the clients.

12. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 11 by executing the executable instructions.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 11.

Images