Image enhancement method and device in video conference

Abstract

The embodiment of the invention provides a method and a device for enhancing an image in a video conference. The method comprises the following steps: acquiring an image frame in the video conference; determining a target alternative image from a computing power server according to the image frame; and carrying out image enhancement on the image frame based on the target alternative image. Through the embodiment of the invention, the high-quality alternative image is adopted to perform image enhancement on the low-quality image frame in the video conference, so that the quality of the image in the video conference is improved, the poor image effect caused by directly using a digital person to replace a real portrait image is avoided, and the video conference experience of participants is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method and device for image enhancement in video conferencing. Background Art

[0002] With the rapid development of global informatization, video conferencing has become an important way for people to work and communicate in their daily lives. However, in some scenarios, such as low-bandwidth environments, the stability and quality of video conferencing are often limited, especially when the picture clarity is low.

[0003] In the prior art, in order to improve the image quality in video conferences, digital human images are usually used to replace the original low-quality images in video conferences. However, the use of digital humans is separated from the images of the original participants, while people are usually accustomed to communicating with real people. In addition, the facial expressions and body movements of digital humans often appear stiff and unnatural, affecting the video conferencing experience of participants. Summary of the invention

[0004] In view of the above problems, a method and device for image enhancement in a video conference is proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for image enhancement in a video conference, the method comprising:

[0006] Get the image frame in the video conference;

[0007] Determining a target candidate image from a computing power server according to the image frame;

[0008] Based on the target candidate image, image enhancement is performed on the image frame.

[0009] Optionally, determining a target candidate image from a computing power server according to the image frame includes:

[0010] Determine from the computing power server the candidate image that matches the current image frame as the target candidate image, or determine the candidate image that matches the previous image frame as the target candidate image;

[0011] The step of performing image enhancement on the image frame based on the target candidate image includes:

[0012] Based on the target candidate image, the current image frame is image enhanced, or the previous image frame is image enhanced and the image-enhanced previous image frame replaces the current image frame.

[0013] Optionally, the determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, includes:

[0014] Comparing image features of a current image frame with a previous image frame, and determining, from a computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, according to the image feature comparison result;

[0015] Alternatively, a clarity comparison is performed between the current image frame and the previous image frame, and based on the clarity comparison result, an alternative image matching the current image frame is determined from the computing power server as the target alternative image, or an alternative image matching the previous image frame is determined as the target alternative image.

[0016] Optionally, determining, from the computing power server, a candidate image matching the current image frame includes:

[0017] Extracting current image features from the current image frame, and using the current image features to determine a candidate image matching the current image frame from a computing power server;

[0018] The determining of the candidate image matching the preceding image frame comprises:

[0019] A preceding image feature is extracted from the preceding image frame, and the preceding image feature is used to determine an alternative image matching the preceding image frame from a computing power server.

[0020] Optionally, the performing image enhancement on the current image frame based on the target candidate image includes:

[0021] Determine high-quality image features from the target candidate image, and integrate the high-quality image features into the current image frame to perform image enhancement on the current image frame;

[0022] The performing image enhancement on the preceding image frame comprises:

[0023] A high-quality image feature is determined from the target candidate image, and the high-quality image feature is integrated into the preceding image frame to perform image enhancement on the preceding image frame.

[0024] Optionally, the determining, from the computing power server according to the clarity comparison result, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, comprises: when the clarity of the current image frame is less than or equal to the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the candidate image matching the previous image frame as the target candidate image;

[0025] The method of performing image enhancement on the current image frame based on the target alternative image, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: performing image enhancement on the previous image frame based on the target alternative image and replacing the current image frame with the image enhanced previous image frame.

[0026] Optionally, the determining, from the computing power server according to the clarity comparison result, the candidate image matching the current image frame as the target candidate image, or the candidate image matching the previous image frame as the target candidate image, comprises: when the clarity of the current image frame is greater than the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the candidate image matching the current image frame as the target candidate image;

[0027] The performing image enhancement on the current image frame based on the target candidate image, or performing image enhancement on the previous image frame and replacing the current image frame with the image-enhanced previous image frame, includes: performing image enhancement on the current image frame based on the target candidate image.

[0028] Optionally, the determining, from the computing power server according to the clarity comparison result, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, comprises: when a clarity difference between the current image frame and the previous image frame is less than a preset difference, selecting the target candidate image from the candidate image matching the current image frame and the candidate image matching the previous image frame;

[0029] The method of performing image enhancement processing on the current image frame based on the target alternative image, or performing image enhancement processing on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: randomly selecting to perform image enhancement on the current image frame based on the target alternative image, or performing image enhancement processing on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

[0030] Optionally, selecting a target candidate image from candidate images matching the current image frame and candidate images matching the previous image frame includes:

[0031] Determining a first similarity between the current image frame and the matching candidate image, and determining a second similarity between the previous image frame and the matching candidate image;

[0032] According to the first similarity and the second similarity, a target candidate image is selected from the candidate images matching the current image frame and the candidate images matching the previous image frame.

[0033] A device for image enhancement in a video conference, the device comprising:

[0034] An image frame acquisition module is used to acquire image frames in a video conference;

[0035] A target candidate image determination module, used to determine a target candidate image from a computing power server according to the image frame;

[0036] An image enhancement module is used to perform image enhancement on the image frame based on the target candidate image.

[0037] The embodiments of the present invention have the following advantages:

[0038] In an embodiment of the present invention, by acquiring image frames in a video conference, determining target candidate images from a computing server based on the image frames, and then enhancing the image frames based on the target candidate images, it is possible to use high-quality alternative images to enhance low-quality image frames in the video conference, thereby improving the quality of the images in the video conference and avoiding the poor picture effects caused by directly using digital humans to replace real portrait images, thereby improving the video conferencing experience of the participants. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0040] Figure 1 is a flowchart of the steps of a method for image enhancement in a video conference provided by some embodiments of the present invention;

[0041] Figure 2 is a schematic diagram of a system architecture provided by some embodiments of the present invention;

[0042] Figure 3is a flowchart of the steps of another method for image enhancement in a video conference provided by some embodiments of the present invention;

[0043] Figure 4 is a flowchart of the steps of another method for image enhancement in a video conference provided by some embodiments of the present invention;

[0044] Figure 5 It is a structural block diagram of a device for image enhancement in a video conference provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Reference Figure 1 , shows a flowchart of the steps of a method for image enhancement in a video conference provided by some embodiments of the present invention, the method can be applied to a server, such as Figure 2 Medium computing power server.

[0047] Specifically, the following steps may be included:

[0048] Step 101, obtaining image frames in a video conference.

[0049] As an example, the video conference may be a video conference based on visual networking, that is, the clients and servers participating in the conference communicate with each other using the visual networking protocol.

[0050] In actual applications, by deploying clients on devices such as mobile terminals, you can join a video conference through the client and conduct a video conference with other clients.

[0051] During a video conference, participants can be photographed through cameras on mobile terminals and other devices to obtain video streams, and the client can send the captured video streams to the server.

[0052] Step 102: Determine a target candidate image from a computing power server according to the image frame.

[0053] In practical applications, multiple candidate images may be stored in advance in the computing server, and the candidate images may be high-quality images, such as high-definition images. In some examples, an image database may be set in the computing server, and the candidate images in the image database are high-quality digital human images.

[0054] During a video conference, the image frames in the video stream may have picture quality problems, and low-quality image frames need to be enhanced. Picture quality problems include: blurred images (such as motion blur, lens out of focus), low picture resolution, video freeze / low frame rate, and other picture quality problems (such as insufficient color brightness, insufficient contrast, noise, over-compression, rain, snow, fog, excessive brightness or darkness, occlusion damage (such as logos, watermarks, subtitles, occlusions), color distortion (such as color cast, color loss), and black and white photos.

[0055] When it is necessary to perform image enhancement on low-quality image frames in a video conference, alternative image frames with image features similar to those of the low-quality image frames can be selected from multiple alternative images on the computing power server as target alternative images, and then high-quality target alternative images can be used to enhance the low-quality image frames frame by frame, thereby improving the picture quality of the video conference.

[0056] In some embodiments of the present invention, determining the target candidate image from the computing power server based on the image frame includes: determining from the computing power server an alternative image that matches the current image frame as the target candidate image, or determining an alternative image that matches the previous image frame as the target candidate image.

[0057] In some examples, the previous image frame is an image frame that is sequenced before the current image frame in the video stream.

[0058] In practical applications, for each image frame in a video conference, an alternative image matching the image frame can be found among multiple alternative images on the computing power server.

[0059] Since the image frames in video conferencing are constantly updated with the video stream, for the current moment, it corresponds to the current image frame in the video stream, but the alternative image matching the current image frame may not be the best option. In this case, a matching alternative image can be selected from the alternative images matching the current image frame and the alternative images matching the previous image frame as the target alternative image for subsequent optimization.

[0060] In some embodiments of the present invention, before determining the target candidate image from the computing power server according to the image frame, the method further includes:

[0061] The clarity of the current image frame is detected, and when the clarity of the current image frame is less than a preset value, a target candidate image is determined from a computing power server based on the image frame.

[0062] In practical applications, the image frames in the video stream can be detected. In the process of detecting the image frames in the video stream, when it is detected that the definition of the current image frame is less than a preset value, such as the preset value is 1024px, the current image frame can be subsequently enhanced. When it is detected that the definition of the current image frame is greater than or equal to the preset value, the current image frame can be directly used without subsequent optimization.

[0063] In some embodiments of the present invention, the determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, includes:

[0064] A clarity comparison is performed between the current image frame and the previous image frame, and based on the clarity comparison result, an alternative image matching the current image frame is determined from the computing power server as the target alternative image, or an alternative image matching the previous image frame is determined as the target alternative image.

[0065] In practical applications, the clarity of the current image frame can be compared with that of the previous image frame, and then according to the clarity comparison result, the target candidate image can be determined from multiple candidate images of the computing power server through continuous matching and optimization principles for subsequent optimization, so that the characters in the low-quality images in the video conference can be gradually restored. In some examples, the target candidate image can be a candidate image that matches the current image frame or a candidate image that matches the previous image frame.

[0066] In some embodiments of the present invention, the determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, includes:

[0067] Perform image feature comparison between the current image frame and the previous image frame, and based on the image feature comparison result, determine from the computing power server an alternative image that matches the current image frame as the target alternative image, or determine an alternative image that matches the previous image frame as the target alternative image.

[0068] In practical applications, the candidate images can be optimized from the image feature level, that is, the image features of the current image frame and the previous image frame can be compared, and then based on the image feature comparison results, a matching candidate image can be selected from the candidate images matching the current image frame and the candidate images matching the previous image frame as the target candidate image.

[0069] In some examples, the image feature comparison may be to compare key image features detected in the current image frame with key image features detected in the previous image frame, such as comparing the number of key image features or comparing the completeness of the key image features.

[0070] For example, key image features include the contour features of a person's eyes, mouth, ears, nose, and other facial features. Another example is key image features include the person's hairstyle features, head shape features, and the like.

[0071] In some embodiments of the present invention, determining, from the computing power server, a candidate image matching the current image frame includes:

[0072] A current image feature is extracted from the current image frame, and the current image feature is used to determine an alternative image matching the current image frame from a computing power server.

[0073] In actual applications, the AI ​​image processing engine can be started to extract current image features from the current image frame through image processing algorithms and image analysis algorithms, such as the fineness of color distribution, the vivid effect of brightness contrast, the clear display of edge contours, etc. Then, the current image features can be used through deep learning and other algorithms to determine alternative images that match the current image frame in the computing server.

[0074] In some examples, there may be multiple candidate images in the computing server that match the current image frame, and a most matching candidate image may be selected from the multiple matching candidate images.

[0075] In some embodiments of the present invention, determining the alternative image that matches the previous image frame includes: extracting previous image features from the previous image frame, and using the previous image features to determine the alternative image that matches the previous image frame from a computing power server.

[0076] In actual applications, the AI ​​image processing engine can be started to extract the previous image features from the previous image frame through image processing algorithms and image analysis algorithms, such as the delicacy of color distribution, the vivid effect of brightness contrast, the clear display of edge contours, etc. Then, the previous image features can be used through deep learning and other algorithms to determine the alternative images that match the previous image frame in the computing server.

[0077] In some examples, there may be multiple candidate images in the computing server that match the previous image frame, and then a most matching candidate image can be selected from the multiple matching candidate images.

[0078] Step 103: Perform image enhancement on the image frame based on the target candidate image.

[0079] After obtaining the target candidate image, the high-quality target candidate image can be used to perform frame-by-frame image enhancement on the low-quality image frames in the video conference, so as to achieve the correction and fitting of the current frame image with the high-definition digital human image, improve the picture quality of the video conference, and perform frame-by-frame enhancement processing on the played video stream under ultra-low bandwidth through the computing power network, thereby improving the texture of the picture during the video conference. It can not only repair the problem of image detail loss and color distortion, but also make the image clearer, smoother and more realistic.

[0080] In some embodiments of the present invention, image enhancement is performed on an image frame based on a target alternative image, including: based on the target alternative image, image enhancement is performed on the current image frame, or, image enhancement is performed on the previous image frame and the image-enhanced previous image frame replaces the current image frame.

[0081] For the current image frame, you can directly use the target alternative image to perform image enhancement on the current image frame, and then continue to use the image enhanced current image frame, or you can perform image enhancement on the previous image frame, and then replace the current image frame with the image enhanced previous image frame for use.

[0082] In some embodiments of the present invention, the image enhancement of the current image frame based on the target candidate image includes: determining high-quality image features from the target candidate image, and integrating the high-quality image features into the current image frame to enhance the current image frame.

[0083] In practical applications, algorithmic operations and image processing technology can be used to extract high-quality image features from high-quality target candidate images, and then integrate the high-quality image features into the low-quality current image frame, thereby achieving image enhancement of the current image frame. The high-definition character elements of the digital human can be integrated into the low-quality portrait, instantly improving the clarity of the low-quality portrait and significantly improving the image quality.

[0084] In some embodiments of the present invention, the image enhancement of the preceding image frame includes: determining high-quality image features from the target candidate image, and integrating the high-quality image features into the preceding image frame to enhance the image of the preceding image frame.

[0085] In practical applications, algorithmic operations and image processing technology can be used to extract high-quality image features from high-quality target candidate images, and then integrate the high-quality image features into the low-quality previous image frame, thereby achieving image enhancement of the previous image frame. The high-definition character elements of the digital human can be integrated into the low-quality portrait, instantly improving the clarity of the low-quality portrait and significantly improving the image quality.

[0086] In some examples, the AI ​​processing mode can be turned on when a trigger event is detected, and the steps of determining a target candidate image from a computing power server based on the image frame and using the target candidate image to perform image enhancement on the image frame are performed in the AI ​​processing mode.

[0087] In a video conference, the video stream may have image quality problems, such as blurred images (such as motion blur, out of focus), low image resolution, video freeze / low frame rate, and other image quality problems (such as insufficient color brightness, insufficient contrast, noise, over-compression, rain, snow, fog, excessive brightness or darkness, occlusion damage (such as logos, watermarks, subtitles, occlusions), color distortion (such as color cast, color loss), black and white photos. These image quality problems can be solved by applying AI processing mode.

[0088] In some examples, the above-mentioned image quality problems are more likely to occur at lower bandwidths, that is, when the bandwidth is less than the bandwidth threshold. For example, the bandwidth threshold of the video stream is 150kb / s. The trigger event can be an event in which the bandwidth is detected to be less than the bandwidth threshold. The server establishes a communication channel with each client. The server can be provided with a bandwidth monitoring module. The bandwidth fluctuation monitoring module can detect the bandwidth of each communication channel. Each channel can be provided with a certain bandwidth threshold. After the video conference is started, the server can detect whether the bandwidth of the channel connected to the client is less than the bandwidth threshold through the bandwidth monitoring module. When the bandwidth is greater than the bandwidth threshold, the trigger event is detected.

[0089] In some examples, the trigger event may also be an event in which the clarity of the video stream is detected to be less than a clarity threshold, such as a clarity threshold of 1024px. The server may monitor the video stream sent by the target conference terminal in real time, and when the video stream clarity is less than the clarity threshold, a trigger event is detected.

[0090] In some examples, the trigger event may also be a user manual operation event, that is, the user manually turns on the AI ​​processing mode. An interactive interface may be provided to the user on the client, and the user may perform manual operations through the interactive interface, that is, the trigger event is detected.

[0091] In some examples, a scheduling platform can be set up, which can be used to control the number of cameras and microphones that users are allowed to turn on and off, and can control whether to allow users to turn on the AI ​​processing mode, which also requires scheduling. In some examples, due to the computing power of the background GPU cluster, each user may not be allowed to turn on and off the AI ​​enhancement and synthesis functions at will.

[0092] The following is a detailed description of the process of determining target candidate images and image enhancement:

[0093] For clarity comparison:

[0094] 1. When the definition of the current image frame is less than or equal to the definition of the previous image frame and the definition difference between the current image frame and the previous image frame is greater than the preset difference:

[0095] The step of determining, from the computing power server, a candidate image matching the current image frame as a target candidate image according to the clarity comparison result, or determining a candidate image matching the previous image frame as a target candidate image, includes:

[0096] When the clarity of the current image frame is less than or equal to the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, the candidate image matching the previous image frame is determined as the target candidate image.

[0097] The method of using the target alternative image to perform image enhancement on the current image frame, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: using the target alternative image to perform image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

[0098] In the case where the clarity of the current image frame is less than or equal to the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than the preset difference, the previous image frame can be used to continue image enhancement, that is, an alternative image matching the previous image frame is used to enhance the previous image frame, and the enhanced previous image frame is used to replace the current image frame for use.

[0099] 2. When the definition of the current image frame is greater than that of the previous image frame and the definition difference between the current image frame and the previous image frame is greater than a preset difference:

[0100] The method of determining, from the computing power server, an alternative image matching the current image frame as a target alternative image based on a clarity comparison result, or determining an alternative image matching the previous image frame as a target alternative image, includes: when the clarity of the current image frame is greater than the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the alternative image matching the current image frame as the target alternative image.

[0101] The adopting of the target candidate image to perform image enhancement on the current image frame, or the adopting of the previous image frame to perform image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: adopting the target candidate image to perform image enhancement on the current image frame.

[0102] In the case where the clarity of the current image frame is greater than that of the previous image frame and the difference in clarity between the current image frame and the previous image frame is greater than a preset difference, using the current image frame may re-match a more preferred alternative image, and the preferred alternative image and the current image frame may be enhanced, that is, the current image frame is enhanced using the alternative image that matches the current image frame.

[0103] 3. When the difference in definition between the current image frame and the previous image frame is less than the preset difference:

[0104] The method of determining, from the computing power server, an alternative image matching the current image frame as a target alternative image based on a result of clarity comparison, or determining an alternative image matching the previous image frame as a target alternative image, comprises: when a clarity difference between the current image frame and the previous image frame is less than a preset difference, selecting a target alternative image from the alternative images matching the current image frame and the alternative images matching the previous image frame.

[0105] The method of selecting a target alternative image from the alternative images matching the current image frame and the alternative images matching the previous image frame includes: determining a first similarity between the current image frame and the matching alternative image, and determining a second similarity between the previous image frame and the matching alternative image; and selecting a target alternative image from the alternative images matching the current image frame and the alternative images matching the previous image frame based on the first similarity and the second similarity.

[0106] The method of using the target candidate image to perform image enhancement processing on the current image frame, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: using the target candidate image, randomly selecting the current image frame to be image enhanced, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

[0107] For the case where the difference in clarity between the current image frame and the previous image frame is less than a preset difference, that is, the clarity of the current image frame is approximately equal to the clarity of the previous image frame, and the alternative image matched by the current image frame and the alternative image matched by the previous image frame may be different, then when the alternative image matched by the current image frame and the alternative image matched by the previous image frame are different, the alternative image with the highest similarity can be obtained as the target alternative image based on the principle of high similarity in feature quantity matching, that is, the target alternative image is selected from the alternative image matched by the current image frame and the alternative image matched by the previous image frame based on the first similarity and the second similarity.

[0108] After determining the target candidate image, the target candidate image can be used to randomly enhance the current image frame or the previous image frame, that is, the target candidate image is used to enhance the current image frame, or the target candidate image is used to enhance the previous image frame, and then the enhanced previous image frame replaces the current image frame for use.

[0109] For the scheme of image feature comparison:

[0110] The image feature comparison may be to compare the key image features detected in the current image frame with the key image features detected in the previous image frame, such as comparing the number of key image features.

[0111] 1. When the number of key image features of the current image frame is less than or equal to the number of key image features of the previous image frame and the difference between the number of key image features of the current image frame and the previous image frame is greater than the preset difference:

[0112] The step of determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image according to the image feature comparison result, includes:

[0113] When the number of key image features of the current image frame is less than or equal to the number of key image features of the previous image frame and the number of key image features of the current image frame and the previous image frame is greater than a preset difference, the alternative image matching the previous image frame is determined as the target alternative image.

[0114] The method of using the target alternative image to perform image enhancement on the current image frame, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: using the target alternative image to perform image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

[0115] In the case where the number of key image features of the current image frame is less than or equal to the number of key image features of the previous image frame and the difference in the number of key image features between the current image frame and the previous image frame is greater than a preset difference, the previous image frame can be used to continue image enhancement, that is, an alternative image matching the previous image frame is used to enhance the previous image frame, and the enhanced previous image frame is used to replace the current image frame.

[0116] 2. When the number of key image features of the current image frame is greater than the number of key image features of the previous image frame and the difference between the number of key image features of the current image frame and the previous image frame is greater than the preset difference:

[0117] The method of determining, from the computing power server, an alternative image matching the current image frame as a target alternative image based on the image feature comparison result, or determining an alternative image matching the previous image frame as a target alternative image, includes: when the number of key image features of the current image frame is greater than the number of key image features of the previous image frame and the difference in the number of key image features between the current image frame and the previous image frame is greater than a preset difference, determining the alternative image matching the current image frame as the target alternative image.

[0118] The adopting of the target candidate image to perform image enhancement on the current image frame, or the adopting of the previous image frame to perform image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: adopting the target candidate image to perform image enhancement on the current image frame.

[0119] In the case where the number of key image features in the current image frame is greater than the number of key image features in the previous image frame and the difference in the number of key image features between the current image frame and the previous image frame is greater than a preset difference, using the current image frame may re-match a more preferred alternative image, and the preferred alternative image and the current image frame may be enhanced, that is, the current image frame is enhanced using the alternative image that matches the current image frame.

[0120] 3. When the difference between the number of key image features of the current image frame and the previous image frame is less than the preset difference:

[0121] The method of determining, from a computing power server, an alternative image matching the current image frame as a target alternative image based on the image feature comparison result, or determining an alternative image matching the previous image frame as a target alternative image, comprises: when a difference in the number of key image features between the current image frame and the previous image frame is less than a preset difference, selecting a target alternative image from the alternative images matching the current image frame and the alternative images matching the previous image frame.

[0122] The method of selecting a target alternative image from the alternative images matching the current image frame and the alternative images matching the previous image frame includes: determining a first similarity between the current image frame and the matching alternative image, and determining a second similarity between the previous image frame and the matching alternative image; and selecting a target alternative image from the alternative images matching the current image frame and the alternative images matching the previous image frame based on the first similarity and the second similarity.

[0123] The method of using the target candidate image to perform image enhancement processing on the current image frame, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: using the target candidate image, randomly selecting the current image frame to be image enhanced, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

[0124] For the case where the difference in the number of key image features between the current image frame and the previous image frame is less than the preset difference, that is, the number of key image features of the current image frame is approximately equal to the number of key image features of the previous image frame, and the alternative image matched with the current image frame and the alternative image matched with the previous image frame may be different, then when the alternative image matched with the current image frame and the alternative image matched with the previous image frame are different, the alternative image with the highest similarity can be obtained as the target alternative image based on the principle of high similarity in feature number matching, that is, the target alternative image is selected from the alternative image matched with the current image frame and the alternative image matched with the previous image frame based on the first similarity and the second similarity.

[0125] After determining the target candidate image, the target candidate image can be used to randomly enhance the current image frame or the previous image frame, that is, the target candidate image is used to enhance the current image frame, or the target candidate image is used to enhance the previous image frame, and then the enhanced previous image frame replaces the current image frame for use.

[0126] In an embodiment of the present invention, by acquiring image frames in a video conference, determining target candidate images from a computing server based on the image frames, and then enhancing the image frames based on the target candidate images, it is possible to use high-quality alternative images to enhance low-quality image frames in the video conference, thereby improving the quality of the images in the video conference and avoiding the poor picture effects caused by directly using digital humans to replace real portrait images, thereby improving the video conferencing experience of the participants.

[0127] Reference Figure 3 , shows a flowchart of another method for image enhancement in a video conference provided by some embodiments of the present invention, which may specifically include the following steps:

[0128] Step 301, obtaining image frames in a video conference.

[0129] Step 302, compare the image features of the current image frame and the previous image frame, and based on the image feature comparison result, determine from the computing power server an alternative image that matches the current image frame as the target alternative image, or determine an alternative image that matches the previous image frame as the target alternative image.

[0130] Step 303: Based on the target candidate image, the current image frame is enhanced, or the previous image frame is enhanced and the enhanced previous image frame replaces the current image frame.

[0131] Reference Figure 4 , shows a flowchart of another method for image enhancement in a video conference provided by some embodiments of the present invention, which may specifically include the following steps:

[0132] Step 401, obtaining image frames in a video conference.

[0133] Step 402, compare the clarity of the current image frame with the previous image frame, and based on the clarity comparison result, determine from the computing power server an alternative image that matches the current image frame as the target alternative image, or determine an alternative image that matches the previous image frame as the target alternative image.

[0134] Step 403: Based on the target candidate image, the current image frame is enhanced, or the previous image frame is enhanced and the enhanced previous image frame replaces the current image frame.

[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0136] Reference Figure 5 , shows a schematic diagram of the structure of an apparatus for image enhancement in a video conference provided by some embodiments of the present invention, which may specifically include the following modules:

[0137] An image frame acquisition module 501 is used to acquire image frames in a video conference;

[0138] A target candidate image determination module 502 is used to determine a target candidate image from a computing power server according to the image frame;

[0139] The image enhancement module 503 is used to perform image enhancement on the image frame based on the target candidate image.

[0140] Optionally, determining a target candidate image from a computing power server according to the image frame includes:

[0141] Determine from the computing power server the candidate image that matches the current image frame as the target candidate image, or determine the candidate image that matches the previous image frame as the target candidate image;

[0142] The step of performing image enhancement on the image frame based on the target candidate image includes:

[0143] Based on the target candidate image, the current image frame is image enhanced, or the previous image frame is image enhanced and the image-enhanced previous image frame replaces the current image frame.

[0144] Optionally, the determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, includes:

[0145] Comparing image features of a current image frame with a previous image frame, and determining, from a computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, according to the image feature comparison result;

[0146] Alternatively, a clarity comparison is performed between the current image frame and the previous image frame, and based on the clarity comparison result, an alternative image matching the current image frame is determined from the computing power server as the target alternative image, or an alternative image matching the previous image frame is determined as the target alternative image.

[0147] Optionally, determining, from the computing power server, a candidate image matching the current image frame includes:

[0148] Extracting current image features from the current image frame, and using the current image features to determine a candidate image matching the current image frame from a computing power server;

[0149] The determining of the candidate image matching the preceding image frame comprises:

[0150] A preceding image feature is extracted from the preceding image frame, and the preceding image feature is used to determine an alternative image matching the preceding image frame from a computing power server.

[0151] Optionally, the performing image enhancement on the current image frame based on the target candidate image includes:

[0152] Determine high-quality image features from the target candidate image, and integrate the high-quality image features into the current image frame to perform image enhancement on the current image frame;

[0153] The performing image enhancement on the preceding image frame comprises:

[0154] A high-quality image feature is determined from the target candidate image, and the high-quality image feature is integrated into the preceding image frame to perform image enhancement on the preceding image frame.

[0155] Optionally, the determining, from the computing power server according to the clarity comparison result, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, comprises: when the clarity of the current image frame is less than or equal to the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the candidate image matching the previous image frame as the target candidate image;

[0156] The method of performing image enhancement on the current image frame based on the target alternative image, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: performing image enhancement on the previous image frame based on the target alternative image and replacing the current image frame with the image enhanced previous image frame.

[0157] Optionally, the determining, from the computing power server according to the clarity comparison result, the candidate image matching the current image frame as the target candidate image, or the candidate image matching the previous image frame as the target candidate image, comprises: when the clarity of the current image frame is greater than the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the candidate image matching the current image frame as the target candidate image;

[0158] The performing image enhancement on the current image frame based on the target candidate image, or performing image enhancement on the previous image frame and replacing the current image frame with the image-enhanced previous image frame, includes: performing image enhancement on the current image frame based on the target candidate image.

[0159] Optionally, the determining, from the computing power server according to the clarity comparison result, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, comprises: when a clarity difference between the current image frame and the previous image frame is less than a preset difference, selecting the target candidate image from the candidate image matching the current image frame and the candidate image matching the previous image frame;

[0160] The method of performing image enhancement processing on the current image frame based on the target alternative image, or performing image enhancement processing on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: randomly selecting to perform image enhancement on the current image frame based on the target alternative image, or performing image enhancement processing on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

[0161] Optionally, selecting a target candidate image from candidate images matching the current image frame and candidate images matching the previous image frame includes:

[0162] Determining a first similarity between the current image frame and the matching candidate image, and determining a second similarity between the previous image frame and the matching candidate image;

[0163] According to the first similarity and the second similarity, a target candidate image is selected from the candidate images matching the current image frame and the candidate images matching the previous image frame.

[0164] In an embodiment of the present invention, by acquiring image frames in a video conference, determining target candidate images from a computing server based on the image frames, and then enhancing the image frames based on the target candidate images, it is possible to use high-quality alternative images to enhance low-quality image frames in the video conference, thereby improving the quality of the images in the video conference and avoiding the poor picture effects caused by directly using digital humans to replace real portrait images, thereby improving the video conferencing experience of the participants.

[0165] Some embodiments of the present invention further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the above method is implemented when the computer program is executed by the processor.

[0166] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0167] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method when executed by a processor.

[0168] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0170] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0171] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0172] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0176] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0177] The above is a detailed introduction to a method and device for image enhancement in a video conference. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for image enhancement in video conferencing, characterized in that: The method comprises: Get the image frame in the video conference; Determining a target candidate image from a computing power server according to the image frame; Based on the target candidate image, image enhancement is performed on the image frame.

2. The method according to claim 1, characterized in that The step of determining a target candidate image from a computing power server according to the image frame includes: Determine from the computing power server the candidate image that matches the current image frame as the target candidate image, or determine the candidate image that matches the previous image frame as the target candidate image; The step of performing image enhancement on the image frame based on the target candidate image includes: Based on the target candidate image, the current image frame is image enhanced, or the previous image frame is image enhanced and the image-enhanced previous image frame replaces the current image frame.

3. The method according to claim 2, characterized in that The step of determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, includes: Comparing image features of a current image frame with a previous image frame, and determining, from a computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, according to the image feature comparison result; Alternatively, a clarity comparison is performed between the current image frame and the previous image frame, and based on the clarity comparison result, an alternative image matching the current image frame is determined from the computing power server as the target alternative image, or an alternative image matching the previous image frame is determined as the target alternative image.

4. The method according to claim 2 or 3, characterized in that: The determining, from the computing power server, a candidate image matching the current image frame includes: Extracting current image features from the current image frame, and using the current image features to determine a candidate image matching the current image frame from a computing power server; The determining of the candidate image matching the previous image frame comprises: A preceding image feature is extracted from the preceding image frame, and the preceding image feature is used to determine an alternative image matching the preceding image frame from a computing power server.

5. The method according to claim 2 or 3, characterized in that: The performing image enhancement on the current image frame based on the target candidate image includes: Determine high-quality image features from the target candidate image, and integrate the high-quality image features into the current image frame to perform image enhancement on the current image frame; The performing image enhancement on the preceding image frame comprises: A high-quality image feature is determined from the target candidate image, and the high-quality image feature is integrated into the preceding image frame to perform image enhancement on the preceding image frame.

6. The method according to claim 3, characterized in that: The step of determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, according to a result of the clarity comparison, comprises: when the clarity of the current image frame is less than or equal to the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the candidate image matching the previous image frame as the target candidate image; The method of performing image enhancement on the current image frame based on the target alternative image, or performing image enhancement on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: performing image enhancement on the previous image frame based on the target alternative image and replacing the current image frame with the image enhanced previous image frame.

7. The method according to claim 3, characterized in that The step of determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image, according to the clarity comparison result, includes: when the clarity of the current image frame is greater than the clarity of the previous image frame and the clarity difference between the current image frame and the previous image frame is greater than a preset difference, determining the candidate image matching the current image frame as the target candidate image; The performing image enhancement on the current image frame based on the target candidate image, or performing image enhancement on the previous image frame and replacing the current image frame with the image-enhanced previous image frame, includes: performing image enhancement on the current image frame based on the target candidate image.

8. The method according to claim 3, characterized in that The step of determining, from the computing power server, a candidate image matching the current image frame as a target candidate image, or determining a candidate image matching the previous image frame as a target candidate image according to the clarity comparison result, comprises: selecting the target candidate image from the candidate image matching the current image frame and the candidate image matching the previous image frame when the clarity difference between the current image frame and the previous image frame is less than a preset difference; The method of performing image enhancement processing on the current image frame based on the target alternative image, or performing image enhancement processing on the previous image frame and replacing the current image frame with the image enhanced previous image frame, includes: randomly selecting to perform image enhancement on the current image frame based on the target alternative image, or performing image enhancement processing on the previous image frame and replacing the current image frame with the image enhanced previous image frame.

9. The method according to claim 8, characterized in that The selecting a target candidate image from the candidate images matching the current image frame and the candidate images matching the previous image frame comprises: Determining a first similarity between the current image frame and the matching candidate image, and determining a second similarity between the previous image frame and the matching candidate image; According to the first similarity and the second similarity, a target candidate image is selected from the candidate images matching the current image frame and the candidate images matching the previous image frame.

10. A device for image enhancement in video conferencing, characterized in that: The device comprises: An image frame acquisition module is used to acquire image frames in a video conference; A target candidate image determination module, used to determine a target candidate image from a computing power server according to the image frame; An image enhancement module is used to perform image enhancement on the image frame based on the target candidate image.

Images