Video background processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video background processing method and device, electronic equipment and storage medium, relating to the technical field of computer. The method comprises: determining the current formality of a video conference according to a current video frame of a speaking terminal of the video conference and audio data corresponding to the current video frame; determining the interference degree of a background image region of the current video frame on the video conference; performing weighted summation on the current formality and the interference degree to obtain a comprehensive evaluation value of the video conference; determining a corresponding background processing strategy according to the comprehensive evaluation value, and processing the background image region of the current video frame according to the background processing strategy to obtain a target video frame; and sending the target video frame and the audio data to each video conference terminal participating in the video conference. Thus, the background image region of the current video frame is automatically processed, thereby reducing the dependence on manual operation of the user and improving the intelligent level of the video conference and the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video background processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of mobile internet technology, network-based video communication has become increasingly popular. Video conferencing, as an important application, has been widely used in scenarios such as remote work and online education. Video conferencing is a meeting method in which participants use terminal devices to transmit video in real time over a network to achieve face-to-face communication.

[0003] During a video conference, if a user feels the conference is too formal and wants to avoid the background of the video feed captured by the camera affecting the meeting, the user needs to manually enable the background blur function. Correspondingly, after receiving the video frame uploaded by the user's terminal, the video conferencing server can virtualize the background of the video frame based on this setting and distribute the processed image to other participating terminals.

[0004] However, this method is highly dependent on the user's active operation. If the speaking user fails to enable the background blur function in advance, the video conferencing service will by default not process the background of the video frames uploaded by the speaking user's terminal, resulting in the original background image being directly transmitted to all participating terminals. This unprocessed background can easily distract the participants and reduce the quality of the video conference. Summary of the Invention

[0005] This disclosure provides a video background processing method, apparatus, electronic device, and storage medium.

[0006] In a first aspect, this disclosure provides a video background processing method, the method comprising: determining the current formality of the video conference based on the current video frame of the speaking end of the video conference and the corresponding audio data of the current video frame; determining the interference degree of the background image region of the current video frame to the video conference; performing a weighted summation of the current formality and the interference degree to obtain a comprehensive evaluation value of the video conference; determining a corresponding background processing strategy based on the comprehensive evaluation value, and processing the background image region of the current video frame according to the background processing strategy to obtain a target video frame; and sending the target video frame and the audio data to each video conferencing terminal participating in the video conference.

[0007] Secondly, this disclosure provides a video background processing apparatus, the apparatus comprising: a first determining module, configured to determine the current formality of the video conference based on the current video frame of the speaking end of the video conference and the audio data corresponding to the current video frame; a second determining module, configured to determine the interference degree of the background image region of the current video frame to the video conference; a weighted summation module, configured to perform a weighted summation of the current formality and the interference degree to obtain a comprehensive evaluation value of the video conference; a processing module, configured to determine a corresponding background processing strategy based on the comprehensive evaluation value, and process the background image region of the current video frame according to the background processing strategy to obtain a target video frame; and a sending module, configured to send the target video frame and the audio data to each video conferencing terminal participating in the video conference.

[0008] Thirdly, this disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the video background processing method disclosed in the embodiments of this disclosure.

[0009] Fourthly, this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the video background processing method disclosed in the embodiments of this disclosure.

[0010] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the video background processing method disclosed in the embodiments of this disclosure.

[0011] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: Based on the current video frame and corresponding audio data from the speaker in the video conference, the current formality of the video conference is determined; the background image region of the current video frame is determined, and its interference level is assessed; the formality level and interference level are weighted and summed to obtain a comprehensive evaluation value for the video conference; based on the comprehensive evaluation value, a corresponding background processing strategy is determined, and the background image region of the current video frame is processed according to the background processing strategy to obtain the target video frame; the target video frame and audio data are then sent to each participating video conference terminal. Thus, by combining the current video frame and corresponding audio data from the speaker, a corresponding background processing strategy is determined, and the background image region of the current video frame is automatically processed according to the background processing strategy, thereby reducing reliance on manual user operation and improving the intelligence level and user experience of the video conference. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] Figure 1 This is a flowchart illustrating a video background processing method according to an exemplary embodiment; Figure 2 A flowchart illustrating another video background processing method according to an exemplary embodiment; Figure 3 A flowchart illustrating another video background processing method according to an exemplary embodiment; Figure 4 This is a schematic diagram illustrating the structure of a video background processing apparatus according to an exemplary embodiment; Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment.

[0014] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0016] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0017] First, combine Figure 1 The present disclosure provides an exemplary description of the video background processing method provided in the embodiments.

[0018] Figure 1 This is a flowchart illustrating a video background processing method according to an exemplary embodiment.

[0019] It should be noted that the video background processing method provided in this embodiment can be executed by a video background processing device, which can be implemented by software and / or hardware. The video background processing device can be an electronic device, or it can be configured within an electronic device.

[0020] It should be noted that the electronic device can be any device with computing capabilities, such as a server, and this embodiment does not specifically limit the electronic device. It should also be noted that this embodiment uses a video conferencing server as an example for illustrative description.

[0021] like Figure 1 As shown, the video background processing method includes the following steps: Step 101: Determine the current formality of the video conference based on the current video frame and the corresponding audio data of the speaker in the video conference.

[0022] In some embodiments, the current video frame of the speaker in the video conference and the corresponding audio data are obtained, and the current formality of the video conference is determined based on the current video frame and the audio data.

[0023] In some embodiments, the formality of the video conference can be calculated based on the current video frame and audio data to obtain the current formality of the video conference.

[0024] The speaking end refers to the terminal used by the user who is speaking in the video conference.

[0025] Step 102: Determine the background image region of the current video frame and its interference level with the video conference.

[0026] In some embodiments, the background image region of the current video frame is determined based on the image region of the second participant in the video conference, where the role of the second participant in the video conference is marked as a conference host, ordinary user, or leader. The background image region of the current video frame is analyzed to determine the degree of interference of the background image region to the video conference.

[0027] In some embodiments, people can be detected in the current video frame to obtain the visual features of the people. If the visual features of the people match the visual features of a participant in the participant list of the video conference, then the people are identified as the second participants, and the image region of the person where the second participants are located is determined in the current video frame.

[0028] It should be noted that the visual characteristics of each participant in the video conference participant list can be obtained in advance by the video conference organizer; for example, the visual characteristics can be facial features.

[0029] The background image region refers to the image region in the current video frame other than the image region of the person.

[0030] Step 103: Weight the current level of formality and the level of interference to obtain the comprehensive evaluation value of the video conference.

[0031] In some embodiments, a preset first weight is used to weight the current formality level; a preset second weight is used to weight the interference level; the weighted current formality level and interference level are summed to obtain the comprehensive evaluation value of the video conference.

[0032] In this embodiment, the sum of the first weight and the second weight is equal to 1.

[0033] The first and second weights are preset according to actual needs.

[0034] In some embodiments, if a participant marked as a leader exists in the participant list of the video conference, the first weight is greater than the second weight. For example, the first weight can be 0.7 and the second weight can be 0.3.

[0035] As an example, this is used to obtain the overall evaluation value C for video conferencing. t The calculation formula can be: C t =λ1•S t +λ2•D t In the formula, λ1 represents the first weight; λ2 represents the second weight; S t Indicates the current level of formality of the video conference; D t Indicates the degree of interference.

[0036] Step 104: Based on the comprehensive evaluation value, determine the corresponding background processing strategy, and process the background image region of the current video frame according to the background processing strategy to obtain the target video frame.

[0037] In some embodiments, a target value range to which the comprehensive evaluation value belongs can be determined, and a background processing strategy corresponding to the target value range can be obtained. The background image region of the current video frame can be processed according to the background processing strategy to obtain the target video frame.

[0038] In some embodiments, the background processing strategy corresponding to the target value range can be obtained based on the pre-saved correspondence between the value range and the background processing strategy.

[0039] The background processing strategy refers to the strategy for processing the background image region of the current video frame.

[0040] It is understandable that different value ranges may correspond to different background processing strategies.

[0041] In other embodiments, after obtaining the comprehensive evaluation value, the comprehensive evaluation value can be compared with the first threshold and the second threshold respectively, wherein the second threshold is greater than the first threshold. If the comprehensive evaluation value is greater than or equal to the first threshold and less than the second threshold, the corresponding background processing strategy is determined to be: blurring the face regions of non-participants in the background image region of the current video frame.

[0042] In other embodiments, when the comprehensive evaluation value is greater than or equal to the second threshold, the corresponding background processing strategy is determined to be: to perform virtualization processing on the background image region of the current video frame.

[0043] In some embodiments, to enable the current video frame to use a background image related to the conference topic, if the comprehensive evaluation value is greater than or equal to a second threshold, the corresponding background processing strategy can be determined as follows: obtain a corresponding virtual background image based on the conference topic, and replace the background image area of ​​the current video frame with the virtual background image. This effectively reduces the risk of external interference and leakage of real-world information, while enhancing the sense of organization and authority of the meeting, thereby optimizing meeting efficiency and improving overall communication quality.

[0044] Step 105: Send the target video frame and audio data to each video conferencing terminal participating in the video conference.

[0045] The video background processing method provided in this disclosure determines the current formality of the video conference based on the current video frame and corresponding audio data of the speaking end; determines the background image region of the current video frame and its interference level; performs a weighted summation of the current formality level and interference level to obtain a comprehensive evaluation value for the video conference; determines a corresponding background processing strategy based on the comprehensive evaluation value, and processes the background image region of the current video frame according to the background processing strategy to obtain a target video frame; and sends the target video frame and audio data to each video conferencing terminal participating in the video conference. Thus, by combining the current video frame and corresponding audio data of the speaking end to determine the corresponding background processing strategy and automatically processing the background image region of the current video frame according to the background processing strategy, the reliance on manual user operation is reduced, improving the intelligence level of the video conference and the user experience.

[0046] Figure 2 This is a flowchart illustrating another video background processing method according to an exemplary embodiment.

[0047] like Figure 2As shown, the method may further include: Step 201: Obtain the current video frame and the corresponding audio data of the speaker in the video conference.

[0048] Step 202: Match the text content of the audio data with a preset word library to determine the first number of words in the text content that belong to the preset word library.

[0049] The preset vocabulary database is built in advance based on words frequently used in formal sample video conferences.

[0050] Step 203: Determine the second number of participants in the current video frame who are the first participants in the video conference, wherein the role of the first participants in the video conference is marked as the leader.

[0051] In some embodiments, one possible implementation for determining the second number of first participants in the video conference included in the current video frame is as follows: determining the visual features of the people in the current video frame; if the visual features match features in a preset participant feature library, identifying the people as first participants, wherein the participant feature library is formed based on the visual features of participants marked as leaders in the video conference; and determining the second number of first participants. This accurately determines the second number of first participants in the video conference included in the current video frame.

[0052] In some embodiments, visual features may include facial features.

[0053] Step 204: Determine the current formality of the video conference based on the first quantity and the second quantity.

[0054] It should be noted that the methods for determining the current formality of a video conference based on the first and second quantities differ in different application scenarios. Examples are provided below: As an example, the formality of the video conference can be calculated based on the first and second quantities to obtain the current formality of the video conference.

[0055] As another example, the initial formality level of the video conference is determined based on the conference description information; the initial formality level is then adjusted based on a first quantity and a second quantity to obtain the current formality level of the video conference. Thus, by combining the initial formality level of the video conference with the first and second quantities determined based on the current video frame, the current formality level of the video conference at the current video frame is accurately determined.

[0056] In some embodiments, the formula for calculating the current formality level can be expressed as: S t=clip(S0+ΔS t ,0,1) ΔS t =w k •k(a t )+w i •i t Wherein, w in the formula k Indicates that for k(a) t Added preset weights; w i Indicates i t Added preset weights; k(a) t The score represents the formality score obtained by counting the first number of words belonging to a predefined vocabulary in the audio data of the current video frame; t The formality score is obtained based on the second number of the first participant detected in the current frame; clip is used to make S t The value ranges from 0 to 1. It should be noted that the formality score here can range from 0 to 1.

[0057] The meeting description information describes the video conference.

[0058] In this embodiment, the meeting description information may include, but is not limited to, the meeting topic, agenda text, etc. This embodiment does not specifically limit the meeting description information.

[0059] In some embodiments, the formality of the video conference can be analyzed based on the conference description information to obtain the initial formality of the video conference.

[0060] In some embodiments, one possible way to determine the initial formality of a video conference based on the conference description information is to: extract keywords from the conference description information to obtain keyword extraction results; and determine the initial formality of the video conference based on the keyword extraction results.

[0061] In some embodiments, to further improve the accuracy of the determined initial formality of the video conference, a third number of participants whose roles are marked as leaders in the participant list of the video conference can be determined, and the formality of the video conference can be calculated based on the keyword extraction results and the third number to obtain the initial formality of the video conference.

[0062] In some embodiments, the initial formality of a video conference can be calculated using a formula for calculating formality, based on the keyword extraction results and a third quantity.

[0063] As an example, the formula S0 used to calculate the initial formality of a video conference can be: S0=σ(w t •k(T)+w l •l(P) in, Compress the result to 0~1.

[0064] Wherein, w in the formula t This represents the preset weights added to k(T); w l represents the preset weight added to l(P); k(T) represents the formality score obtained based on the keyword extraction results, and l(P) represents the formality score obtained based on the third quantity.

[0065] Step 205: Determine the background image region of the current video frame and its interference level with the video conference.

[0066] In some embodiments, the background image region of the current video frame can be determined based on the image region of the second participant in the video conference, where the role of the second participant in the video conference is marked as a meeting host, a regular user, or a leader; abnormal behavior identification is performed on the background image region of the current video frame to obtain an abnormal behavior confidence level; if there are non-participants in the background image region, the area ratio of the target image region containing the non-participants in the background image region is determined; and the interference degree of the background image region of the current video frame on the video conference is determined based on the abnormal behavior confidence level and the area ratio.

[0067] In some embodiments, the background image region of the current video frame can be input into the abnormal behavior recognition model to obtain the confidence level of the abnormal behavior through the abnormal behavior recognition model.

[0068] Among them, the abnormal behavior confidence score is used to indicate the confidence level that there is abnormal behavior in the background image region of the current video frame.

[0069] In some embodiments, an image region other than the human image region in the current video frame can be used as a background image region.

[0070] In some embodiments, the interference degree D of the background image region of the current video frame to the video conference is obtained. t The calculation formula can be expressed as: D t =α•(1-M t )+β•B t +γ•A t Among them, M t This indicates whether there are non-participants in the background image region of the current video frame. If non-participants exist, M... t The value of M is equal to 1. If there are no non-attendees, M is...t The value of B is equal to 0; t A represents the confidence level of abnormal behavior in the background image region of the current video frame; t This represents the area ratio of the target image region containing non-participants to the background image region; α, β, and γ are pre-set weights, where α+β+γ=1.

[0071] In some embodiments, the confidence level of the above-mentioned abnormal behavior ranges from 0 to 1; the area ratio ranges from 0 to 1.

[0072] Step 206: Weight the current level of formality and the level of interference to obtain the comprehensive evaluation value of the video conference.

[0073] Step 207: Based on the comprehensive evaluation value, determine the corresponding background processing strategy, and process the background image region of the current video frame according to the background processing strategy to obtain the target video frame.

[0074] Step 208: Send the target video frame and audio data to each video conferencing terminal participating in the video conference.

[0075] It should be noted that for a detailed description of steps 206 to 208, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0076] In this embodiment, after acquiring the current video frame and corresponding audio data of the speaker in the video conference, the text content of the audio data is matched with a preset word library to determine the first number of words belonging to the preset word library in the text content, and the second number of participants in the video conference included in the current video frame is determined. Based on the first and second numbers, the current formality of the video conference is determined. Therefore, by analyzing the current video frame and its corresponding audio data, the current formality of the video conference at that particular video frame is accurately determined, which helps to improve the accuracy of the subsequently determined background processing strategy and enhances the accuracy of processing the background image region of the current video frame.

[0077] To facilitate a clear understanding of this disclosure, the following will be combined with... Figure 3 The video background processing method of this embodiment is described by way of example.

[0078] Figure 3 This is a flowchart illustrating another video background processing method according to an exemplary embodiment.

[0079] like Figure 3 As shown, the method may further include: Step 301: Determine the initial formality of the video conference based on the conference description information and the third number of participants whose roles are marked as leaders in the participant list of the video conference.

[0080] For a detailed description of determining the initial formality of the video conference, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0081] Step 302: Obtain the current video frame and the corresponding audio data of the speaker in the video conference.

[0082] Step 303: Determine the current formality level of the video conference at the current video frame based on the initial formality level, the current video frame, and the audio data.

[0083] It should be noted that for a detailed description of determining the current level of formality, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0084] Step 304: Determine the background image region of the current video frame and its interference level with the video conference.

[0085] Step 305: Weighted summation of the current formality and interference levels to obtain the comprehensive evaluation value of the video conference.

[0086] It should be noted that for a detailed description of steps 304 and 305, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0087] Step 306: If the comprehensive evaluation value is greater than or equal to the first threshold and less than the second threshold, blur the face regions of non-participants in the background image region of the current video frame to obtain the target video frame.

[0088] Step 307: If the comprehensive evaluation value is greater than or equal to the second threshold, obtain the corresponding virtual background image according to the meeting topic of the video conference, and replace the background image area of ​​the current video frame with the virtual background image to obtain the target video frame.

[0089] In some embodiments, if the comprehensive evaluation value is less than a first threshold, the current video frame can be directly used as the target video frame.

[0090] Step 308: Send the target video frame and audio data to each video conferencing terminal participating in the video conference.

[0091] In this embodiment, during a speaker's presentation in a video conference, the video conferencing server can determine the background processing strategy for the current video frame based on the speaker's current video frame and corresponding audio data. The server then automatically processes the background image area of ​​the current video frame according to the background processing strategy to obtain the target video frame. This target video frame, along with the audio data, is then sent to the participating video conferencing terminals. This eliminates the need for the user to enable background blurring beforehand, as the video conferencing server can automatically process the background image area of ​​the current video frame. This reduces reliance on manual user intervention and improves the intelligence level and user experience of the video conferencing.

[0092] Figure 4 This is a schematic diagram illustrating the structure of a video background processing apparatus according to an exemplary embodiment.

[0093] like Figure 4 As shown, the video background processing device 400 includes: a first determining module 401, a second determining module 402, a weighted summation module 403, a processing module 404, and a sending module 405, wherein: The first determining module 401 is used to determine the current formality of the video conference based on the current video frame of the speaker and the corresponding audio data of the current video frame.

[0094] The second determining module 402 is used to determine the background image region of the current video frame and its interference level with the video conference.

[0095] The weighted summation module 403 is used to perform a weighted summation of the current formality and interference levels to obtain a comprehensive evaluation value for the video conference.

[0096] The processing module 404 is used to determine the corresponding background processing strategy based on the comprehensive evaluation value, and to process the background image region of the current video frame according to the background processing strategy to obtain the target video frame.

[0097] The sending module 405 is used to send the target video frame and audio data to each video conferencing terminal participating in the video conference.

[0098] In one embodiment of this disclosure, the first determining module 401 may include: The first determining unit is used to match the text content of the audio data with a preset word library to determine the first number of words in the text content that belong to the preset word library; The second determining unit is used to determine the second number of participants in the video conference included in the current video frame, wherein the role of the first participant in the video conference is marked as the leader; The third determining unit is used to determine the current formality of the video conference based on the first and second quantities.

[0099] In one embodiment of this disclosure, the second determining unit is specifically configured to: determine the visual characteristics of a person in the current video frame; if the visual characteristics match the characteristics in a preset participant feature library, designate the person as the first participant, wherein the participant feature library is formed based on the visual characteristics of participants marked as leaders in the video conference; and determine a second number of the first participants.

[0100] In one embodiment of this disclosure, the third determining unit is specifically configured to: determine the initial formality of the video conference based on the conference description information of the video conference; and adjust the initial formality based on the first quantity and the second quantity to obtain the current formality of the video conference.

[0101] In one embodiment of this disclosure, one possible way to determine the initial formality of a video conference based on the conference description information is as follows: extract keywords from the conference description information of the video conference to obtain keyword extraction results; and determine the initial formality of the video conference based on the keyword extraction results.

[0102] In one embodiment of this disclosure, the device may further include: The third determination module is used to determine the third number of participants in the video conference participant list whose role is marked as leader; One possible way to determine the initial formality of a video conference based on keyword extraction results is to calculate the formality of the video conference based on the keyword extraction results and the third quantity, thereby obtaining the initial formality of the video conference.

[0103] In one embodiment of this disclosure, the second determining module 402 may include: The fourth determining unit is used to determine the background image area of ​​the current video frame based on the image area of ​​the second participant in the video conference contained in the current video frame, wherein the role of the second participant in the video conference is marked as a meeting host, ordinary user or leader; An abnormal behavior recognition unit is used to identify abnormal behavior in the background image region of the current video frame and obtain the confidence level of the abnormal behavior. The fifth determining unit is used to determine the area ratio of the target image region containing non-participants in the background image region when there are non-participants in the background image region. The sixth determining unit is used to determine the degree of interference of the background image region of the current video frame to the video conference based on the confidence level of abnormal behavior and the area ratio.

[0104] In one embodiment of this disclosure, the weighted summation module 403 is specifically used to: weight the current formality using a preset first weight; weight the interference using a preset second weight, wherein, if there is a participant marked as a leader in the participant list of the video conference, the first weight is greater than the second weight; and sum the weighted current formality and interference to obtain a comprehensive evaluation value of the video conference.

[0105] In one embodiment of this disclosure, the processing module 404 determines the corresponding background processing strategy based on the comprehensive evaluation value. One possible implementation is as follows: when the comprehensive evaluation value is greater than or equal to the first threshold and less than the second threshold, the corresponding background processing strategy is determined to be: blurring the face regions of non-participants in the background image region of the current video frame.

[0106] In one embodiment of this disclosure, the processing module 404 is further configured to: determine the corresponding background processing strategy as follows when the comprehensive evaluation value is greater than or equal to the second threshold: to perform virtualization processing on the background image region of the current video frame.

[0107] In one embodiment of this disclosure, when the comprehensive evaluation value is greater than or equal to the second threshold, the corresponding background processing strategy is determined as follows: obtain the corresponding virtual background image according to the conference theme of the video conference, and replace the background image area of ​​the current video frame with the virtual background image.

[0108] It should be noted that the foregoing description of the video background processing method embodiment also applies to the video background processing device of this embodiment, and will not be repeated here.

[0109] The video background processing apparatus provided in this embodiment determines the current formality of the video conference based on the current video frame and corresponding audio data of the speaking end; determines the background image area of ​​the current video frame and its interference level; performs a weighted summation of the current formality and interference level to obtain a comprehensive evaluation value for the video conference; determines a corresponding background processing strategy based on the comprehensive evaluation value; processes the background image area of ​​the current video frame according to the background processing strategy to obtain a target video frame; and sends the target video frame and audio data to each video conferencing terminal participating in the video conference. Thus, by combining the current video frame and corresponding audio data of the speaking end to determine the corresponding background processing strategy and automatically processing the background image area of ​​the current video frame according to the background processing strategy, the reliance on manual user operation is reduced, improving the intelligence level of the video conference and the user experience.

[0110] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0111] According to an embodiment of this disclosure, an electronic device is also provided, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the video background processing method disclosed in the embodiment of this disclosure.

[0112] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the video background processing method disclosed in this disclosure.

[0113] To implement the above embodiments, this disclosure also provides a computer program product.

[0114] The computer program product includes a computer program that, when executed by a processor, implements the video background processing method disclosed in the embodiments of this disclosure.

[0115] Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0116] like Figure 5 As shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 112 or a program loaded from memory 116 into random access memory (RAM) 113. The RAM 113 also stores various programs and data required for the operation of the electronic device 1000. The processor 111, ROM 112, and RAM 113 are interconnected via a bus 114. An input / output (I / O) interface 115 is also connected to the bus 114.

[0117] The following components are connected to I / O interface 115: memory 116 including hard disks, etc.; and communication section 117 including network interface cards such as local area network (LAN) cards, modems, etc., communication section 117 performs communication processing via a network such as the Internet; and driver 118 is also connected to I / O interface 115 as needed.

[0118] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 117. When the computer program is executed by processor 111, it performs the functions defined in the methods of this disclosure.

[0119] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processor 111 of the electronic device 1000 to perform the above-described method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0120] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0121] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0122] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video background processing method, characterized in that, The method includes: The current formality of the video conference is determined based on the current video frame of the speaker and the corresponding audio data of the current video frame. Determine the background image region of the current video frame and its interference level with the video conference; The overall evaluation value of the video conference is obtained by weighted summing of the current formality level and the interference level. Based on the comprehensive evaluation value, a corresponding background processing strategy is determined, and the background image region of the current video frame is processed according to the background processing strategy to obtain the target video frame. The target video frame and the audio data are sent to each video conferencing terminal participating in the video conference; The step of determining the current formality of the video conference based on the current video frame of the speaker and the corresponding audio data includes: The text content of the audio data is matched with a preset word library to determine the first number of words in the text content that belong to the preset word library; Determine a second number of participants in the current video frame who are the first participants in the video conference, wherein the role of the first participants in the video conference is marked as a leader; The current formality of the video conference is determined based on the first quantity and the second quantity; The determination of the interference level of the background image region of the current video frame on the video conference includes: Based on the image region of the second participant in the video conference contained in the current video frame, the background image region of the current video frame is determined, wherein the role of the second participant in the video conference is marked as a meeting host, ordinary user, or leader; Abnormal behavior is identified in the background image region of the current video frame to obtain the abnormal behavior confidence level; In the case where there are non-attendees in the background image area, determine the area ratio of the target image area containing the non-attendees in the background image area; Based on the confidence level of the abnormal behavior and the area ratio, the interference level of the background image region of the current video frame on the video conference is determined.

2. The method as described in claim 1, characterized in that, Determining the second number of participants in the video conference that are included in the current video frame includes: Determine the visual characteristics of the people in the current video frame; If the visual features match the features in a preset participant feature database, the person is designated as the first participant, wherein the participant feature database is formed based on the visual features of participants marked as leaders in the video conference. Determine the second number of the first participants.

3. The method as described in claim 2, characterized in that, Determining the current formality of the video conference based on the first quantity and the second quantity includes: The initial formality of the video conference is determined based on the conference description information. The initial formality level is adjusted based on the first quantity and the second quantity to obtain the current formality level of the video conference.

4. The method as described in claim 3, characterized in that, Determining the initial formality of the video conference based on the conference description information includes: Keyword extraction is performed on the meeting description information of the video conference to obtain keyword extraction results; Based on the keyword extraction results, the initial formality of the video conference is determined.

5. The method as described in claim 4, characterized in that, The method further includes: Determine the third number of participants in the participant list of the video conference whose role is marked as leader; The step of determining the initial formality of the video conference based on the keyword extraction results includes: Based on the keyword extraction results and the third quantity, the formality of the video conference is calculated to obtain the initial formality of the video conference.

6. The method as described in claim 1, characterized in that, The weighted summation of the current formality level and the interference level to obtain the comprehensive evaluation value of the video conference includes: The current level of formality is weighted using a preset first weight; The interference level is weighted using a preset second weight, wherein if there is a participant marked as a leader in the participant list of the video conference, the first weight is greater than the second weight; The weighted summation of the current formality and interference levels yields the comprehensive evaluation value of the video conference.

7. The method according to any one of claims 1-6, characterized in that, The step of determining the corresponding background processing strategy based on the comprehensive evaluation value includes: If the comprehensive evaluation value is greater than or equal to the first threshold and less than the second threshold, the corresponding background processing strategy is determined as follows: blur the face regions of non-participants in the background image region of the current video frame.

8. The method as described in claim 7, characterized in that, The step of determining the corresponding background processing strategy based on the comprehensive evaluation value also includes: If the comprehensive evaluation value is greater than or equal to the second threshold, the corresponding background processing strategy is determined as follows: virtualize the background image region of the current video frame.

9. The method as described in claim 8, characterized in that, If the comprehensive evaluation value is greater than or equal to the second threshold, the corresponding background processing strategy is determined as follows: obtain the corresponding virtual background image according to the conference theme of the video conference, and replace the background image area of ​​the current video frame with the virtual background image.

10. A video background processing device, characterized in that, The device includes: The first determining module is used to determine the current formality of the video conference based on the current video frame of the speaker in the video conference and the audio data corresponding to the current video frame. The second determining module is used to determine the background image region of the current video frame and its interference level with the video conference. The weighted summation module is used to perform a weighted summation of the current formality level and the interference level to obtain a comprehensive evaluation value of the video conference; The processing module is used to determine the corresponding background processing strategy based on the comprehensive evaluation value, and to process the background image region of the current video frame according to the background processing strategy to obtain the target video frame. The sending module is used to send the target video frame and the audio data to each video conferencing terminal participating in the video conference; Specifically, the first determining module is used to: match the text content of the audio data with a preset word library to determine a first number of words in the text content that belong to the preset word library; Determine a second number of participants in the current video frame who are the first participants in the video conference, wherein the role of the first participants in the video conference is marked as a leader; The current formality of the video conference is determined based on the first quantity and the second quantity; The second determining module is specifically used to: determine the background image area of ​​the current video frame based on the image area of ​​the second participant in the video conference contained in the current video frame, wherein the role of the second participant in the video conference is marked as a meeting host, ordinary user, or leader; Abnormal behavior is identified in the background image region of the current video frame to obtain the abnormal behavior confidence level; In the case where there are non-attendees in the background image area, determine the area ratio of the target image area containing the non-attendees in the background image area; Based on the confidence level of the abnormal behavior and the area ratio, the interference level of the background image region of the current video frame on the video conference is determined.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images