Video conference processing method and device, electronic equipment, storage medium and product

Abstract

The embodiment of the invention discloses a video conference processing method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: in a video conference proceeding process, shooting a picture of a whiteboard through an image acquisition device, and carrying out image detection on the picture through a first large model to obtain a sub-picture of the whiteboard; collecting audios of the video conference through a voice collection device; generating a conference summary of the video conference, wherein the conference summary comprises first content data representing whiteboard content information; wherein the first content data is generated in the following mode that audio information obtained based on the audio and the sub-picture are input into a second large model, the first content data is obtained based on an identification result output by the second large model, and the audio information serves as context information of the sub-picture. The first content data obtained through the method can completely and accurately describe information conveyed by the content written on the whiteboard as much as possible.

Description

Technical Field

[0001] This document relates to the field of communication technology, and in particular to a video conferencing processing method, device, electronic device, storage medium and product. Background Art

[0002] With the long-term development of video conferencing, deploying hardware equipment such as hosts, speakers, microphones, cameras, and installing professional video conferencing client software in conference rooms has gradually become a mature digital solution for conference rooms.

[0003] While users can conveniently conduct remote meetings using video conferencing software, some discussions and communications still require writing and sketching on a physical whiteboard. Due to poor image quality, remote participants face difficulties viewing the whiteboard's content via image capture devices. This often requires on-site participants to capture the whiteboard using their mobile phones so they can manually record the whiteboard's content during the conversation. However, this method often results in inaccurate whiteboard content and missing information due to factors such as poor image quality and sloppy handwriting. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a video conferencing processing method, device, electronic device, storage medium and product, which utilizes the object positioning and recognition capabilities of a large model to accurately extract the content written on a whiteboard from the whiteboard image captured by an image acquisition device, and utilizes the text converted from the audio collected by a voice acquisition device to identify and supplement the content written on the whiteboard, so that the obtained first content data can describe the information conveyed by the content written on the whiteboard as completely and accurately as possible, so that it can be used by participating users and reduce the workload of manual sorting.

[0005] In order to achieve the above objectives, the embodiments of this specification adopt the following technical solutions: In a first aspect, a method for processing a video conference is provided, comprising: During the video conference, an image of the whiteboard is captured by an image acquisition device, and image detection is performed on the image using the first large model to obtain a sub-image of the whiteboard; collecting audio of the video conference through a voice collection device; generating a meeting minutes of the video conference, wherein the meeting minutes include first content data representing whiteboard content information; The first content data is generated in the following manner: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

[0006] In a second aspect, a video conference processing device is provided, comprising: A first processing module is configured to capture a picture of the whiteboard using an image acquisition device during a video conference, and perform image detection on the picture using a first large model to obtain a sub-picture of the whiteboard; A collection module, configured to collect the audio of the video conference through a voice collection device; A first generating module, configured to generate a meeting minutes of the video conference, wherein the meeting minutes include first content data representing whiteboard content information; The first content data is generated in the following manner: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

[0007] According to a third aspect, an electronic device is provided, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video conference processing method provided in the first aspect.

[0008] In a fourth aspect, a computer-readable storage medium is provided. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the video conference processing method provided in the first aspect.

[0009] In a fifth aspect, a computer program product is provided, comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the video conferencing processing method provided in the first aspect.

[0010] According to the solution of the embodiment of this specification, during the video conference, the image of the whiteboard is captured by an image acquisition device, and the audio of the video conference is captured by a voice acquisition device. Since the audio contains a full record of the video conference, it can be used to assist in identifying the content of the whiteboard. Therefore, the powerful object positioning and recognition capability of the large model is utilized to accurately locate the sub-screen of the whiteboard from the screen, and then the audio information obtained based on the audio is used as the context information of the sub-screen and input into the second large model together with the sub-screen. The second large model identifies and supplements the content of the sub-screen, and obtains the first content data representing the whiteboard content information based on the recognition result output by the second large model. In the case of poor picture quality, the completeness and accuracy of the whiteboard content extraction are improved, so that the first content data can describe the information conveyed by the content written on the whiteboard as completely and accurately as possible, so that it can be used by participating users and the workload of manual sorting is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings: Figure 1 A flowchart of a method for processing a video conference provided in accordance with an embodiment of the present disclosure; Figure 2 A flowchart of a method for processing a video conference provided in accordance with another embodiment of the present disclosure; Figure 3 A schematic diagram of the structure of a video conferencing processing device provided in one embodiment of this specification; Figure 4 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0013] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. The term "responsive to" is used to indicate the conditions or states on which the operations performed depend. When the dependent conditions or states are met, the one or more operations performed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order of the multiple operations performed.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this document are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0015] It should be noted that the modifiers such as "one" and "multiple" mentioned in this document are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, they should be understood as "one or more".

[0016] The names of the messages or information exchanged between multiple devices in the embodiments of this document are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0017] As mentioned earlier, users can conveniently conduct remote meetings through video conferencing software, but some discussions and communications still require writing text and sketching on a physical whiteboard device. The whiteboard in the conference room may be fixed to the wall of the conference room, or a movable whiteboard may be configured. Therefore, the image captured by the camera in the conference room may be distorted or the text may be unclear. This makes it inconvenient for remote participants to view the content of the whiteboard through the image captured by the image acquisition device. This requires on-site participants to use their mobile phones to capture the image of the whiteboard so that they can manually organize and record the written content on the whiteboard during the conversation. However, when writing and drawing on the whiteboard, users often speak and draw at the same time, and the content is usually sloppy and brief. As a result, some information may be lost when the whiteboard is sorted out after the meeting because there is only a picture but not the actual conversation scene. In addition, the poor image quality of the whiteboard can also lead to inaccurate whiteboard content.

[0018] In view of this, an embodiment of the present specification proposes a method for processing video conferences. During the video conference, the image of the whiteboard is captured by an image acquisition device, and the audio of the video conference is captured by a voice acquisition device. Since the audio contains a full record of the video conference, it can be used to assist in identifying the content of the whiteboard. Therefore, the powerful object positioning and recognition capabilities of the large model are utilized to accurately locate the sub-screen of the whiteboard from the screen, and then the audio information obtained based on the audio is used as the context information of the sub-screen and input into the second large model together with the sub-screen. The second large model identifies and supplements the content of the sub-screen, and obtains the first content data representing the whiteboard content information based on the recognition result output by the second large model. In the case of poor picture quality, the completeness and accuracy of the whiteboard content extraction are improved, so that the first content data can describe the information conveyed by the content written on the whiteboard as completely and accurately as possible, so that it can be used by participating users and reduce the workload of manual sorting.

[0019] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0020] Please refer to Figure 1 , which is a flow chart of a method for processing a video conference provided in one embodiment of this specification, the method may include: S102 , during the video conference, a whiteboard image is captured by an image acquisition device, and the image is detected by a first large model to obtain a sub-image of the whiteboard.

[0021] The image acquisition device may include any hardware device capable of capturing, converting, or recording visual information, such as a camera connected to a video conference terminal, a professional camera deployed in a conference room, etc., and this specification is not limited to this embodiment. During the video conference, the image acquisition device captures the whiteboard in real time.

[0022] Image detection of a whiteboard image refers to a process of identifying and locating the whiteboard from the whiteboard image.

[0023] The first large model in the embodiment of this specification has the ability to process multimodal information such as images, audio, and text. The specific type of the large model can be selected according to actual needs, such as a multimodal model, etc., and this embodiment of this specification does not limit this. The object positioning ability of the traditional image processing algorithm is weak, and it can usually only identify the approximate position of the whiteboard in the picture, so manual intervention and adjustment are required, which is time-consuming and labor-intensive. The large model has a strong ability to locate objects. By performing image detection on the whiteboard picture through the first large model, the position of the whiteboard in the picture can be accurately located, thereby accurately obtaining the sub-picture of the whiteboard without manual intervention.

[0024] In one implementation, features are extracted from the whiteboard image through the first large model and converted into high-dimensional feature representations. Based on this, the whiteboard is classified and judged to determine its position in the image. The area where the whiteboard is located is then obtained and used as a sub-image of the whiteboard.

[0025] In another implementation, the above S102 includes the following steps: S1022: Reduce the resolution of the image to obtain a low-resolution image.

[0026] S1024: Perform image detection on the low-resolution image using the first large model to obtain first position information of the whiteboard in the low-resolution image.

[0027] For example, the first large model extracts features from the low-resolution image, converts them into high-dimensional feature representations, and uses these features to classify the whiteboard and determine first position information of the whiteboard in the low-resolution image. The first position information may include the position coordinates of key points of the whiteboard in the low-resolution image.

[0028] Key points can be selected based on actual needs. For example, key points can include the four corners of a whiteboard (top left, top right, bottom left, and bottom right) to determine the whiteboard's bounding box. Alternatively, key points can include the whiteboard's center or diagonal corners for quick calculation of the whiteboard's position. Another example is key points that can include feature points on the whiteboard's border to address partially occluded or non-rectangular whiteboard scenarios.

[0029] S1026 : Determine second position information of the whiteboard in the picture based on the first position information and the zoom ratio between the low-resolution picture and the picture.

[0030] The second position information may include position coordinates of key points of the whiteboard in the picture.

[0031] When the resolution of the copied image is reduced, the resulting low-resolution image will be larger than the image captured by the image acquisition device (also known as the original image), and the position of the whiteboard in the low-resolution image will also change. To accurately position the whiteboard in the original image, the position of the whiteboard in the low-resolution image must be mapped back to the original image based on the scaling ratio.

[0032] The scaling ratio refers to the size ratio between the original image and the low-resolution image, which can include horizontal scaling ratio and vertical scaling ratio. For example, if the size of the original image is W×H and the size of the low-resolution image is W'×H', the horizontal scaling ratio is W / W' and the vertical scaling ratio is H / H'.

[0033] Assume that the first position of the whiteboard in the low-resolution image is (x1', y1', x2', y2'), where (x1', y1') represents the coordinates of the upper left corner of the whiteboard in the low-resolution image, and (x2', y2') represents the coordinates of the lower right corner of the whiteboard in the low-resolution image. Based on the zoom ratio, the second position of the whiteboard in the original image (x1, y1, x2, y2) can be calculated using the following formula: x1 = x1' × horizontal zoom ratio, y1 = y1' × vertical zoom ratio, x2 = x2' × horizontal zoom ratio, y2 = y2' × vertical zoom ratio.

[0034] S1028: Obtain a sub-image of the whiteboard from the image based on the second position information.

[0035] As an example, the area of ​​the whiteboard in the picture may be located based on the second position information, and the area may be cropped out from the whiteboard to obtain a sub-picture of the whiteboard.

[0036] As another example, a whiteboard might be fixed to a conference room wall, or a separate, movable whiteboard might be installed. This often results in perspective distortion in the image captured by the image acquisition device. Directly cropping a sub-image of the whiteboard from this image would affect the subsequent recognition accuracy of the whiteboard's content. Therefore, the image captured by the image acquisition device is first corrected using perspective transformation, and then the sub-image of the whiteboard is cropped from this corrected image, thereby improving the recognition accuracy of the whiteboard's content.

[0037] Specifically, the picture is perspective transformed based on the position coordinates of the key points in the picture to obtain a corrected picture, and the position coordinates of the key points in the corrected picture are determined; based on the position coordinates of the key points in the corrected picture, a sub-picture of the whiteboard is cropped from the corrected picture.

[0038] A perspective transformation is a geometric transformation that projects an image from one perspective to another. In the embodiments of this specification, given the coordinates of the four corner points of a whiteboard within the image and a preset perspective transformation matrix, a perspective transformation can be performed on the image, mapping each corner point from one position to another to improve image distortion and produce a corrected image. The positions to which the four corner points are mapped are the positions of the key points within the corrected image, and the coordinates of the key points within the corrected image are then obtained.

[0039] Afterwards, the area where the whiteboard is located in the correction image is determined according to the position coordinates of the key points in the correction image, and the area is cropped from the correction image as a sub-image of the whiteboard.

[0040] In the second implementation method described above, since the resolution of the images captured by the image acquisition device is relatively high in some conference scenarios, as the number of images captured by the image acquisition device continues to increase during the video conference, if the first large model directly performs image detection on the high-resolution images, it will take a long time and the overhead will increase exponentially with the resolution of the processed images, making it unsuitable for real-time video image processing. To this end, by reducing the resolution of the images captured by the image acquisition device to obtain a low-resolution image, the first large model performs image detection on the low-resolution image and obtains the position of the whiteboard in the image captured by the image acquisition device based on this, thereby obtaining a sub-image of the whiteboard. This can reduce the amount of calculation of the large model, thereby improving the calculation speed and reducing the overhead, thereby better meeting the real-time image processing needs during the video conference.

[0041] The above describes some implementations of performing image detection on a screen using the first large model. Of course, it should be understood that image detection on a screen can also be achieved through other methods, which are not limited in this specification.

[0042] S104: Collect the audio of the video conference through a voice collection device.

[0043] The voice acquisition device may include various hardware devices capable of capturing, converting, or recording sound signals, such as a microphone on a conference terminal connected to a video conference, a professional recording device deployed in a conference room, etc., and this specification is not limited to this. During the video conference, the voice acquisition device collects the audio of the video conference in real time.

[0044] S106: Generate minutes of the video conference.

[0045] The meeting minutes include first content data representing the whiteboard content information. The first content data is generated by: inputting the audio information and sub-picture obtained based on the audio into the second large model; and obtaining the first content data based on the recognition result output by the second large model, wherein the audio information serves as the context information of the sub-picture. Among them, the second large model has the ability to process multimodal information such as images, audio, and text. The specific type of the large model can be selected according to actual needs, such as a multimodal model, etc., and this embodiment of the present specification does not limit this. The second large model and the first large model can be the same large model or different large models.

[0046] In some examples, the meeting minutes may also include a meeting summary generated based on the audio.

[0047] In the implementation of this specification, the meeting minutes can be generated during the video conference or after the video conference ends.

[0048] The audio information obtained based on the audio can include various appropriate content, which is not limited in the embodiments of this specification. In some examples, the audio information obtained based on the audio can include the audio itself or other information obtained by analyzing the audio, extracting features, etc. In other examples, the audio information obtained based on the audio can include any of the following: recognized text obtained by performing speech recognition on the audio, or a meeting summary obtained based on the recognized text.

[0049] In the application, various appropriate speech recognition technologies can be used to convert audio into recognized text. For example, the audio can be converted into recognized text through the second largest model, or an external online speech-to-text tool can be called to convert the audio into recognized text. After obtaining the text, various appropriate summary extraction or generation technologies can be used to summarize the recognized text to obtain a meeting summary of the video conference. In one implementation, summary extraction technology is used to extract key information from the recognized text, such as keywords representing entities such as time, place, name, event, and the relationship between entities, etc., and the key information is filled into a preset template to obtain a meeting summary of the video conference.

[0050] Since the audio content includes the entire record of the video conference, it can be used to assist in identifying the content of the whiteboard. Therefore, the powerful object positioning and recognition capabilities of the large model are utilized to use the audio information obtained based on the audio as the context information of the sub-screen to identify and supplement the content of the sub-screen. In the case of poor picture quality, the completeness and accuracy of the whiteboard content extraction are improved, so that the first content data can describe the information conveyed by the content written on the whiteboard as completely and accurately as possible, so that it can be used by participating users and reduce the workload of manual sorting. In the above S106, inputting the audio information and the sub-picture obtained based on the audio into the second largest model can be achieved in various ways.

[0051] The recognition result output by the second largest model may include content data representing whiteboard content information extracted from the sub-picture.

[0052] In one implementation, the content data output by the second largest model may be directly determined as the first content data.

[0053] Specifically, the audio information and sub-picture obtained based on the audio are input into the second largest model, including: instructing the second largest model to use the audio information as the context information of the sub-picture, performing image recognition on the sub-picture, and obtaining second content data representing the whiteboard content information; and, formatting the second content data based on the grammatical rules of the standard format of the preset graphic library to obtain the first content data.

[0054] Specifically, the audio information obtained based on the audio may include any one of the following: a recognized text obtained by performing speech recognition on the audio, or a meeting summary obtained based on the recognized text.

[0055] The preset graphic library can be set according to actual needs and is not limited in this embodiment of the present invention. For example, the preset graphic library can be Mermaid, which is a text-based icon generation library that supports various types of charts, such as flow charts, sequence diagrams, Gantt charts, class diagrams, state diagrams, etc. These charts can intuitively reflect the content written on the whiteboard.

[0056] For example, the following prompt words are generated based on the meeting summary: Please carefully analyze the whiteboard content based on the meeting summary and output the data in the standard format of the Mermaid graphics library.

[0057] Meeting Summary: {{Below I'll briefly illustrate the app startup process. After the app starts, it will first determine whether it has been bound. If not, it will jump to the binding page. Otherwise, it will continue to determine whether the meeting is disabled. If disabled, it will display the disabled status. If not, it will continue to determine whether the current meeting room is in progress and jump to the meeting sign-in page or the meeting page respectively. The above-mentioned prompt words and sub-pictures are input into the big model, and the big model uses natural language understanding capabilities to perform semantic analysis on the prompt words, and uses powerful image recognition capabilities to identify the sub-pictures in combination with the semantics of the semantic prompt words, and extracts the app startup flowchart on the whiteboard (i.e., the second content data), and then converts the app startup flowchart into the standard format of the Mermaid graphics library to obtain the first content data.

[0058] In another implementation, the content data output by the second large model can be further processed to obtain first content data. Specifically, the audio information and sub-image obtained based on the audio are input into the second large model, including: instructing the second large model to use the audio information as context information for the sub-image, perform image recognition on the sub-image, and obtain second content data representing the whiteboard content information. Accordingly, the first content data is obtained based on the recognition results output by the second large model, including: converting the second content data into a format based on grammatical rules of a standard format of a preset graphic library to obtain the first content data.

[0059] The above describes some implementations of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, which are not limited in the embodiments of this specification.

[0060] In another embodiment of the present specification, improperly designed whiteboard installation positions or image acquisition device placement may result in unclear images captured by the image acquisition device, which in turn may lead to unclear sub-images of the whiteboard. Directly recognizing the whiteboard content from the sub-images will affect recognition accuracy. Therefore, prior to S106, the sub-images are further enhanced, and the second large model then extracts the first content data from the enhanced sub-images.

[0061] Specifically, the image quality enhancement processing of the sub-picture includes at least one of the following methods: Method 1: Remove shadows in the sub-image, such as human reflections, changes in light and darkness, etc.

[0062] A prompt word and a sub-image indicating shadow removal can be input into the second largest model, and the powerful image processing capabilities of the second largest model can be used to remove the shadow in the sub-image. Alternatively, since shadows typically have low luminance (V / Y) but small changes in chrominance (H / CbCr) in the HSV / YCbCr color space, image processing techniques can be used to convert the sub-image from RGB to HSV or YCbCr space, segment the shadow area based on the differences in the luminance channel (V / Y) and chrominance channel (H / CbCr), and perform linear adjustments to the pixel values ​​in the shadow area, such as increasing the brightness, to achieve the effect of removing the shadow in the sub-image.

[0063] Method 2: Enhance the contrast of the sub-image.

[0064] The second model can be fed with a prompt indicating enhanced contrast, along with the sub-image. Leveraging the second model's powerful image processing capabilities, the sub-image's contrast can be enhanced. Alternatively, the brightness and contrast values ​​of the pixels in the sub-image can be adjusted to increase the difference between pixel values ​​and enhance the contrast between light and dark in the sub-image, thereby enhancing the sub-image's contrast.

[0065] Method 3: Highlight the characters and lines in the sub-picture.

[0066] The second model can be fed with a prompt and a sub-image indicating highlighted characters and lines. Leveraging the powerful image processing capabilities of the second model, the characters and lines within the sub-image can be identified and highlighted. Alternatively, image processing technologies such as character recognition and line detection can be used to identify the characters and lines within the sub-image. These characters and lines can then be bolded or set to a color that contrasts sharply with the background, achieving the desired effect.

[0067] In another embodiment of the present specification, the meeting minutes also include a meeting summary generated based on the audio. After performing image quality enhancement processing on the sub-picture, the method further includes: inserting the sub-picture that has undergone image quality enhancement processing into the meeting summary.

[0068] The sub-image with enhanced image quality can be inserted into any position in the meeting summary, and can be set according to actual needs, such as the end of the meeting summary, the position of the text in the meeting summary that semantically matches the sub-image, etc. This embodiment of the specification does not limit this.

[0069] This will help improve the information integrity of the meeting summary, allowing users who view the meeting summary to quickly understand the key points and details of the video conference discussion, and provide reliable data support for users to make decisions and review.

[0070] In another embodiment of the present specification, after the above S106, the method further includes: generating a reference picture of the whiteboard based on the first content data; and sending the reference picture to a conference client of the users participating in the video conference.

[0071] The reference image is a picture used to describe the first content data. The prompt word indicating the generated image and the first content data can be input into the third model. Leveraging the third model's powerful semantic understanding and generation capabilities, the first content data is converted into a corresponding reference image. The reference image is then sent to the conference client of video conference participants, helping them quickly and easily understand the complete content of the video conference.

[0072] The third large model is capable of processing multimodal information such as images, audio, and text. The specific type of large model can be selected based on actual needs, such as a multimodal model, and is not limited in this embodiment. The third large model can be the same as the first and second large models, or different large models.

[0073] In another embodiment of the present specification, the meeting minutes further include a meeting summary generated based on the audio. After generating a reference screen of the whiteboard based on the first content data, the method further includes: inserting the reference screen into the meeting summary.

[0074] The reference picture can be inserted into any position in the meeting summary, and can be set according to actual needs, such as the end of the meeting summary, the position of the text in the meeting summary that semantically matches the first content data, etc. This embodiment of the specification does not limit this.

[0075] This will help improve the information integrity of the meeting summary, allowing users who view the meeting summary to quickly understand the key points and details of the video conference discussion, avoid omissions, and provide reliable data support for users to make decisions and review.

[0076] One or more embodiments of the present specification provide a method for processing video conferences. During the video conference, an image capture device is used to capture the image of a whiteboard, and a voice capture device is used to capture the audio of the video conference. Since the audio contains a full record of the video conference, it can be used to assist in identifying the content of the whiteboard. Therefore, the powerful object positioning and recognition capabilities of the large model are utilized to accurately locate the sub-screen of the whiteboard from the screen, and then the audio information obtained based on the audio is used as the context information of the sub-screen and input into the second large model together with the sub-screen. The second large model identifies and supplements the content of the sub-screen, and obtains first content data representing the whiteboard content information based on the recognition result output by the second large model. In the case of poor picture quality, the completeness and accuracy of the whiteboard content extraction are improved, so that the first content data can describe the information conveyed by the content written on the whiteboard as completely and accurately as possible for use by participating users, reducing the workload of manual sorting.

[0077] In order to facilitate the understanding of the video conferencing processing method provided in the embodiment of this specification, the following Figure 2 , using a specific embodiment to illustrate the video conferencing processing method provided in the embodiments of this specification.

[0078] like Figure 2 As shown, the video conference processing method provided in the embodiment of this specification includes three steps.

[0079] The first step: locate the sub-screen of the whiteboard.

[0080] After the video conference starts, the image acquisition device and the voice acquisition device are started, and the image acquisition device is used to capture the whiteboard image in real time, and the voice acquisition device is used to capture the audio of the video conference in real time.

[0081] For a whiteboard image, the image is copied and the resolution of the copied image is reduced to obtain a low-resolution image. Image detection is performed on the low-resolution image using the first large model to identify and locate the position of the whiteboard, thereby obtaining the positions of the key points of the whiteboard in the low-resolution image. The positions of the key points of the whiteboard in the image are determined based on the scaling ratio between the low-resolution image and the image. Furthermore, based on the positions of the key points of the whiteboard in the image, the image is corrected by performing a perspective transformation on the image, and the area where the whiteboard is located is cropped from the corrected image as a sub-image of the whiteboard.

[0082] For audio, convert the audio to text after the video conference ends.

[0083] The second step: enhance the image quality of the whiteboard sub-image.

[0084] The image quality enhancement processing for the sub-picture includes at least one of the following methods: removing shadows in the sub-picture, enhancing the contrast of the sub-picture, highlighting text and lines in the sub-picture, etc.

[0085] The third step: extracting the first content data.

[0086] On the one hand, the sub-picture after image quality enhancement is saved; on the other hand, a meeting summary of the video conference is generated based on the text converted from the audio.

[0087] Furthermore, the second large model uses the meeting summary as context information for the sub-screen, and extracts first content data representing the whiteboard content information from the sub-screen. For example, the second large model uses the meeting summary as context information for the sub-screen to identify the sub-screen, obtaining second content data representing the whiteboard content information, and then converts the second content data into the format of the first content data based on grammatical rules of a standard format in a preset graphic library.

[0088] Further, at least one of the following operations is performed: inserting the sub-image with enhanced image quality into the conference summary; generating a reference image of the whiteboard based on the first content data, and sending the reference image to the conference client of the participating users; inserting the reference image into the conference summary.

[0089] It is worth noting that the video conferencing processing method provided in the embodiments of this specification can be independently executed by the conference terminal. In this case, a large model suitable for the conference terminal is deployed on the conference terminal, which can process multimodal information such as images, audio, and text.

[0090] Alternatively, the above-mentioned video conference processing method can also be executed by a server. In this case, a large model is deployed on the server, which is capable of processing multimodal information such as images, audio, and text. Specifically, an image acquisition device can capture the whiteboard image in real time and transmit it to the server, and a voice acquisition device can capture the audio of the video conference in real time and transmit it to the server. Afterwards, the server performs image detection on the received image using the first large model to obtain a sub-image of the whiteboard, and uses the second large model to use the audio information obtained based on the audio as the context information of the sub-image, and extracts the first content data representing the content information of the whiteboard from the sub-image.

[0091] Alternatively, the above-described video conferencing processing method can also be performed jointly by the conference terminal and the server. In this case, a large model is deployed on the server. During the video conference, the conference terminal located in the conference room uses its camera to capture the whiteboard image in real time and transmits it to the server, and uses its microphone to capture the video conference audio in real time and transmit it to the server. The server then performs image detection on the received image using the first large model to obtain a sub-image of the whiteboard. Using the second large model, the server uses the audio information obtained based on the audio as context information for the sub-image and extracts first content data representing the whiteboard content from the sub-image.

[0092] It should be understood that the conference terminal can be any terminal device equipped with a conference client, such as a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, etc., and this specification is not limited to this embodiment. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0093] In addition, with the above Figure 1 Corresponding to the video conference processing method shown, the embodiment of this specification also provides a video conference processing device. Figure 3 3 is a structural diagram of a video conferencing processing device 300 provided in an embodiment of this specification, including: a first processing module 310, a collection module 320 and a first generation module 330.

[0094] The first processing module 310 is used to capture a picture of the whiteboard using an image acquisition device during a video conference, and perform image detection on the picture using a first large model to obtain a sub-picture of the whiteboard.

[0095] The acquisition module 320 is configured to acquire the audio of the video conference through a voice acquisition device.

[0096] The first generating module 330 is used to generate the minutes of the video conference, wherein the minutes include first content data representing the whiteboard content information; The first content data is generated by: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

[0097] The video conference processing device provided in the embodiment of this specification captures the image of the whiteboard through an image acquisition device and captures the audio of the video conference through a voice acquisition device during the video conference. Since the audio contains a full record of the video conference, it can be used to assist in identifying the content of the whiteboard. Therefore, the powerful object positioning and recognition capability of the large model is utilized to accurately locate the sub-screen of the whiteboard from the screen, and then the audio information obtained based on the audio is used as the context information of the sub-screen and input into the second large model together with the sub-screen. The second large model recognizes and supplements the content of the sub-screen, and obtains the first content data representing the whiteboard content information based on the recognition result output by the second large model. In the case of poor picture quality, the completeness and accuracy of the whiteboard content extraction are improved, so that the first content data can describe the information conveyed by the content written on the whiteboard as completely and accurately as possible, so that it can be used by participating users and reduce the workload of manual sorting.

[0098] In another embodiment, the first processing module includes: An adjustment submodule, configured to reduce the resolution of the image to obtain a low-resolution image; a detection submodule, configured to perform image detection on the low-resolution image using a first large model to obtain first position information of the whiteboard in the low-resolution image; a determination submodule, configured to determine second position information of the whiteboard in the picture based on the first position information and a zoom ratio between the low-resolution picture and the picture; A cropping submodule is configured to obtain a sub-image of the whiteboard from the image based on the second position information.

[0099] In another embodiment, the second position information includes position coordinates of key points of the whiteboard in the picture; The cutting submodule is used to: Performing perspective transformation on the picture based on the position coordinates of the key point in the picture to obtain a corrected picture, and determining the position coordinates of the key point in the corrected picture; Based on the position coordinates of the key points in the correction picture, a sub-picture of the whiteboard is cut out from the correction picture.

[0100] In another embodiment, inputting the audio information obtained based on the audio and the sub-picture into the second large model includes: instructing the second large model to use the audio information as context information of the sub-picture, perform image recognition on the sub-picture, and obtain second content data representing whiteboard content information; and The second large model is instructed to perform format conversion on the second content data based on grammatical rules of a standard format of a preset graphic library to obtain the first content data.

[0101] In another embodiment, the audio information obtained based on the audio includes any one of the following: a recognized text obtained by performing speech recognition on the audio, or a meeting summary obtained based on the recognized text.

[0102] In another embodiment, the video conference processing device further includes: The image quality enhancement module is configured to perform at least one of the following image quality enhancement processes on the sub-image: removing shadows from the sub-picture; enhancing the contrast of the sub-picture; The characters and lines in the sub-picture are highlighted.

[0103] In another embodiment, the meeting minutes further include a meeting summary generated based on the audio; The video conference processing device further includes: The first inserting module is configured to insert the sub-picture having undergone image quality enhancement processing into the conference summary.

[0104] In another embodiment, the video conference processing device further includes: a second generating module, configured to generate a reference picture of the whiteboard based on the first content data; The sending module is used to send the reference picture to the conference client of the users participating in the video conference.

[0105] In another embodiment, the meeting minutes further include a meeting summary generated based on the audio; The video conference processing device further includes: The second inserting module is configured to insert the reference picture into the conference summary.

[0106] Obviously, the video conferencing processing device of the embodiment of this specification can be used as the above Figure 1 The execution subject of the video conference processing method shown in FIG. Figure 1 Since the principle is the same, the functions realized are not described here.

[0107] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of this specification. Figure 4At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0108] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0109] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0110] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a video conferencing processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: The first content data is generated in the following manner: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

[0111] The above is as in this manual Figure 1The methods performed by the video conferencing processing device disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits within the processor or by software instructions. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this specification can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of this specification can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0112] It should be understood that the electronic device of the embodiment of this specification can realize the processing device of the video conference in Figure 1 Since the principles are the same, the embodiments of this specification will not be described in detail here.

[0113] Of course, in addition to software implementation, the electronic device in this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0114] The embodiment of this specification also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple application programs, the electronic device can execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations: The first content data is generated in the following manner: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

[0115] An embodiment of this specification also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the video conference processing method provided in the first aspect.

[0116] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included in the scope of protection of this specification.

[0118] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0121] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A method for processing a video conference, characterized in that: include: During the video conference, an image of the whiteboard is captured by an image acquisition device, and image detection is performed on the image using the first large model to obtain a sub-image of the whiteboard; collecting audio of the video conference through a voice collection device; generating a meeting minutes of the video conference, wherein the meeting minutes include first content data representing whiteboard content information; The first content data is generated in the following manner: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

2. The method according to claim 1, characterized in that The step of performing image detection on the screen by using the first large model to obtain the sub-screen of the whiteboard includes: reducing the resolution of the picture to obtain a low-resolution picture; Performing image detection on the low-resolution image using a first large model to obtain first position information of the whiteboard in the low-resolution image; determining second position information of the whiteboard in the picture based on the first position information and a zoom ratio between the low-resolution picture and the picture; A sub-image of the whiteboard is obtained from the image based on the second position information.

3. The method according to claim 2, characterized in that The second position information includes the position coordinates of the key points of the whiteboard in the picture; The obtaining the sub-picture of the whiteboard from the picture based on the second position information includes: Performing perspective transformation on the picture based on the position coordinates of the key point in the picture to obtain a corrected picture, and determining the position coordinates of the key point in the corrected picture; Based on the position coordinates of the key points in the correction picture, a sub-picture of the whiteboard is cut out from the correction picture.

4. The method according to claim 1, wherein The step of inputting the audio information obtained based on the audio and the sub-picture into the second large model comprises: instructing the second large model to use the audio information as context information of the sub-picture, perform image recognition on the sub-picture, and obtain second content data representing whiteboard content information; and The second large model is instructed to perform format conversion on the second content data based on grammatical rules of a standard format of a preset graphic library to obtain the first content data.

5. The method according to claim 4, characterized in that The audio information obtained based on the audio includes any one of the following: a recognized text obtained by performing speech recognition on the audio, or a meeting summary obtained based on the recognized text.

6. The method according to any one of claims 1 to 5, characterized in that Before generating the minutes of the video conference, the method further includes: Perform at least one of the following image quality enhancement processes on the sub-picture: removing shadows from the sub-picture; enhancing the contrast of the sub-picture; The characters and lines in the sub-picture are highlighted.

7. The method according to claim 6, characterized in that The meeting minutes also include a meeting summary generated based on the audio; After performing image quality enhancement processing on the sub-picture, the method further includes: The sub-picture after image quality enhancement is inserted into the conference summary.

8. The method according to any one of claims 1 to 5, characterized in that After generating the minutes of the video conference, the method further includes: generating a reference picture of the whiteboard based on the first content data; The reference picture is sent to the conference client of the users participating in the video conference.

9. The method according to claim 8, characterized in that The meeting minutes also include a meeting summary generated based on the audio; After generating the reference picture of the whiteboard based on the first content data, the method further includes: The reference picture is inserted into the meeting summary.

10. A video conference processing device, characterized in that: include: A first processing module is configured to capture a picture of the whiteboard using an image acquisition device during a video conference, and perform image detection on the picture using a first large model to obtain a sub-picture of the whiteboard; A collection module, configured to collect the audio of the video conference through a voice collection device; A first generating module, configured to generate a meeting minutes of the video conference, wherein the meeting minutes include first content data representing whiteboard content information; The first content data is generated in the following manner: inputting the audio information obtained based on the audio and the sub-picture into a second large model; and The first content data is obtained based on the recognition result output by the second large model, wherein the audio information serves as context information of the sub-picture.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video conference processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video conference processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the video conference processing method according to any one of claims 1 to 9.

Images