Conference data transmission method and device, computer equipment and readable storage medium

Abstract

The invention provides a conference data transmission method and device, computer equipment and a computer readable storage medium. The conference data transmission method comprises the following steps: acquiring a screen display picture on the display equipment in a current conference; acquiring a scene shooting picture captured by image acquisition equipment of the display equipment in the current conference; carrying out fusion coding on the screen display picture and the scene shooting picture to obtain video coding content of the current conference; and sending the video coding content to a screen projection sending end on the electronic equipment, so that the screen projection sending end decodes the screen display picture and the scene shooting picture based on the video coding content and presents the screen display picture and the scene shooting picture to conference software running in the electronic equipment. According to the embodiment of the invention, the audio or video coding times in the online and offline mixed conference scene can be reduced, and the problem that the delay or jamming phenomenon easily occurs in the online and offline mixed conference scene is avoided.

Description

Conference data transmission method, device, computer equipment and readable storage medium Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a conference data transmission method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Meetings (commonly referred to as corporate meetings, teaching, discussions, brainstorming and other multi-person interactive scenarios) are a common form of communication in life or work. With the rapid development of information technology, the limitations of traditional offline meetings based on listening and watching are becoming increasingly apparent. Electronic meeting tools (such as meeting software) have become the trend of the times.

[0003] In the existing technology, on the one hand, local information is projected onto the screen so that all local participants offline can receive auditory and visual information; on the other hand, local information is synchronized to online remote participants in a screen-sharing manner through conference software so that online remote participants can receive auditory and visual information; thereby realizing a mixed online and offline conference scenario, allowing offline local participants and online remote participants to synchronize auditory and visual information for information transmission and realize interactive communication.

[0004] However, during the actual research and development process, the inventors of the embodiments of the present application discovered that: since the screen projection function and the online conference function are implemented by different software or hardware, the synchronization of visual information requires multiple video encodings, resulting in the occupation of more CPU (Central Processing Unit) or GPU (Graphic Processing Unit) computing resources, which in turn leads to resource competition between the screen projection function and the online conference function, resulting in delays or freezes in online and offline mixed conference scenarios.

[0005] Summary of the Invention

[0006] In view of this, the present application provides a conference data transmission method, device, computer equipment and computer-readable storage medium, which can reduce the number of video encoding times in online and offline mixed conference scenarios, and avoid the problems of delays or jamming that are prone to occur in online and offline mixed conference scenarios.

[0007] In a first aspect, the present application provides a conference data transmission method, which is applied to a display device in a conference system, wherein the conference system includes a display device and an electronic device. The conference data transmission method includes:

[0008] Obtain the screen display image on the display device in the current meeting;

[0009] Acquire a scene camera image captured by an image acquisition device of the display device in the current conference;

[0010] Fusion encoding is performed on the screen display image and the scene camera image to obtain the video encoding content of the current meeting;

[0011] The video encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed on the electronic device or the screen projector electrically connected to the electronic device.

[0012] In addition, the present application provides a conference data transmission method, which is applied to a display device in a conference system, wherein the conference system includes a display device and an electronic device. The conference data transmission method includes:

[0013] Obtain the scene audio captured by the audio capture device of the display device in the current conference;

[0014] Obtain the playback audio of the display device in the current conference;

[0015] Fusion-encoding the scene audio and the playback audio to obtain audio encoding content of the current meeting;

[0016] The audio encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the scene audio and the playback audio based on the audio encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed on the electronic device or the screen projector electrically connected to the electronic device.

[0017] In addition, the present application provides a conference data transmission method, which is applied to a screen projector in a conference system, wherein the conference system includes a display device, an electronic device, and a screen projector communicatively connected to the electronic device. The conference data transmission method includes:

[0018] Receiving video encoding content sent by the display device, wherein the video encoding content is obtained by fusion encoding a screen display image on the display device and a scene camera image captured by an image acquisition device of the display device;

[0019] Decoding the video encoding content to obtain decoded video content of the video encoding content;

[0020] The decoded video content is transmitted to the conference software running in the electronic device connected to the screen projector.

[0021] In addition, the present application provides a conference data transmission method, which is applied to a screen projector in a conference system, wherein the conference system includes a display device, an electronic device, and a screen projector communicatively connected to the electronic device. The conference data transmission method includes:

[0022] Receiving audio encoding content sent by the display device, wherein the audio encoding content is obtained by fusing and encoding scene audio captured by an audio acquisition device of the display device and playback audio of the display device;

[0023] Decoding the audio encoding content to obtain decoded audio content of the audio encoding content;

[0024] The decoded audio content is transmitted to the conference software running in the electronic device connected to the screen projector.

[0025] In a second aspect, the present application further provides a conference data transmission device, the conference data transmission device comprising:

[0026] An acquisition unit, configured to acquire the screen display image on the display device in the current conference;

[0027] The acquisition unit is further configured to acquire a scene camera image captured by an image acquisition device of the display device;

[0028] An encoding unit, configured to fuse and encode the screen display image and the scene camera image to obtain the video encoding content of the current meeting;

[0029] A sending unit is used to send the video encoding content to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed on the electronic device or the screen projector electrically connected to the electronic device.

[0030] In a third aspect, the present application further provides a computer device, comprising:

[0031] one or more processors;

[0032] Memory; and

[0033] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the above-mentioned conference data transmission method.

[0034] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the conference data transmission method.

[0035] In the present application, the screen display image on the display device and the scene camera image captured by the image acquisition device of the display device are fused and encoded by the display device, and then sent to the screen projector or screen projection software (i.e., the screen projection sending end) on the electronic device, and presented to the conference software running in the electronic device through the screen projection sending end; on the one hand, since only the process of fusion encoding of the screen display image on the display device and the scene camera image captured by the image acquisition device of the display device and the encoding process of local screen projection need to be completed once, local screen projection, remote camera sharing and remote screen sharing can be realized; there is no need to perform video encoding for local screen projection, remote camera sharing and remote screen sharing respectively, which reduces the number of video encodings in the online and offline mixed conference scenario. In this way, the problem that the synchronization of visual information requires multiple video encodings due to the fact that the screen projection function and the online conference function are implemented by different software (or hardware) in the related technology can be solved, the CPU or GPU computing resources occupied can be reduced, and the resource competition problem between the screen projection function and the online conference function can be reduced, and the problem of delay or freezing in the online and offline mixed conference scenario can be avoided to a certain extent.

[0036] Secondly, since the screen display image on the display device and the scene camera image captured by the image acquisition device of the display device are captured by the display device and fused and encoded to be shared with the online remote participants, when the display device is not running the conference software, the presentation content of the display device 104 can be shared with the remote participants at the same time, avoiding the problem that the presentation content of the display device cannot be shared with the remote participants when it does not come from an electronic device that realizes the online conference function, thereby improving the online remote participants' visual conference experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] FIG1 is a schematic diagram of a scenario of a conference system provided in an embodiment of the present application;

[0039] FIG2 is a schematic diagram of another scenario of the conference system provided in an embodiment of the present application;

[0040] FIG3 is a flow chart of an embodiment of a conference data transmission method provided in an embodiment of the present application;

[0041] Figure 4 is a schematic diagram of a scenario of a hybrid online and offline meeting;

[0042] FIG5 is a schematic diagram of a scenario of screen projection of a current conference provided in an embodiment of the present application;

[0043] FIG6 is a schematic diagram illustrating the content of screen projection after drawing provided in an embodiment of the present application;

[0044] FIG7 is a schematic diagram illustrating a picture splicing format provided in an embodiment of the present application;

[0045] FIG8 is a schematic diagram illustrating a process of remote sharing of visual information provided in an embodiment of the present application;

[0046] FIG9 is a schematic diagram illustrating a remote sharing process of auditory information provided in an embodiment of the present application;

[0047] FIG10 is a schematic diagram of a scenario of remote sharing of audio-visual information provided in an embodiment of the present application;

[0048] FIG11 is a schematic diagram of another scenario of remote sharing of audio-visual information provided in an embodiment of the present application;

[0049] FIG12 is a schematic structural diagram of an embodiment of a conference data transmission device according to an embodiment of the present application;

[0050] FIG13 is a schematic diagram of the structure of a computer device according to an embodiment of the present application.

[0051] Implementation Methods of the Application

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0053] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present application, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0054] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that one of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0055] The embodiments of the present application provide a conference data transmission method, apparatus, computer equipment, and computer-readable storage medium. Each embodiment is described in detail below.

[0056] First, the conference data transmission method provided in the embodiment of the present application can be applied to the conference system provided in the embodiment of the present application. In order to better understand the conference data transmission method provided in the embodiment of the present application, the conference system provided in the embodiment of the present application is first introduced in conjunction with Figures 1 and 2 (wherein Figure 1 is a scene diagram of the conference system provided in the embodiment of the present application, and Figure 2 is another scene diagram of the conference system provided in the embodiment of the present application). The conference system in this embodiment includes at least an electronic device 101 and a display device 104, wherein the electronic device 101 can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a handheld computer, a desktop computer or a personal digital assistant (PDA). Among them, the display device 104 can be an interactive tablet, a large display screen, a projection box, an audio and video conferencing terminal, a TV, etc. Among them, a projection connection between the electronic device 101 and the display device 104 can be established through the projection software installed in the electronic device 101, or a projection connection between the electronic device 101 and the display device 104 can be established through the projection device 103 to realize the projection of the audio and video data content displayed on the current screen of the electronic device 101 to the display device 104.

[0057] As shown in Figure 1, in some embodiments, the conference system may include an electronic device 101 (such as a participant's computer with third-party conference software installed), a connecting cable 102, a screen projector 103 and a display device 104, and a communication connection is established between the electronic device 101, the screen projector 103 and the display device 104. Among them, the electronic device 101 and the screen projector 103 can be connected through the connecting cable 102, and the screen projector 103 and the display device 104 can be a wireless communication connection (such as establishing a communication connection between the screen projector 103 and the display device 104 through WIFI, Bluetooth, etc.), or a wired communication connection (such as establishing a communication connection between the screen projector 103 and the display device 104 through an HDMI cable). Among them, the connecting cable 102 can be detachably connected to the screen projector 103, or it can be integrally formed with the screen projector 103.

[0058] As shown in Figure 2, in other embodiments, the conference system may include an electronic device 101 (such as a smart phone installed with third-party conference software and screen projection software) and a display device 104. The electronic device 101 can establish a communication connection with the display device 104 through the screen projection software.

[0059] Meetings (in this embodiment, unless otherwise specified, "meetings" include multi-person interactive scenarios such as corporate meetings, teaching, discussions, brainstorming, etc.) are a commonly used communication method in life or work. It is inevitable that there will be hybrid meetings that combine online and offline. Local offline participants and remote online participants communicate with each other through visual and auditory information transmission. For example, as shown in Figure 1 or Figure 2, in a hybrid meeting scenario that combines online and offline, on the one hand, in order to enable local participants to share auditory and visual information, a screen projector 103 or screen projection software running on one of the local participants' electronic devices 101 (such as the computer of participant A who serves as the host) can be used to project the visual content (such as pictures or documents, collectively referred to as videos) of participant A's computer to the display device 104 for display, and the auditory content (such as audio) of participant A's computer can be transmitted to the display device 104 and output through the speaker of the display device 104. On the other hand, to enable remote participants to share local auditory information (such as the recorded sound from the offline microphone and the auditory content on participant A's computer) and visual information (such as the recorded content from the offline camera and the visual content on participant A's computer), third-party conferencing software can be used on the computer of one of the local participants (such as participant A, the host) to share auditory and visual information with the remote participant (such as participant C). To do this, the computer of local participant A needs to implement the following functions:

[0060] 1) Screen projection: The screen projection function requires the implementation of desktop screen recording encoding (generally 1080p~4k) and speaker recording encoding;

[0061] 2) Remote video conferencing function: Remote video conferencing function requires camera encoding (generally 720p~4k) and microphone encoding;

[0062] 3) Remote desktop sharing function: The remote desktop sharing function requires the implementation of local desktop recording encoding (generally 1080p~4k) and speaker recording encoding.

[0063] However, since the screen projection function and the online conference function (the online conference function includes the remote video conferencing function and the remote desktop sharing function) are implemented by different software or hardware (for example, the screen projection function is implemented by the screen projection device 103 or screen projection software on the local participant A's computer, and the online conference function is implemented by the third-party conference software running on the local participant A's computer); therefore, in order to synchronize the visual information of the online and offline participants, at least three video encoding processes are required, namely:

[0064] ① When the screen projection device 103 or the screen projection software projects the visual content of participant A's computer onto the display device 104 for display (referred to as "local screen projection"), one video encoding is required;

[0065] ② When third-party conferencing software shares the recorded content of the local camera with remote participants (referred to as "remote camera sharing"), one video encoding is required;

[0066] ③ When third-party conferencing software shares the visual content of participant A's computer with remote participants (referred to as "remote screen sharing"), video encoding is required once.

[0067] It can be seen that since the screen projection function and the online meeting function are implemented by different software or hardware, the synchronization of visual information requires multiple video encodings, which results in more CPU or GPU computing resources being occupied, and then causes resource competition between the screen projection function and the online meeting function, resulting in delays or freezes in online and offline mixed meeting scenarios.

[0068] In addition, since the online conference function needs to be implemented by conference software, when the display device 104 is not running the conference software, the online conference function needs to be implemented by the conference software running on the electronic device 101. If the presentation content of the display device 104 does not come from the electronic device 101 that implements the online conference function, for example, the electronic device 101 that implements the screen projection function (such as the computer of other local participant B) and the electronic device 101 that implements the online conference function (the computer of participant A who serves as the host) are not the same device; for example, when writing and drawing on the display device 104, the presentation content of the display device 104 cannot be shared with remote participants at the same time, resulting in the online remote participants being unable to synchronize the visual information shared on the display device 104.

[0069] In view of this, an embodiment of the present application provides a conference data transmission method, which uses a display device to fuse and encode the screen display image on the display device and the scene camera image captured by the image acquisition device of the display device, and then sends it to the screen projector or screen projection software (i.e., the screen projection sending end) on the electronic device, and then presents it to the conference software running on the electronic device through the screen projection sending end; this method has the following technical effects:

[0070] On the one hand, the method only needs to complete the process of fusing and encoding the screen display image on the display device and the scene camera image captured by the image acquisition device of the display device, as well as the encoding process of local screen projection once, to achieve local screen projection, remote camera sharing and remote screen sharing; there is no need to perform video encoding for local screen projection, remote camera sharing and remote screen sharing respectively, which reduces the number of video encoding times in online and offline mixed conference scenarios. In this way, the problem of multiple video encodings required for the synchronization of visual information due to the fact that the screen projection function and the online conference function are implemented by different software (or hardware) in the related technology can be solved, the CPU or GPU computing resources occupied can be reduced, and the resource competition problem between the screen projection function and the online conference function can be reduced, and the problem of delay or freezing in online and offline mixed conference scenarios can be avoided to a certain extent.

[0071] Secondly, since this method captures the screen display image on the display device and the scene camera image captured by the image acquisition device of the display device through a display device, and then fuses and encodes them to share them with online remote participants, when the display device is not running the conference software, the presentation content of the display device can be shared with remote participants at the same time, avoiding the problem that the presentation content of the display device cannot be shared with remote participants when it does not come from an electronic device that realizes the online conference function, thereby improving the online conference visual experience of remote participants.

[0072] The following is a detailed description of each embodiment in conjunction with the accompanying drawings. It should be noted that the order in which the following embodiments are described does not limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in an order different from that shown in the drawings.

[0073] With reference to Figures 1, 2, 3, 10, and 11, in order to achieve remote sharing of visual information, so that online remote participants can fully receive local visual information, and so that remote participants can synchronize the recorded content of the local camera with the screen sharing content, an embodiment of the present application provides a conference data transmission method, which is applied to the display device 104 in the conference system. As shown in Figure 3, the specific process of the conference data transmission method can be as follows: Steps 301 to 304, wherein:

[0074] 301. Obtain the screen display image on the display device 104 in the current meeting.

[0075] Among them, the current meeting can be a multi-person interactive scenario commonly referred to as a corporate meeting, teaching, discussion, brainstorming, etc. The current meeting can specifically be an online and offline hybrid meeting. For example, as shown in Figure 4, Figure 4 is a scene diagram of an online and offline hybrid meeting. The offline local participants include participant A, participant B, participant C and participant D, and the online remote participants include participant E. The offline local participants receive visual information and auditory information through the display device 104; the online remote participant (such as participant E) computer and one of the offline electronic devices 101 (such as participant A's computer) both run conference software, and the offline microphone and camera collect data and transmit it to the online remote participant through the offline electronic device 101 through the conference software, so that the remote participant can receive visual information and auditory information.

[0076] Please refer to Figure 4. Specifically, in a mixed online and offline conference scenario, on the one hand, in order to enable local participants to share auditory and visual information, a screen projector 103 or screen projection software can be run on the electronic device 101 of one of the local participants (such as the computer of participant A who serves as the host, or the computer of participant B), so that the visual content (such as pictures or documents, collectively referred to as videos) of the electronic device 101 can be projected to the display device 104 for display, and the auditory content (such as audio) of the electronic device 101 can be transmitted to the display device 104 and output through the speaker of the display device 104. On the other hand, in order to enable remote participants to share local auditory information (such as the recorded sound of the offline microphone, the auditory content of participant A's computer, and the auditory content of participant B's computer) and visual information (such as the recorded content of the local camera, the visual content of participant A's computer, and the visual content of participant B's computer), auditory and visual information can be shared with remote participants (such as participant E) through third-party conference software on one of the local electronic devices 101 (such as the computer of participant A serving as the host). The offline microphone is a microphone in the offline conference room of the current meeting. For example, the offline microphone can be a microphone integrated in the display device 104 or a peripheral microphone device connected to the display device 104.

[0077] Among them, the screen display refers to the content displayed on the display device 104. For example, it can be the projection content of the electronic device 101 in the conference system on the display device 104 (such as pictures, videos, PPT, etc. projected to the display device 104), the drawing content on the display device 104 (for example, writing, drawing, annotation, etc. on the display device 104), etc.

[0078] For example, the content being displayed on the display device 104 may be obtained as a screen display image by taking a screenshot or capturing a video stream.

[0079] 302. Obtain a scene camera image captured by an image acquisition device of the display device 104 in the current meeting.

[0080] The scene camera image refers to the captured content of the offline conference scene, for example, the image of the local participants captured by the camera on the display device 104 .

[0081] For example, referring to Figure 10 or Figure 11, an image acquisition device (such as a camera) can be integrated on the display device 104, or the display device 104 can establish a connection with other image acquisition devices (such as an independent camera), so as to capture images or videos of offline meeting scenes (for example, offline local participants, offline display items) as scene camera images.

[0082] 303. Fusion-encode the screen display image and the scene camera image to obtain the video encoding content of the current meeting.

[0083] The video encoding content is obtained by fusing and encoding the screen display image and the scene camera image.

[0084] Exemplarily, step 303 may specifically include the following steps 3031 to 3032, wherein:

[0085] 3031. Based on the image stitching format of the current conference, stitch the screen display image and the scene camera image to obtain stitched video content.

[0086] Among them, the picture splicing format is the layout method when the screen display picture and the scene camera picture are spliced ​​together. For example, as shown in Figure 7, the picture splicing format can be split screen (that is, the screen display picture and the scene camera picture are displayed in divided screen areas, refer to (a) in Figure 7, for example, the screen is divided into left and right sides to display the scene camera picture and the screen display picture respectively, and the screen is divided into upper and lower sides to display the scene camera picture and the screen display picture respectively), picture-in-picture with desktop as the main focus (that is, the screen display picture is the main focus, refer to (b) in Figure 7, and the scene camera picture is displayed on the screen display picture), picture-in-picture with camera as the main focus (that is, the scene camera picture is the main focus, refer to (c) in Figure 7, and the screen display picture is displayed on the scene camera picture), desktop only (that is, only the screen display picture is displayed, refer to (d) in Figure 7), camera only (that is, only the scene camera picture is displayed, refer to (e) in Figure 7), etc.

[0087] In some embodiments, the screen display image and the scene camera image have the same signal format. In this case, in step 3031, the screen display image and the scene camera image can be directly spliced ​​according to the image splicing format of the current meeting to obtain the spliced ​​video content.

[0088] For example, if the user sets the current meeting to be displayed in split screen (such as left and right split screen, with the scene camera screen on the left and the screen display screen on the right) / or it is displayed in split screen by default and the user has not changed the setting, that is, the screen splicing format of the current meeting is split screen (such as left and right split screen, with the scene camera screen on the left and the screen display screen on the right), then the scene camera screen can be placed on the left and the screen display screen can be placed on the right for splicing to form the spliced ​​video content, as shown in (a) in Figure 7.

[0089] For example, if the user sets the current meeting to be displayed in a picture-in-picture mode with the desktop as the main feature (such as displaying the scene camera image in the upper right corner of the screen display) / or the default picture-in-picture mode is displayed with the desktop as the main feature and the user has not changed the setting, that is, the screen splicing format of the current meeting is a picture-in-picture mode with the desktop as the main feature (such as displaying the scene camera image in the upper right corner of the screen display), then the scene camera image can be placed in the upper right corner of the screen display for splicing to form the spliced ​​video content, as shown in (b) in Figure 7.

[0090] For example, if the user sets the current meeting to be displayed in a picture-in-picture mode with the camera as the main focus (such as displaying the screen display image in the upper right corner of the scene camera image) / or the default is to display in a picture-in-picture mode with the camera as the main focus and the user has not changed the settings, that is, the screen splicing format of the current meeting is a picture-in-picture mode with the camera as the main focus (such as displaying the screen display image in the upper right corner of the scene camera image), then the screen display image can be placed in the upper right corner of the scene camera image for splicing to form the spliced ​​video content, as shown in (c) in Figure 7.

[0091] For example, if the user sets the current meeting to be displayed in desktop-only mode (such as full-screen display of the screen display image) / or it is displayed in desktop-only mode by default and the user has not changed the setting, that is, the screen splicing format of the current meeting is desktop-only, then only the screen display image mode can be retained to merge the screen display image with the scene camera image to form the spliced ​​video content, as shown in (d) in Figure 7.

[0092] For example, if the user sets the current meeting to be displayed in camera-only mode (such as full-screen display of the scene camera image) / or it is displayed in camera-only mode by default and the user has not changed the settings, that is, the screen splicing format of the current meeting is camera-only, then only the scene camera image mode can be retained to merge the screen display image and the scene camera image to form the spliced ​​video content, as shown in (e) in Figure 7.

[0093] Furthermore, in some other embodiments, the screen display image and the scene camera image have the same signal format. For example, the signal of the image acquisition device that captures the scene camera image is generally in MJPEG, YUV, or RGB format, and the screen display image obtained by the display device 104 by taking a screenshot or capturing a video stream is in 8-bit ARGB format. Therefore, the scene camera image needs to be format converted before the images are stitched together. In this case, in step 3031, if the content format of the screen display image is different from the signal format of the scene camera image, the scene camera image is format converted to obtain a converted camera image; based on the image stitching format, the screen display image and the converted camera image are stitched together to obtain stitched video content.

[0094] 3032. Encode the spliced ​​video content to obtain the video encoding content.

[0095] 304. Send the video encoding content to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device 101.

[0096] Among them, the screen projection sending end is the screen projection software installed on the electronic device or the screen projection device electrically connected to the electronic device.

[0097] Please refer to Figures 5, 8, 10 and 11. Figure 8 is a schematic diagram illustrating the remote sharing process of visual information provided in an embodiment of the present application. For example, taking the image acquisition device as the camera of the display device 104 as an example, first, on the one hand, the scene camera image can be captured by the camera of the display device 104 (refer to the "camera" part in Figure 8), and when the content format of the screen display image is different from the signal format of the scene camera image, the scene camera image is transcoded (that is, the scene camera image is format converted, refer to the "transcoding" part in Figure 8) to obtain the converted camera image. On the other hand, the display device 104 obtains the screen display image by taking a desktop screenshot or capturing a video stream (refer to the "desktop screenshot" part in Figure 8, and the detailed acquisition method is referred to the relevant description in the previous text, which will not be repeated here); for example, as shown in Figure 11, the screen display image can be the screen projection content from the computer of the local participant D displayed on the display device 104.

[0098] Then, the scene camera image (or the converted camera image) and the screen display image are fused and encoded (including image stitching and encoding of the spliced ​​video content, refer to the "image stitching" and "video encoding" parts in Figure 8) to obtain the video encoding content.

[0099] Finally, referring to Figure 10, the video encoding content is sent to the screen projection sending end of the electronic device 101 (the screen projection device 103 connected to the computer of the local participant A as shown in Figure 5) through the display device 104; in this way, the screen projection sending end enumerates the USB camera in the form of a USB composite device (refer to the "virtual camera" part in Figure 10), and the USB camera enumerated by the screen projection sending end can decode the video encoding content, thereby decoding the screen display image and the scene camera image to present them to the conference software running in the electronic device 101, so that the remote participants can synchronously see the offline visual information (refer to the "transmit to the screen projection sending end" and "expose as a camera" parts in Figure 8), thereby realizing video data transmission for remote video conferencing. For example, as shown in Figure 5 or Figure 11, the screen projection content from the local participant D's computer displayed on the display device 104 sends the video encoding content to the screen projector 103 connected to the local participant A's computer through the display device 104; the screen projector 103 connected to the local participant A's computer enumerates the USB camera in the form of a USB composite device to decode the video encoding content, thereby decoding the screen display image and the scene camera image to present to the conference software running in the local participant A's computer; at this time, the online remote participant E can see the screen projection content of the local participant D's computer. It can be seen that when the display device 104 is not running the conference software, even if the presentation content of the display device 104 is not from the local participant A's computer that realizes the online conference function, but from the local participant D's computer, the online remote participant E can still synchronously see the screen projection content of the local participant D's computer.

[0100] It can be understood that the screen projection sending end that decodes and exposes the video encoding content can be a screen projector or a screen projection software running on the electronic device 101 (such as the screen projection software running on the computer of local participant A). The screen projection software uses a virtual USB drive to decode the video encoding content and the screen display image and scene camera image to present them to the conference software running in the electronic device 101, so that remote participants can synchronously see the screen projection content of the display device 104 and the content captured by the camera of the display device 104.

[0101] It can be seen that, on the one hand, when offline local participants and online remote participants need to synchronize visual information, only a process of fusion encoding of the screen display image on the display device 104 and the scene camera image captured by the image acquisition device of the display device 104, as well as a local screen projection encoding process, is required to achieve local screen projection, remote camera sharing and remote screen sharing; there is no need to perform video encoding for local screen projection, remote camera sharing and remote screen sharing respectively, which reduces the number of video encoding times in online and offline mixed conference scenarios. On the other hand, by collecting the screen display image on the display device 104 and the scene camera image captured by the image acquisition device of the display device 104, the fusion encoding is performed to share it with the online remote participants, so that the presentation content of the display device 104 can be shared with the remote participants at the same time. When the display device 104 is not running the conference software, the problem of the content displayed on the screen of the display device 104 (such as the painting content on the display device 104, the screen projection content of the local participant B computer on the display device 104 that does not run the conference software as shown in Figure 5) not being able to be shared with the remote participants is avoided when it does not come from the electronic device 101 that realizes the online conference function. This allows the online remote participants to fully receive local visual information, thereby improving the online remote participants' conference visual experience.

[0102] There are many ways to obtain the screen display image on the display device 104 in step 301, illustratively including:

[0103] (1) The screen display image is the screen content of the electronic device 101 in the conference system projected on the display device 104. In this case, step 301 may specifically include the following steps 3011A to 3012A:

[0104] 3011A. Display the preliminary projection content of the target projection device in the current meeting on the display device 104.

[0105] The target screen projection device is a device whose data content needs to be projected to the display device 104, and specifically can be the electronic device 101 in the conference system terminal. For example, as shown in FIG5 , the conference system includes the electronic device 101, a connecting line 102, a screen projector 103, and a display device 104. The electronic device 101 includes the computer of participant A and the computer of participant B in the current conference. Participant A's computer is connected to the screen projector 103 via the connecting line 102 so that the data content on participant A's computer can be sent to the display device 104 for display via the screen projector 103. Participant B's computer has screen projection software installed on it so that the data content on participant B's computer can be sent to the display device 104 for display via the screen projection software. When participant A's computer is projecting its screen, participant A's computer can be used as the target screen projection device; when participant B's computer is projecting its screen, participant B's computer can be used as the target screen projection device.

[0106] Among them, the preliminary projection content is the data content displayed on the display device 104 and projected to the display device 104 by the target projection device.

[0107] In some embodiments, in step 3011A, the electronic device 101 running the conference software locally can be used as the target screen projection device, and the data content on the electronic device 101 running the conference software can be sent to the display device 104 for display through the screen projection device 103 or the screen projection software running on the electronic device 101 running the conference software; at this time, the data content being displayed on the display device 104 is the preliminary screen projection content, thereby realizing the display of the preliminary screen projection content of the target screen projection device on the display device 104 in the current meeting. For example, as shown in Figure 5, the computer of participant A, who is the host, runs the conference software to realize the online conference function, and the visual content (such as pictures, charts, documents, etc.) of participant A's computer can be projected to the display device 104 for display through the screen projection device 103 running on the computer of participant A. At this time, the visual content being displayed on the display device 104 is the preliminary screen projection content, thereby realizing the display of the preliminary screen projection content on the display device 104.

[0108] In other embodiments, in step 3011A, the local electronic device 101 that is not running the conference software can be used as the target screen projection device, and the data content on the electronic device 101 can be sent to the display device 104 for display through the screen projection device 103 or the screen projection software running on the electronic device 101 that is not running the conference software; at this time, the data content being displayed on the display device 104 is the preliminary screen projection content, thereby achieving the display of the preliminary screen projection content on the display device 104. For example, as shown in Figure 5, if the local participant B's computer is not running the conference software, the visual content (such as pictures, charts, documents, etc.) of participant B's computer can be projected to the display device 104 for display through the screen projection software running on participant B's computer, thereby achieving the display of the preliminary screen projection content of the target screen projection device in the current meeting on the display device 104.

[0109] 3012A. Use the preliminary projection content of the target projection device on the display device 104 as the screen display image.

[0110] For example, the preliminary projection content being displayed on the display device 104 can be obtained by methods such as screenshots or capturing video streams, thereby obtaining the preliminary projection content as a screen display image.

[0111] (2) The screen display is the content of the drawing on the display device 104. In this case, step 301 may specifically include the following steps 3011B to 3013B:

[0112] 3011B. Display the preliminary projection content of the target projection device in the current meeting on the display device 104.

[0113] The implementation of step 3011B is similar to that of step 3011A. For details, please refer to the relevant instructions above and will not be repeated here.

[0114] 3012B. In response to a first drawing and marking operation on the preliminary projection content on the display device 104 , display the projection content after drawing on the display device 104 .

[0115] The drawing and marking operation refers to an operation of drawing or marking on the display device 104 , such as writing, annotating, circling, etc. on the display device 104 .

[0116] Among them, the first drawing and marking operation refers to the operation of drawing and marking based on the preliminary projection content displayed on the display device 104. For example, in a teaching scenario, the preliminary projection content displayed on the display device 104 is calculation formula a, and the teacher annotates, circles, and performs other operations on the various parameters of calculation formula a on the display device 104.

[0117] The projection content after painting is a picture displayed on the display device 104, which includes the preliminary projection content and the first painting content.

[0118] The first drawing content is the drawing marking information corresponding to the first drawing marking operation, for example, in a teaching scenario, the teacher's annotations, circles, and other information on the display device 104 for the various parameters of the calculation formula a.

[0119] For example, referring to Figures 5 and 6, the conference software is running on the computer of participant A, who is the host, to realize the online conference function. The visual content (such as pictures) of participant A's computer can be projected to the display device 104 for display through the projector 103 running on the computer of participant A. At this time, the visual content (such as pictures) being displayed on the display device 104 is the preliminary projection content, as shown in (a) in Figure 6; then, the local participant draws and marks the preliminary projection content on the display device 104 (such as drawing a circle on the picture). At this time, in addition to displaying the picture, the circle drawn by the local participant is also displayed on the display device 104, as shown in (b) in Figure 6.

[0120] 3013B. Project the painted content as the screen display image.

[0121] For example, the projection content of the painting being displayed on the display device 104 can be obtained as a screen display picture by means of screenshots or video stream capture, thereby simultaneously capturing the projection content of the electronic device 101 in the conference system on the display device 104, and the painting content based on the projection content displayed on the display device 104 as a screen display picture, so that the offline painting marking information on the projection content can be synchronized to the online remote participants later, avoiding the problem that when the display device 104 does not run the conference software, the presentation content (such as the painting content) of the display device 104 cannot be shared with the remote participants when it does not come from the electronic device 101 that realizes the online conference function, thereby improving the online remote participants' visual conference experience.

[0122] (3) The screen display image is the second drawing content on the display device 104 in the conference system. In this case, step 301 may specifically include the following steps 3011C to 3012C:

[0123] 3011C. In response to a second drawing and marking operation on the display device 104 in the current conference, display a second drawing content on the display device 104 .

[0124] The second drawing and marking operation refers to an operation of drawing and marking on the display device 104 , for example, a local participant writes, draws, etc. on a blank page displayed on the display device 104 .

[0125] The second drawing content is the drawing marking information corresponding to the second drawing marking operation, for example, the drawing marking information such as words written or pictures drawn by the local participant on the blank page displayed on the display device 104 .

[0126] 3012C. Based on the second painting content, obtain the screen display image.

[0127] For example, the second painting content being displayed on the display device 104 can be obtained as a screen display image by methods such as screenshots or video stream capture; thereby capturing the painting content displayed on the display device 104 as a screen display image, so that the offline painting marking information on the display device 104 can be synchronized to the online remote participants. This avoids the problem that when the display device 104 does not run the conference software, the presentation content (such as painting content) of the display device 104 cannot be shared with the remote participants when it does not come from the electronic device 101 that implements the online conference function, thereby improving the online remote participants' visual conference experience.

[0128] Referring to Figures 10 and 11, in order to achieve remote sharing of visual information, so that online remote participants can fully receive local visual information, and so that remote participants can synchronize the recorded content of the local camera with the screen sharing content, the embodiment of the present application also provides a conference data transmission method, which is applied to the screen projector 103 in the conference system. The specific process of the conference data transmission method can be as follows: Steps D1 to D3, wherein:

[0129] D1. Receive the video encoding content sent by the display device 104.

[0130] The video encoding content is obtained by fusing and encoding the screen display image on the display device 104 and the scene camera image captured by the image acquisition device of the display device 104.

[0131] In some implementations, the screen display image is the preliminary screen projection content of the target screen projection device on the display device 104 in the current meeting.

[0132] In some embodiments, the screen display is the post-drawing projection content on the display device 104. The post-drawing projection content includes the initial projection content and the first drawing content, where the first drawing content is the drawing mark information corresponding to the first drawing mark operation on the initial projection content on the display device 104.

[0133] In some implementations, the screen display image is the second drawing content on the display device 104 . The second drawing content is drawing marking information corresponding to the second drawing marking operation on the display device 104 .

[0134] In some implementations, the video encoding content is obtained by encoding the spliced ​​video content, wherein the spliced ​​video content is obtained by splicing the screen display image and the scene camera image based on the image splicing format of the current conference.

[0135] In some embodiments, if the content format of the screen display image is different from the signal format of the scene camera image, the spliced ​​video content is obtained by splicing the screen display image and the converted camera image based on the image splicing format. The converted camera image is obtained by format conversion of the scene camera image.

[0136] Specifically, the video encoding content is obtained by the display device 104 fusing and encoding the screen display image and the scene camera image. The specific method of obtaining the video encoding content can refer to the relevant description of steps 301 to 303, which will not be repeated here.

[0137] D2. Decode the video encoding content to obtain decoded video content of the video encoding content.

[0138] The decoded video content includes screen display images and scene camera images.

[0139] D3. Transmit the decoded video content to the conference software running in the electronic device 101 connected to the screen projector 103.

[0140] For example, the display device 104 performs fusion encoding on the screen display image (as shown in FIG5 , the screen projection content of the local participant B's computer on the display device 104) and the scene camera image (such as the video image of the offline local participant) to obtain the video encoding content of the current meeting; and sends the video encoding content to the screen projector 103. The screen projector 103 can receive the video encoding content sent by the display device 104, and enumerate the USB camera as a USB composite device on the electronic device 101 through the screen projector 103. The USB camera enumerated by the screen projector 103 can decode the video encoding content, thereby decoding the screen display image and the scene camera image and presenting them to the conference software running in the electronic device 101, thereby realizing the transmission of the decoded video content (including the screen display image and the scene camera image) to the conference software running in the electronic device 101 connected to the screen projector 103, so that the remote participants can synchronously see the offline visual information.

[0141] Similarly, since the screen projection function and the online conference function (the online conference function includes the remote video conferencing function and the remote desktop sharing function) are implemented by different software or hardware (for example, the screen projection function is implemented by the screen projection device 103 or screen projection software on the computer of the local participant A), if the auditory information of the online and offline participants is to be synchronized, at least three audio encoding processes are required, namely:

[0142] ① When the screen projection device 103 or the screen projection software transmits the auditory content of participant A's computer to the display device 104 and outputs it through the speaker of the display device 104 (referred to as "local audio output"), audio encoding is required once;

[0143] ② When third-party conferencing software shares the recorded audio of an offline microphone with remote participants (referred to as "remote microphone sharing"), audio encoding is required once;

[0144] ③ When third-party conferencing software shares the auditory content of participant A's computer with remote participants (referred to as "remote audio sharing"), audio encoding is required once.

[0145] It can be seen that since the screen projection function and the online conference function are implemented by different software or hardware, the synchronization of auditory information requires multiple audio encodings, which results in more CPU or GPU computing resources being occupied, and then causes resource competition between the screen projection function and the online conference function, resulting in delays or freezes in mixed online and offline conference scenarios.

[0146] In view of this, an embodiment of the present application provides a conference data transmission method, which uses a display device to fuse and encode the scene audio captured by the audio acquisition device of the display device and the playback audio of the display device, and then sends it to the screen projector or screen projection software on the electronic device (i.e., the screen projection sending end), and then presents it to the conference software running in the electronic device through the screen projection sending end; this method has the following technical effects:

[0147] On the one hand, the method only needs to complete the process of fusing the scene audio captured by the audio acquisition device of the display device with the playback audio of the display device once, as well as the encoding process of the local audio output once, to achieve local audio output, remote microphone sharing and remote audio sharing; there is no need to perform audio encoding for local audio output, remote microphone sharing and remote audio sharing respectively, which reduces the number of audio encoding times in online and offline mixed conference scenarios. In this way, the problem of multiple audio encodings required for the synchronization of auditory information due to the fact that the screen projection function and the online conference function are implemented by different software (or hardware) in the related technology can be solved, the CPU or GPU computing resources occupied can be reduced, and the resource competition problem between the screen projection function and the online conference function can be reduced, and the problem of delay or freezing in online and offline mixed conference scenarios can be avoided to a certain extent.

[0148] Secondly, since this method fuses and encodes the scene audio captured by the audio acquisition device of the display device and the playback audio of the display device to share them with online remote participants, when the display device is not running the conference software, the presentation content of the display device can be shared with remote participants at the same time, avoiding the problem that the presentation content of the display device cannot be shared with remote participants when it does not come from an electronic device that realizes the online conference function, thereby improving the online conference auditory experience of remote participants.

[0149] 10 and 11 , in order to achieve remote sharing of auditory information, so that online remote participants can fully receive local auditory information, and so that remote participants can synchronize the recorded sound of the offline microphone with the sound of the electronic device 101 itself, an embodiment of the present application provides a conference data transmission method, which is applied to the display device 104 in the conference system. The specific process of the conference data transmission method can be as follows: Steps B1 to B4, wherein:

[0150] B1. Acquire the scene audio captured by the audio acquisition device of the display device 104 in the current conference.

[0151] The played audio is the sound played by the display device 104 .

[0152] For example, the sound output by the speaker of display device 104 can be obtained to obtain the audio played by display device 104 in the current meeting. For example, as shown in Figure 5, the auditory content of the computer of local participant A is transmitted to display device 104 via screen projection device 103 or screen projection software and amplified and outputted via the speaker of display device 104. The sound output from the speaker of display device 104 at this time from the computer of local participant A can be obtained to obtain the audio played by display device 104. For another example, the speaker of display device 104 outputs the local audio of display device 104. The local audio output by the speaker of display device 104 at this time can be obtained to obtain the audio played by display device 104.

[0153] B2. Obtain the playback audio of the display device 104 in the current conference.

[0154] The played audio is the sound played by the display device 104 .

[0155] For example, the sound output by the speaker of display device 104 can be obtained to obtain the audio played by display device 104 in the current meeting. For example, as shown in Figure 5, the auditory content of the computer of local participant A is transmitted to display device 104 via screen projection device 103 or screen projection software and amplified and outputted via the speaker of display device 104. The sound output from the speaker of display device 104 at this time from the computer of local participant A can be obtained to obtain the audio played by display device 104. For another example, the speaker of display device 104 outputs the local audio of display device 104. The local audio output by the speaker of display device 104 at this time can be obtained to obtain the audio played by display device 104.

[0156] B3. Fusion-encode the scene audio and the playback audio to obtain the audio encoding content of the current meeting.

[0157] The audio coding content is the coding content obtained by fusion coding the scene audio and the playback audio.

[0158] There are many ways to perform fusion encoding to obtain audio encoding content in step B3, illustratively including:

[0159] (1) The scene audio and the playback audio are mixed according to their original audio amplitudes. At this time, in step B3, the scene audio and the playback audio can be fused and encoded according to the original audio amplitudes to obtain the audio encoding content of the current meeting.

[0160] (2) In order to enable the remote participants to clearly hear the speech of the local participants, the playback audio (i.e., the speaker sound of the display device 104) can be appropriately suppressed. At this time, in step B3, if the scene audio is not empty, the scene audio and the playback audio are mixed according to the preset weighting coefficient to obtain mixed audio, wherein the preset weighting coefficient is used to suppress the playback audio; and the mixed audio is encoded to obtain the audio encoding content.

[0161] Among them, the preset weighting coefficient is used to indicate the volume coefficient of the scene audio and the playback audio when the scene audio and the playback audio are mixed; the preset weighting coefficient can be set according to the actual business scenario requirements, and the specific value of the preset weighting coefficient is not restricted here. For example, the preset weighting coefficient can be 100% for the scene audio and 25% for the playback audio, so that the volume of the scene audio (i.e., the microphone signal) remains unchanged, while the volume of the playback audio (i.e., the speaker) is attenuated to 25% of the original; for another example, the preset weighting coefficient can be 200% for the scene audio and 100% for the playback audio, so that the volume of the playback audio (i.e., the speaker) remains unchanged, while the scene audio (i.e., the microphone signal) is increased to 200% of the original.

[0162] Furthermore, since in some cases the audio output by the speaker (i.e., the playback audio) is multi-channel and has a higher sampling rate than the microphone input (i.e., the scene audio), and the microphone input (i.e., the scene audio) is monophonic, it is necessary to resample the audio output by the speaker (i.e., the playback audio) so that the number of channels and the sampling rate of the audio output by the speaker (i.e., the playback audio) are aligned with the audio of the microphone (i.e., the scene audio). At this time, it is also necessary to obtain and resample the playback audio according to the number of channels and the sampling rate of the scene audio before fusion encoding with the scene audio. For example, if the scene audio is not empty, the scene audio and the playback audio are mixed according to a preset weighting coefficient to obtain mixed audio, wherein the preset weighting coefficient is used to suppress the playback audio; the number of channels and the sampling rate of the scene audio are obtained; the playback audio is resampled according to the number of channels and the sampling rate to obtain sampled audio; the scene audio and the sampled audio are mixed to obtain the mixed audio.

[0163] Furthermore, in order to improve the quality of the scene audio transmitted to the remote participants, the scene audio can also be subjected to noise reduction processing.

[0164] B4. Send the audio encoding content to the screen projection sending end, so that the screen projection sending end can decode the scene audio and the playback audio based on the audio encoding content and present them to the conference software running in the electronic device 101.

[0165] Among them, the screen projection sending end is the screen projection software installed on the electronic device or the screen projection device electrically connected to the electronic device.

[0166] Please refer to Figures 5 and 9. Figure 9 is a schematic diagram illustrating the remote sharing process of auditory information provided in an embodiment of the present application. For example, taking the microphone of the display device 104 as an example, first, on the one hand, the speaking voice of the offline participants can be collected by the microphone of the display device 104 and noise reduction can be performed to obtain scene audio (refer to the "microphone input" and "noise reduction" parts in Figure 9). On the other hand, the playback audio is obtained by obtaining the audio output by the speaker of the display device 104; and when the number of channels and the sampling rate of the playback audio do not match the scene audio, the playback audio is resampled according to the number of channels and the sampling rate of the scene audio to obtain the sampled audio, so that the number of channels and the sampling rate of the sampled audio are aligned with the scene audio (refer to the "speaker output" and "resampling" parts in Figure 9. For detailed acquisition methods, please refer to the relevant instructions above and will not be repeated here).

[0167] Then, the playback audio (or sampled audio) and the scene audio are fused and encoded (including mixing and encoding of the mixed audio content, refer to the "Mixing" and "Audio Encoding" parts in Figure 9) to obtain audio encoding content.

[0168] Finally, referring to Figure 10, the audio encoding content is sent to the screen projection sending end of the electronic device 101 (the screen projection device 103 connected to the computer of the local participant A as shown in Figure 5) through the display device 104; in this way, the screen projection sending end enumerates the USB microphone in the form of a USB composite device (refer to the "virtual microphone" part in Figure 10), and the USB microphone enumerated by the screen projection sending end can decode the audio encoding content, thereby decoding the playback audio (such as the audio from the computer of the local participant A output by the speaker of the display device 104) and the scene audio (such as the speaking voice of the offline local participant) and presenting them to the conference software running in the electronic device 101, so that the remote participants can synchronously hear the offline auditory information (refer to the "transmitted to the screen projection sending end" and "exposed as microphone" parts in Figure 9), thereby realizing the audio data transmission of the remote video conference.

[0169] It can be understood that the screen projection sending end that decodes and exposes the audio encoding content can be a screen projector or a screen projection software running on the electronic device 101 (such as the screen projection software running on the computer of local participant A). The screen projection software uses a virtual USB drive to decode the video encoding content and present the playback audio and scene audio to the conference software running in the electronic device 101, so that remote participants can synchronously hear the offline auditory information.

[0170] It can be seen that, on the one hand, when offline local participants and online remote participants need to synchronize auditory information, only a process of fusion encoding of the scene audio captured by the audio acquisition device of the display device 104 and the playback audio of the display device 104 is required, as well as a local audio output encoding process, to achieve local audio output, remote microphone sharing and remote audio sharing; there is no need to perform audio encoding for local audio output, remote microphone sharing and remote audio sharing respectively, which reduces the number of audio encoding times in online and offline mixed conference scenarios. On the other hand, by fusing and encoding the scene audio captured by the audio acquisition device of the display device 104 with the playback audio of the display device 104 and sharing them with the online remote participants, the presentation content of the display device 104 can be shared with the remote participants at the same time. When the display device 104 is not running the conference software, the problem of the audio output by the speaker of the display device 104 (such as the local audio of the display device 104, as shown in Figure 5, from the computer of the local participant B who is not running the conference software) not coming from the electronic device 101 that realizes the online conference function and not being able to be shared with the remote participants is avoided. This allows the online remote participants to fully receive the local auditory information, thereby improving the online remote participants' conference auditory experience.

[0171] 10 and 11 , in order to achieve remote sharing of auditory information, so that online remote participants can fully receive local auditory information, and so that remote participants can synchronize the recorded sound of the offline microphone with the sound of the electronic device 101 itself, an embodiment of the present application provides a conference data transmission method, which is applied to the screen projector 103 in the conference system. The specific process of the conference data transmission method can be as follows: Steps F1 to F3, wherein:

[0172] F1. Receive the audio encoding content sent by the display device 104.

[0173] The audio encoding content is obtained by fusing and encoding the scene audio captured by the audio acquisition device of the display device 104 and the playback audio of the display device 104 .

[0174] In some embodiments, if the scene audio is not empty, the scene audio and the playback audio are mixed according to a preset weighting coefficient to obtain mixed audio, wherein the preset weighting coefficient is used to suppress the playback audio; and the mixed audio is encoded to obtain the audio encoding content.

[0175] In some implementations, the mixed audio is obtained by resampling the playback audio according to the number of channels and the sampling rate to obtain sampled audio, and mixing the scene audio and the sampled audio.

[0176] Specifically, the audio encoding content is obtained by the display device 104 by fusion encoding the playback audio and the scene audio. The specific method of obtaining the audio encoding content can refer to the relevant instructions in the previous text (for example, refer to steps B1 to B3), which will not be repeated here.

[0177] F2. Decode the audio encoding content to obtain decoded audio content of the audio encoding content.

[0178] Among them, the decoded audio content includes playback audio and scene audio.

[0179] F3. Transmit the decoded audio content to the conference software running in the electronic device 101 connected to the screen projector 103.

[0180] For example, the display device 104 fuses and encodes the playback audio (as shown in FIG5 , the sound from the local participant B's computer, output through the speaker of the display device 104) and the scene audio (such as the voice of the offline local participant), and obtains the audio encoding content of the current meeting; and sends the audio encoding content to the screen projector 103. The screen projector 103 can receive the audio encoding content sent by the display device 104, and enumerate the USB microphone as a USB composite device on the electronic device 101 through the screen projector 103. The USB microphone enumerated by the screen projector 103 can decode the audio encoding content, thereby decoding the playback audio and scene audio and presenting them to the conference software running in the electronic device 101, thereby realizing the transmission of the decoded audio content (including playback audio and scene audio) to the conference software running in the electronic device 101 connected to the screen projector 103, so that the remote participants can synchronously hear the offline auditory information.

[0181] Therefore, on the one hand, when offline local participants and online remote participants need to synchronize auditory information, only a process of fusion encoding of the scene audio captured by the audio acquisition device of the display device 104 and the playback audio of the display device 104, and a process of encoding the local audio output are required to achieve local audio output, remote microphone sharing and remote audio sharing; there is no need to perform audio encoding for local audio output, remote microphone sharing and remote audio sharing respectively, which reduces the number of audio encoding times in online and offline mixed conference scenarios. On the other hand, by fusing and encoding the scene audio captured by the audio acquisition device of the display device 104 with the playback audio of the display device 104 and sharing them with the online remote participants, the presentation content of the display device 104 can be shared with the remote participants at the same time. When the display device 104 is not running the conference software, the problem of the audio output by the speaker of the display device 104 (such as the local audio of the display device 104, as shown in Figure 5, from the computer of the local participant B who is not running the conference software) not coming from the electronic device 101 that realizes the online conference function and not being able to be shared with the remote participants is avoided. This allows the online remote participants to fully receive the local auditory information, thereby improving the online remote participants' conference auditory experience.

[0182] In addition, it is understandable that in order to enable online remote participants to receive local auditory information and local visual information simultaneously and comprehensively, remote participants can synchronize the recorded sound of the offline microphone, the sound of the electronic device 101 itself, the recorded content of the local camera, and the screen sharing content, and can also collect auditory information (i.e., audio encoding content) and visual information (i.e., video encoding content) at the same time for synchronization. For example, for the display device 104 in the conference system, you can refer to the above steps 301 to 304 and steps B1 to B4 to collect video encoding content and audio encoding content respectively and send them to the screen projection sending end, so that the screen projection sending end can decode the scene audio based on the audio encoding content, play the audio and present it to the conference software running in the electronic device 101, and decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device 101.

[0183] For example, for the screen projector 103 in the conference system, you can refer to steps D1 to D3 and steps F1 to F3 above, and simultaneously receive the video encoding content and audio encoding content sent by the display device 104; then decode the audio encoding content to obtain the decoded audio content, and transmit it to the conference software running in the electronic device 101 connected to the screen projector 103; and decode the video encoding content to obtain the decoded video content, and transmit it to the conference software running in the electronic device 101 connected to the screen projector 103.

[0184] In order to better implement the conference data transmission method in the embodiment of the present application, based on the conference data transmission method, the embodiment of the present application further provides a conference data transmission device, as shown in FIG12 , the conference data transmission device:

[0185] The acquisition unit 1201 is used to acquire the screen display image on the display device in the current meeting;

[0186] The acquisition unit 1201 is further configured to acquire a scene camera image captured by an image acquisition device of the display device;

[0187] The encoding unit 1202 is configured to perform fusion encoding on the screen display image and the scene camera image to obtain the video encoding content of the current meeting;

[0188] The sending unit 1203 is used to send the video encoding content to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed on the electronic device or the screen projector electrically connected to the electronic device.

[0189] In some implementations of the present application, the acquiring unit 1201 is specifically configured to:

[0190] Display the preliminary projection content of the target projection device on the display device in the current meeting;

[0191] The screen display image is obtained based on the preliminary projection content of the target projection device on the display device.

[0192] In some implementations of the present application, the acquiring unit 1201 is specifically configured to:

[0193] In response to a first drawing mark operation on the preliminary projection content on the display device, displaying the projection content after drawing on the display device;

[0194] Based on the screen projection content after the painting, the screen display image is obtained.

[0195] In some implementations of the present application, the acquiring unit 1201 is specifically configured to:

[0196] In response to a second drawing markup operation on the display device in the current meeting, displaying a second drawing content on the display device;

[0197] Based on the second painting content, the screen display image is acquired.

[0198] In some implementations of the present application, the encoding unit 1202 is specifically configured to:

[0199] Based on the picture splicing format of the current meeting, the screen display picture and the scene camera picture are spliced ​​together to obtain spliced ​​video content;

[0200] The spliced ​​video content is encoded to obtain the video encoding content.

[0201] In some implementations of the present application, the encoding unit 1202 is specifically configured to:

[0202] If the content format of the screen display image is different from the signal format of the scene camera image, performing format conversion on the scene camera image to obtain a converted camera image;

[0203] Based on the picture splicing format, the screen display picture and the converted camera picture are spliced ​​to obtain spliced ​​video content.

[0204] In some implementations of the present application, the encoding unit 1202 is specifically configured to:

[0205] Obtaining scene audio captured by an audio acquisition device of the display device in the current conference;

[0206] Obtain the playback audio of the display device in the current conference;

[0207] Fusion-encoding the scene audio and the playback audio to obtain audio encoding content of the current meeting;

[0208] In some implementations of the present application, the sending unit 1203 is specifically configured to:

[0209] The audio encoding content is sent to the screen projection sending end on the electronic device, so that the screen projection sending end can decode the scene audio and the playback audio based on the audio encoding content and present them to the conference software running in the electronic device.

[0210] In some implementations of the present application, the encoding unit 1202 is specifically configured to:

[0211] If the scene audio is not empty, mixing the scene audio and the playback audio according to a preset weighting coefficient to obtain a mixed audio, wherein the preset weighting coefficient is used to suppress the playback audio;

[0212] The mixed audio is encoded to obtain the audio encoding content.

[0213] In some implementations of the present application, the encoding unit 1202 is specifically configured to:

[0214] Obtain the number of channels and sampling rate of the scene audio;

[0215] Resampling the played audio according to the number of channels and the sampling rate to obtain sampled audio;

[0216] The scene audio and the sampled audio are mixed to obtain the mixed audio.

[0217] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the previous method embodiments and will not be repeated here.

[0218] Since the conference data transmission device can execute the conference data transmission method in any embodiment of the present application, it can achieve the beneficial effects that can be achieved by the conference data transmission method in any embodiment of the present application. Please see the previous description for details and will not be repeated here.

[0219] In addition, in order to better implement the conference data transmission method in the embodiment of the present application, based on the conference data transmission method, the embodiment of the present application further provides a computer device, which integrates any conference data transmission device provided in the embodiment of the present application, and the computer device includes:

[0220] one or more processors;

[0221] Memory;

[0222] and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor in accordance with any one of the embodiments of the conference data transmission method.

[0223] The present application also provides a computer device that integrates any of the conference data transmission methods provided in the present application. As shown in Figure 13, it shows a schematic diagram of the structure of the computer device involved in the present application embodiment. Specifically:

[0224] The computer device may include components such as a processor 1301 with one or more processing cores, a memory 1302 with one or more computer-readable storage media, a power supply 1303, and an input unit 1304. Those skilled in the art will appreciate that the computer device structure shown in FIG13 does not limit the computer device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0225] Processor 1301 is the control center of the computer device. It connects the various components of the entire computer device using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 1302 and accessing data stored in memory 1302, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the computer device. Optionally, processor 1301 may include one or more processing cores. Preferably, processor 1301 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1301.

[0226] The memory 1302 can be used to store software programs and modules. The processor 1301 executes various functional applications and data processing by running the software programs and modules stored in the memory 1302. The memory 1302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1302 may also include a memory controller to provide the processor 1301 with access to the memory 1302.

[0227] The computer device also includes a power supply 1303 for supplying power to various components. Preferably, the power supply 1303 can be logically connected to the processor 1301 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 1303 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0228] The computer device may further include an input unit 1304 , which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0229] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 1301 in the computer device will load the executable files corresponding to one or more application processes into the memory 1302 according to the following instructions, and the processor 1301 will run the application stored in the memory 1302 to implement various functions as follows:

[0230] Obtain the screen display image on the display device in the current meeting;

[0231] Acquire a scene camera image captured by an image acquisition device of the display device in the current conference;

[0232] Fusion encoding is performed on the screen display image and the scene camera image to obtain the video encoding content of the current meeting;

[0233] The video encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed on the electronic device or the screen projector electrically connected to the electronic device.

[0234] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0235] To this end, embodiments of the present application provide a computer-readable storage medium, which may include a read-only memory (ROM), random access memory (RAM), a disk, or an optical disk. A computer program is stored thereon, and the computer program is loaded by a processor to execute any of the conference data transmission methods provided in embodiments of the present application. For example, the computer program loaded by the processor may execute the following steps:

[0236] Obtain the screen display image on the display device in the current meeting;

[0237] Acquire a scene camera image captured by an image acquisition device of the display device in the current conference;

[0238] Fusion encoding is performed on the screen display image and the scene camera image to obtain the video encoding content of the current meeting;

[0239] The video encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed on the electronic device or the screen projector electrically connected to the electronic device.

[0240] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above and will not be repeated here.

[0241] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to implement as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments and will not be repeated here.

[0242] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0243] The above is a detailed introduction to a conference data transmission method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A conference data transmission method, characterized in that: A display device applied to a conference system, the conference system comprising a display device and an electronic device, the conference data transmission method comprising: Obtain the screen display image on the display device in the current meeting; Acquire a scene camera image captured by an image acquisition device of the display device in the current conference; Fusion-encode the screen display image and the scene camera image to obtain the video encoding content of the current meeting; The video encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed in the electronic device or the screen projector electrically connected to the electronic device.

2. The conference data transmission method according to claim 1, characterized in that: The fusing and encoding the screen display image and the scene camera image to obtain the video encoding content of the current meeting includes: Based on the picture stitching format of the current conference, stitching the screen display picture with the scene camera picture to obtain stitched video content; The spliced ​​video content is encoded to obtain the video encoding content.

3. The conference data transmission method according to claim 2, characterized in that: The step of combining the screen display image and the scene camera image based on the image combination format of the current conference to obtain combined video content includes: If the content format of the screen display picture is different from the signal format of the scene camera picture, performing format conversion on the scene camera picture to obtain a converted camera picture; Based on the picture splicing format, the screen display picture and the converted camera picture are spliced ​​to obtain spliced ​​video content.

4. The conference data transmission method according to claim 1, characterized in that: The method further comprises: Acquire the scene audio captured by the audio acquisition device of the display device in the current conference; Obtain the playback audio of the display device in the current conference; Fusion-encode the scene audio and the playback audio to obtain the audio encoding content of the current meeting; The audio encoding content is sent to the screen projection sending end, so that the screen projection sending end decodes the scene audio and the playback audio based on the audio encoding content and presents them to the conference software running in the electronic device.

5. The conference data transmission method according to claim 4, characterized in that: The fusing and encoding the scene audio and the playback audio to obtain the audio encoding content of the current conference includes: If the scene audio is not empty, mixing the scene audio and the playback audio according to a preset weighting coefficient to obtain a mixed audio, wherein the preset weighting coefficient is used to suppress the playback audio; The mixed audio is encoded to obtain the audio encoding content.

6. The conference data transmission method according to claim 5, characterized in that: The mixing the scene audio and the playback audio according to the preset weighting coefficient to obtain the mixed audio includes: Obtain the number of channels and sampling rate of the scene audio; Resampling the played audio according to the number of channels and the sampling rate to obtain sampled audio; The scene audio and the sampled audio are mixed to obtain the mixed audio.

7. The conference data transmission method according to claim 1, characterized in that: The obtaining of the screen display image on the display device in the current meeting includes: Display the preliminary screen projection content of the target screen projection device in the current meeting on the display device; Based on the preliminary projection content of the target projection device on the display device, obtain The screen displays a picture.

8. The conference data transmission method according to claim 7, characterized in that: The obtaining the screen display picture based on the preliminary screen projection content of the target screen projection device on the display device includes: In response to a first drawing mark operation on the preliminary projection content on the display device, displaying the projection content after drawing on the display device; Based on the screen projection content after the painting, the screen display image is obtained.

9. The conference data transmission method according to claim 1, characterized in that: The obtaining of the screen display image on the display device in the current meeting includes: In response to a second drawing marking operation on the display device in the current meeting, displaying a second drawing content on the display device; Based on the second painting content, the screen display picture is acquired.

10. A conference data transmission method, characterized in that: A screen projector used in a conference system, the conference system comprising a display device, an electronic device, and a screen projector electrically connected to the electronic device, the conference data transmission method comprising: Receiving video encoding content sent by the display device, wherein the video encoding content is obtained by fusing and encoding a screen display picture on the display device and a scene camera picture captured by an image acquisition device of the display device; Decoding the video encoding content to obtain decoded video content of the video encoding content; The decoded video content is transmitted to the conference software running in the electronic device connected to the screen projector.

11. The conference data transmission method according to claim 10, characterized in that: The method further comprises: Receiving audio encoding content sent by the display device, wherein the audio encoding content is obtained by fusing and encoding scene audio captured by an audio acquisition device of the display device and playback audio of the display device; Decoding the audio encoding content to obtain decoded audio content of the audio encoding content; The decoded audio content is transmitted to the conference software running in the electronic device connected to the screen projector.

12. A conference data transmission method, characterized in that: A display device applied to a conference system, the conference system comprising a display device and an electronic device, the conference data transmission method comprising: Obtain the scene audio captured by the audio acquisition device of the display device in the current conference; Obtain the playback audio of the display device in the current conference; Fusion-encode the scene audio and the playback audio to obtain the audio encoding content of the current meeting; The audio encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the scene audio and the playback audio based on the audio encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed in the electronic device or the screen projector electrically connected to the electronic device.

13. The conference data transmission method according to claim 12, characterized in that: The method further comprises: Obtaining a screen display image on the display device in the current conference; Acquire a scene camera image captured by an image acquisition device of the display device in the current conference; Fusion-encode the screen display image and the scene camera image to obtain the video encoding content of the current meeting; The video encoding content is sent to the screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device.

14. A conference data transmission method, characterized in that: A screen projector used in a conference system, the conference system comprising a display device, an electronic device, and a screen projector electrically connected to the electronic device, the conference data transmission method comprising: Receiving audio encoding content sent by the display device, wherein the audio encoding content is obtained by fusing and encoding scene audio captured by an audio acquisition device of the display device and playback audio of the display device; Decoding the audio encoding content to obtain decoded audio content of the audio encoding content; The decoded audio content is transmitted to the conference software running in the electronic device connected to the screen projector.

15. The conference data transmission method according to claim 14, characterized in that: The method further comprises: Receiving video encoding content sent by the display device, wherein the video encoding content is obtained by fusing and encoding a screen display picture on the display device and a scene camera picture captured by an image acquisition device of the display device; Decoding the video encoding content to obtain decoded video content of the video encoding content; The decoded video content is transmitted to the conference software running in the electronic device connected to the screen projector.

16. A conference data transmission device, characterized in that: The conference data transmission device comprises: An acquisition unit, used to acquire a screen display image on a display device in the current conference; The acquisition unit is further used to acquire a scene camera image captured by an image acquisition device of the display device; An encoding unit, used for fusing and encoding the screen display image and the scene camera image to obtain the video encoding content of the current meeting; A sending unit is used to send the video encoding content to a screen projection sending end, so that the screen projection sending end can decode the screen display image and the scene camera image based on the video encoding content and present them to the conference software running in the electronic device. The screen projection sending end is the screen projection software installed in the conference data transmission device or the screen projector electrically connected to the conference data transmission device.

17. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the conference data transmission method according to any one of claims 1 to 15 is executed.

18. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the conference data transmission method according to any one of claims 1 to 15.

Images