Conference interface display method, conference data processing method and electronic equipment
By adjusting the position and visual focus of the video window within the shared window using mirroring technology, the problem of inconsistency between the video window and the shared document window on the video conferencing interface is solved, enhancing the user's immersion and interactive experience.
Patent Information
- Application Number
- CN202310499088.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2024-11-05
AI Technical Summary
When existing video conferencing applications display both the video window and the shared document window on the meeting interface simultaneously, the video visuals are not coordinated enough, affecting the user's immersion.
By mirroring the image displayed in the video window, the multimedia images in the video window and the multimedia images in the shared window are more coordinated. The mirroring technology is used to adjust the superposition position and visual focus relationship of the video window in the shared window, ensuring that the video images in the video window and the video images in the shared window maintain dynamic coordination.
It enhances the user's immersion in the multimedia content displayed on the meeting interface, ensures that the video window and the sharing window are consistent, reduces the obstruction of shared content by the video window, and improves the user's interactive experience.
Smart Images

Figure CN118900312A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of terminal devices, and in particular to a method for displaying a conference interface, a method for processing conference data, and an electronic device. Background Art
[0002] At present, with the development of computer technology, video conferencing applications are becoming increasingly popular. Video conferencing applications can remotely connect multiple terminals located in different geographical locations and conduct video conferences.
[0003] Whether in an enterprise office scenario or a personal life scenario, users can use video conferencing applications to remotely connect with one or more other users, and each terminal connected to the conference can interact with voice and video. In addition, in the video conferencing application, the terminal connected to the conference can also share the data of the terminal (such as documents, pictures, etc.) with other terminals connected to the conference through document sharing or desktop sharing.
[0004] However, when current video conferencing applications display a video window and a shared document sharing window on a conference interface at the same time, the displayed video images are not coordinated enough, which affects the user's immersion in the video. Summary of the invention
[0005] In order to solve the above technical problems, the present application provides a conference interface display method, a conference data processing method and an electronic device. In the method, at least one frame of image displayed in a video window is mirrored so that the multimedia screen of the video window displayed in the shared window is more coordinated with the multimedia screen in the shared window, so as to enhance the user's immersion in the multimedia screen displayed on the conference interface.
[0006] In a first aspect, an embodiment of the present application provides a method for displaying a conference interface, which is applied to a first electronic device. The method includes: displaying a conference interface; the conference interface includes a shared window and n video windows displayed in the shared window, wherein n≥1, and n is an integer; wherein the n video windows are located at n first positions in the shared window; wherein the shared window is used to display multimedia images related to the shared first multimedia stream; wherein the n video windows are respectively used to display multimedia images of n third multimedia streams respectively related to n second multimedia streams, wherein the n second multimedia streams include multimedia streams collected by n electronic devices, and the n electronic devices include the first electronic device; for any one of the n video windows, there is a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream related to the any one video window, and optionally, there is an identical first frame image; whether there is a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream is related to the positional relationship between the first position corresponding to the any one video window and the corresponding second position; wherein the second position corresponding to the any one video window is: the position of the visual focus of the user of the electronic device corresponding to the any one video window in the shared window.
[0007] According to the first aspect, the conference interface is an interface displayed in response to a received first user operation, wherein the first user operation is used to instruct an operation of sharing a first window or a screen.
[0008] The shared window is used to display a multimedia screen of a first multimedia stream shared to the first window or the screen.
[0009] The cameras (eg, front cameras) of the n electronic devices participating in the conference may each capture video images of the participants, such as the second multimedia stream mentioned above.
[0010] For example, the second multimedia streams collected by the n electronic devices may be multimedia streams collected by the electronic devices after the first electronic device receives the first user operation.
[0011] The first electronic device is an electronic device for sharing documents in this video conference, referred to as the electronic device of the speaker. The n electronic devices participating in this conference exchange data through the cloud.
[0012] The first position of the video window is also referred to as the overlay position of the video window on the shared window.
[0013] The second position corresponding to the video window is also referred to as the position where the visual focus of the participant corresponding to the video window on the video picture of the first multimedia stream displayed in the sharing window is located.
[0014] Among them, the second position of the visual focus (also called the first conference focus or speech focus) corresponding to the speaker's video window in the shared window can be the position of the focus object (such as a mouse icon, highlighted text, etc.) in the video screen of the first multimedia stream in the shared window, or it can be the second position in the shared window determined by performing a visual heat map analysis of the speaker's second multimedia stream in the recent period during the video conference.
[0015] The second position of the visual focus (also called the second conference focus) corresponding to the video window of the participant other than the speaker in the shared window can be the same as the second position of the speech focus corresponding to the speaker's video window, or it can be the first position of the speaker's video window on the shared window (such as the above-mentioned overlay position).
[0016] In this way, the visual focus of the participants other than the speaker on the video screen in the shared window can be the position of the speaker's speech focus or the first position of the speaker's video window.
[0017] In addition, for the n video windows (here, at least one video window) displayed in the shared window, the first positions of the respective video windows in the shared window are related to the video images related to the first multimedia stream displayed in the shared window at different times. In other words, for the same video window (e.g., the speaker's video window), the superimposed position of the video window in the shared window may change as the video images displayed in the shared window change.
[0018] In a video conference, participants can share documents through their respective electronic devices, and the sharing method can be through the first window where the document is located, or through the screen sharing method. In this embodiment, the speaker shares the document with other participants by opening the first window where the document is located or sharing the screen on the first electronic device, so that the participants of this video conference can browse the video screen of the document shared by the speaker through the sharing window of their respective electronic devices (corresponding to the above-mentioned first multimedia stream).
[0019] In addition, for the video screen of the video window superimposed on the shared window, in order to make the video screen match the visual focus in the shared window, at least one frame of the image displayed in the video screen in the video window is mirrored, and whether the frame image is mirrored is related to the positional relationship between the superimposed position of the video window in the shared window (for example, the first position mentioned above) and the position of the visual focus of the user corresponding to the video window in the shared window.
[0020] For example, the video window superimposed in the shared window may include a video window corresponding to the speaker who is sharing the document this time, and the speaker has a second position of visual focus (such as a mouse) for the video screen of the shared window, and the superimposed position (such as the first position) of the speaker's video window in the shared window. The second multimedia stream collected by the speaker's first electronic device and the first multimedia stream used to be displayed in the shared window are multimedia streams that correspond to each other in time. Then, it can be determined whether it is necessary to mirror the corresponding frame of the second multimedia stream in combination with the positional relationship between the first position and the second position that correspond to each other. As a result, the speaker's line of sight in the speaker's video window is looking at the position of the corresponding visual focus (such as a mouse) in the shared window. The video screen of the video window superimposed in the shared window is made more coordinated and matched with the video screen in the shared window at the current moment, and the sense of coordination between the video screen of the shared window and the video screen of the video window displayed in the shared window is improved, so as to enhance the user's immersion in the video screen displayed on the conference interface.
[0021] According to the first aspect, or any implementation of the first aspect above, the second frame images that are mirror images of each other between the third multimedia stream and the second multimedia stream are frame images that are mirror images of each other in a height direction relative to the corresponding video window.
[0022] The two frames of images that are mirror images of each other indicate that they are mirror images of each other in the height direction of the corresponding video window (for example, the height direction is regarded as a plane mirror).
[0023] In addition, the two mirrored frames are symmetrical in the height direction of the corresponding video window.
[0024] According to the first aspect, or any implementation of the first aspect above, when the first position corresponding to any one of the video windows and the corresponding second position are located on the same side of the center line, the third multimedia stream includes a frame image that is mirrored to the corresponding frame image in the second multimedia stream; wherein the center line is a line related to the height direction of any one of the video windows.
[0025] In this way, there exists a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream of any video window.
[0026] According to the first aspect, or any implementation of the first aspect above, when the first position corresponding to any one of the video windows and the corresponding second position are located on different sides of the center line, the third multimedia stream includes the corresponding original frame image in the second multimedia stream.
[0027] In this way, the same first frame image exists between the third multimedia stream and the second multimedia stream of any one video window.
[0028] According to the first aspect, or any implementation of the first aspect above, the angle between the center line and the height direction of the corresponding video window is greater than or equal to 0 degrees and less than or equal to 45 degrees.
[0029] According to the first aspect, or any implementation of the first aspect above, the corresponding first position of the same video window in the shared window is related to the video picture related to the first multimedia stream currently displayed in the shared window.
[0030] In this way, when the video picture related to the first multimedia stream displayed in the shared window changes, the overlay position of the video window superimposed in the shared window may change accordingly, thereby realizing dynamic change of the overlay position of the video window and ensuring that the video picture of the shared window in the conference interface and the video picture of the video window are dynamically coordinated.
[0031] According to the first aspect, or any implementation of the first aspect above, the first position of at least one of the n video windows in the shared window is located in a target area (also called a blank area) in the video screen related to the first multimedia stream displayed in the shared window; wherein the target area does not include foreground elements related to the first multimedia stream.
[0032] Among them, the video picture corresponding to the first multimedia stream displayed in the shared window may include foreground elements and optionally background elements. Then the overlay position of the video window superimposed in the shared window can be the target area in the video picture that does not include the foreground elements, thereby avoiding the video window superimposed on the shared window from blocking the content currently displayed in the shared window (such as document content).
[0033] According to the first aspect, or any implementation of the first aspect above, the n first positions of the n video windows in the shared window change as the second positions corresponding to the corresponding visual focuses change in the preset area in the shared window.
[0034] Among them, as the video screen related to the first multimedia stream displayed in the shared window changes, the perspective focus of the speaker or participant in the shared window may also change. Then, in order to make the superimposed position of the video window displayed in the shared window (for example, the first position mentioned above) change with the visual focus related to the video window. In this embodiment, the shared window can be pre-divided into multiple preset areas, so when the preset area to which the visual focus related to the video window belongs to the second position on the shared window is switched, the superimposed position of the video window in the shared window can also change dynamically. In this way, the superimposed position of the video window of the video screen of the speaker or other participant superimposed in the shared window can change dynamically with the changes in the area where the respective visual focus is located. In this way, through the dynamic change of the superimposed position of the video window in the shared window, each participant can be prompted that the corresponding visual focus (for example, the speech focus) has changed.
[0035] According to the first aspect, or any implementation of the first aspect above, at least one of the n first positions is located in the preset area where the corresponding second position is located.
[0036] Among them, in the scenario where the overlay position of the video window dynamically changes with the preset area where the second position of the visual focus related to the video window is located, the overlay position of the video window (for example, the first position mentioned above) can also dynamically change to the corresponding preset area. In other words, the overlay position of the video window and the second position of the corresponding visual focus are located in the same preset area. In this way, in the conference interface, the video window can be closer to the corresponding visual focus and more coordinated on the screen.
[0037] According to the first aspect, or any implementation of the first aspect above, at least one of the n first positions is located in an adjacent area of the preset area where the corresponding second position in the shared window is located.
[0038] In which, in a scenario where the overlay position of the video window dynamically changes with the preset area where the second position of the visual focus related to the video window is located, the overlay position of the video window (for example, the first position mentioned above) can also dynamically change to an adjacent area (for example, also a preset area) of the corresponding preset area.
[0039] In this way, the superimposed position of the video window of the presenter or other participants can be in the adjacent area of the preset area where the corresponding visual focus is located, so as to ensure that the video window does not block the visual focus.
[0040] According to the first aspect, or any implementation of the first aspect above, the n video windows include a first video window, wherein the first video window is used to display a video screen of a third multimedia stream related to a first electronic device, and the method further includes: based on the first multimedia stream related to the shared window, determining a first position of the first video window to be displayed in the shared window; determining a second position of the visual focus corresponding to a user of the first electronic device related to the first video window in the shared window; based on the positional relationship between the first position and the second position corresponding to the first video window, processing at least one frame of the second multimedia stream related to the first video window to obtain a third multimedia stream related to the first video window; synthesizing the first multimedia stream and the third multimedia stream related to the first video window according to the first position related to the first video window to obtain a target multimedia stream; wherein the conference interface includes a multimedia screen of the target multimedia stream.
[0041] For example, the video screen of the target multimedia stream includes the video screen of a shared window and the video screen of a first video window, wherein the first video window is located at a corresponding first position within the shared window; wherein the video screen of the shared window is the video screen related to the first multimedia stream, and the video screen of the first video window is the video screen related to a third multimedia stream related to the first electronic device.
[0042] According to the first aspect, or any implementation of the first aspect above, the n electronic devices also include at least one second electronic device, and the n video windows include a second video window related to the second electronic device, and the second video window is used to display the video screen of a third multimedia stream related to the second electronic device; wherein the second video window is displayed at a corresponding first position in the shared window at at least one first moment; wherein the at least one moment includes a moment when user voice exists in the second multimedia stream related to the second electronic device.
[0043] Participants other than the speaker (eg, the user who shares the document in the sharing window) may speak during the video conference, and the at least one moment includes the moment when the participant speaks.
[0044] For example, the at least one moment may be the moment when the participant speaks.
[0045] For another example, the at least one moment is the moment when the participant speaks and a period of time after the speech ends.
[0046] For example, during a participant's speech, and during a period when the participant interrupts the speech to listen to the speaker's speech, the at least one moment may be used, and there is no limitation here.
[0047] When a participant other than the speaker is speaking, the microphone of the participant's electronic device (here the second electronic device) can collect the participant's voice. Then the second multimedia stream collected by the second electronic device may include not only the image of the participant captured by its camera, but also the user voice of the participant captured by its microphone.
[0048] Then the video window of the participant can be displayed at the corresponding overlay position (such as the first position mentioned above) in the shared window at at least one moment, so that during the participant's speech, the video window of the participant can also be overlaid in the shared window, and the video screen of the third multimedia stream in the video window also has at least one frame of image that is mirrored from the second multimedia stream.
[0049] In this way, the shared window of the conference interface can be superimposed with the video screen of the speaker and the video screen of the second speaker (such as the above-mentioned participant who spoke), so that each participant of the meeting can see the video screens of the speaker and the speaker (for example, asking questions) at the same time through the conference interface on their respective electronic devices, and the visual focus of the second speaker (also called the second conference focus) can also be the speech focus, so that the video windows of the speaker and other speaking participants can be presented on the conference interface, and the visual sense of interaction with the shared content in the shared window (such as the speech focus). Alternatively, the visual focus of the second speaker (also called the second conference focus) can also be the superimposed position of the speaker's video window, so that the video windows of the speaker and the speaker can be superimposed on the conference interface to present the visual sense of interaction between the speaker and other speaking participants, which can enhance the sense of participation of other participants in the current question content and enhance the sense of immersion.
[0050] According to the first aspect, or any implementation of the first aspect above, the method also includes: obtaining (for example, receiving from the cloud) the second multimedia stream related to the second electronic device; obtaining the first position of the second video window to be displayed in the shared window (the first position can be received from the cloud, and the first position can be determined by the cloud or the second electronic device); obtaining the second position of the visual focus corresponding to the user of the first electronic device related to the second video window in the shared window (the second position can be received from the cloud, and the second position can be determined by the cloud or the second electronic device); based on the positional relationship between the first position and the second position corresponding to the second video window, processing at least one frame of the second multimedia stream related to the second video window to obtain a third multimedia stream related to the second video window; according to the first position related to the second video window, synthesizing the first multimedia stream and the third multimedia stream related to the second video window to obtain the target multimedia stream.
[0051] In a second aspect, an embodiment of the present application provides a method for displaying a conference interface, which is applied to a second electronic device. The method comprises: in response to received user voice information, displaying a conference interface; the conference interface comprises a sharing window and n video windows displayed in the sharing window, wherein n≥2, and n is an integer; wherein the n video windows are located at n first positions in the sharing window; wherein the sharing window is used to display multimedia pictures related to a first multimedia stream shared with a first electronic device; wherein the n video windows are respectively used to display multimedia pictures related to n third multimedia streams respectively related to n second multimedia streams, wherein the n second multimedia streams comprise multimedia streams respectively collected by n electronic devices, and the n electronic devices comprise the first electronic device and the second electronic device; for any one of the n video windows, there exists a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream related to the any one video window; whether there exists a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream is related to the positional relationship between the first position corresponding to the any one video window and the corresponding second position; wherein the second position corresponding to the any one video window is: the position of the visual focus corresponding to the user of the electronic device corresponding to the any one video window in the sharing window.
[0052] The implementation principle of the second aspect is similar to that of the first aspect, except that the second electronic device of the second aspect does not share the first window or screen of the shared window. In the application scenario, the second electronic device can be an electronic device of a participant who is speaking other than the speaker (the user who shares the document in the shared window).
[0053] In this way, the shared window of the conference interface can be superimposed with the video screen of the speaker and the video screen of the second speaker (such as the above-mentioned participant who spoke), so that each participant of the meeting can see the video screens of the speaker and the speaker (for example, asking questions) at the same time through the conference interface on their respective electronic devices, and the visual focus of the second speaker (also called the second conference focus) can also be the speech focus, so that the video windows of the speaker and other speaking participants can be presented on the conference interface, and the visual sense of interaction with the shared content in the shared window (such as the speech focus). Alternatively, the visual focus of the second speaker (also called the second conference focus) can also be the superimposed position of the speaker's video window, so that the video windows of the speaker and the speaker can be superimposed on the conference interface to present the visual sense of interaction between the speaker and other speaking participants, which can enhance the sense of participation of other participants in the current question content and enhance the sense of immersion.
[0054] According to a second aspect, the n video windows include a second video window associated with the second electronic device, the second video window being used to display a video screen of a third multimedia stream associated with the second electronic device;
[0055] The second video window is displayed at a corresponding first position in the sharing window at at least one first moment; wherein the at least one moment includes a moment when user voice exists in the second multimedia stream related to the second electronic device.
[0056] In this embodiment, while a participant is speaking, the video window of the participant can be displayed at a corresponding overlay position (eg, the first position) in the shared window during the speech, or during the speech and a period of time before and after the speech.
[0057] According to the second aspect, or any implementation of the second aspect above, the n video windows include a first video window, and the first video window is used to display a video screen of a third multimedia stream related to the first electronic device. The method also includes: obtaining the second multimedia stream related to the first electronic device (for example, the second multimedia stream is a second multimedia stream from the first electronic device received by the second electronic device from the cloud and forwarded by the cloud); obtaining a first position of the first video window to be displayed in the shared window (the first position can be received from the cloud, and the first position can be determined by the cloud or the first electronic device); obtaining a second position of the visual focus corresponding to the user of the first electronic device related to the first video window in the shared window (the second position can be received from the cloud, and the second position can be determined by the cloud or the second electronic device); based on the positional relationship between the first position and the second position corresponding to the first video window, processing at least one frame of the second multimedia stream related to the first video window to obtain a third multimedia stream related to the first video window; according to the first position related to the first video window and the first position related to the second video window, synthesizing the first multimedia stream, the third multimedia stream related to the first video window, and the third multimedia stream related to the second video window to obtain a target multimedia stream.
[0058] According to the second aspect, or any implementation of the second aspect above, the second frame images that are mirror images of each other between the third multimedia stream and the second multimedia stream are frame images that are mirror images of each other in a height direction relative to the corresponding video window.
[0059] According to the second aspect, or any implementation of the second aspect above, when the first position corresponding to any one of the video windows and the corresponding second position are located on the same side of the center line, the third multimedia stream includes a frame image that is mirrored to the corresponding frame image in the second multimedia stream; wherein the center line is a line related to the height direction of any one of the video windows.
[0060] According to the second aspect, or any implementation of the second aspect above, when the first position corresponding to any one of the video windows and the corresponding second position are located on different sides of the center line, the third multimedia stream includes the corresponding original frame image in the second multimedia stream.
[0061] According to the second aspect, or any implementation of the second aspect above, the angle between the center line and the height direction of the corresponding video window is greater than or equal to 0 degrees and less than or equal to 45 degrees.
[0062] According to the second aspect, or any implementation of the second aspect above, the corresponding first position of the same video window in the shared window is related to the multimedia screen related to the first multimedia stream currently displayed in the shared window.
[0063] According to the second aspect, or any implementation of the second aspect above, the first position of at least one of the n video windows in the shared window is located in a target area of the video screen related to the first multimedia stream displayed in the shared window; wherein the target area does not include foreground elements related to the first multimedia stream.
[0064] According to the second aspect, or any implementation of the second aspect above, the n first positions of the n video windows in the shared window change as the second positions corresponding to the corresponding visual focuses change in the preset area in the shared window.
[0065] According to the second aspect, or any implementation of the second aspect above, at least one of the n first positions is located in the preset area where the corresponding second position is located.
[0066] According to the second aspect, or any implementation of the second aspect above, at least one of the n first positions is located in an adjacent area of the preset area where the corresponding second position in the shared window is located.
[0067] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.
[0068] In a third aspect, an embodiment of the present application provides a display device for a conference interface, which is applied to a first electronic device, and the device is used to execute the method in the first aspect and any one of the implementation modes of the first aspect.
[0069] For example, the display device of the conference interface may be the first electronic device mentioned above, or a software module in the first electronic device, or a chip of the first electronic device, etc.
[0070] The technical effects corresponding to the third aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0071] In a fourth aspect, an embodiment of the present application provides a first electronic device. The first electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the first electronic device executes the method in the first aspect and any one of the implementations of the first aspect.
[0072] The technical effects corresponding to the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0073] In a fifth aspect, an embodiment of the present application provides a computer-readable medium. The computer-readable medium is used to store a computer program, and when the computer program is run on an electronic device (such as a first electronic device), the electronic device executes the method in the first aspect and any one of the implementations of the first aspect.
[0074] The technical effects corresponding to the fifth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0075] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is caused to execute the method in the first aspect and any one of the implementations of the first aspect.
[0076] The technical effects corresponding to the sixth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0077] In a seventh aspect, an embodiment of the present application provides a chip. The chip includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a first electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the first electronic device executes the method in the first aspect and any one of the implementations of the first aspect.
[0078] The technical effects corresponding to the seventh aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.
[0079] In an eighth aspect, an embodiment of the present application provides a display device for a conference interface, which is applied to a second electronic device, and the device is used to execute the method in the second aspect and any one of the implementation modes of the second aspect.
[0080] For example, the display device of the conference interface may be the second electronic device mentioned above, or a software module in the second electronic device, or a chip of the second electronic device, etc.
[0081] The technical effects corresponding to the eighth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation method of the second aspect, and will not be repeated here.
[0082] In a ninth aspect, an embodiment of the present application provides a second electronic device. The second electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the second electronic device executes the method in the second aspect and any one of the implementations of the second aspect.
[0083] The technical effects corresponding to the ninth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation method of the second aspect, and will not be repeated here.
[0084] In a tenth aspect, an embodiment of the present application provides a computer-readable medium. The computer-readable medium is used to store a computer program, and when the computer program is run on an electronic device (such as a second electronic device), the electronic device executes the method in the second aspect and any one of the implementations of the second aspect.
[0085] The technical effects corresponding to the tenth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation method of the second aspect, and will not be repeated here.
[0086] In an eleventh aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is caused to execute the method in the second aspect and any one of the implementations of the second aspect.
[0087] The technical effects corresponding to the eleventh aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation method of the second aspect, and will not be repeated here.
[0088] In the twelfth aspect, an embodiment of the present application provides a chip. The chip includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a second electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the second electronic device executes the method in the second aspect and any one of the implementations of the second aspect.
[0089] The technical effects corresponding to the twelfth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation method of the second aspect, and will not be repeated here.
[0090] In the thirteenth aspect, the embodiment of the present application provides a method for processing conference data, which is applied to a server. The method includes: obtaining a first multimedia stream related to a shared window; obtaining n second multimedia streams related to n video windows, wherein n≥1, and n is an integer; determining n first positions to be displayed in the shared window for the n video windows; determining n second positions corresponding to the visual focus of the user of the electronic device corresponding to the n video windows in the shared window; based on the positional relationship between the first position and the second position corresponding to any one of the n video windows, processing at least one frame of the second multimedia stream related to the any one video window to obtain a third multimedia stream related to the any one video window; synthesizing the n third multimedia streams and the first multimedia stream related to the corresponding n video windows according to the n first positions to obtain a target multimedia stream; wherein the conference interface corresponding to the target multimedia stream includes the shared window and the n video windows displayed at the n first positions in the shared window; the shared window is used to display a multimedia screen related to the first multimedia stream; and the n video windows are used to display multimedia screens related to the n third multimedia streams respectively.
[0091] For example, the server may be a cloud server.
[0092] For example, the first multimedia stream may be received from the first electronic device, and the n second multimedia streams may be received from the n electronic devices, which may include the first electronic device and optionally include the second electronic device.
[0093] The server may send the target multimedia stream to n electronic devices, so that the n electronic devices may display the above conference interface.
[0094] The technical effects corresponding to the thirteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any one of the implementation methods of the first aspect, as well as the technical effects corresponding to the second aspect and any one of the implementation methods of the second aspect, and will not be repeated here.
[0095] In the fourteenth aspect, an embodiment of the present application provides a display system for a conference interface, the system comprising a first electronic device, at least one second electronic device, and a server, wherein the first electronic device is used to execute the method as in the first aspect or any one of the implementations of the first aspect; the second electronic device is used to execute the method as in the second aspect or any one of the implementations of the second aspect, and the server is used to execute the method for processing conference data as described in the thirteenth aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0097] Figure 1 One of the structural schematic diagrams of an electronic device is shown as an example;
[0098] Figure 2 A schematic diagram of the software structure of an electronic device is shown as an example;
[0099] Figure 3 A schematic diagram of an application interface of a video conferencing application in a related art is shown as an example;
[0100] Figure 4 A schematic diagram of an application interface of a video conferencing application in a related art is shown as an example;
[0101] Figure 5a The following is a schematic diagram of an application scenario of a conference system shown as an example;
[0102] Figure 5b The following is a schematic diagram of an application scenario of a conference system shown as an example;
[0103] Figure 6a A schematic diagram of a processing process of a conference system is shown as an example;
[0104] Figure 6b A schematic diagram of a processing process of a conference system is shown as an example;
[0105] Figure 7 A schematic diagram of an application scenario is shown as an example;
[0106] Figure 8 A schematic diagram of an application scenario is shown as an example;
[0107] Figure 9a A schematic diagram of an application scenario is shown as an example;
[0108] Figure 9b A schematic diagram of an application scenario is shown as an example;
[0109] Fig.9c A schematic diagram of an application scenario is shown as an example;
[0110] Fig.10a A schematic diagram of an application scenario is shown as an example;
[0111] Fig.10b A schematic diagram of an application scenario is shown as an example;
[0112] Fig.11 A schematic diagram of an application scenario is shown as an example;
[0113] Fig.12 A schematic diagram of an application scenario is shown as an example;
[0114] Fig.13 A schematic diagram of the structure of the device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0115] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0116] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0117] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.
[0118] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0119] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.
[0120] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. It should be understood that Figure 1 The electronic device 100 shown is only an example of an electronic device. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a device with a display screen such as a cellular phone, a tablet computer (pad), a laptop, a personal computer (PC), a smart car or an Internet of Things device, which is not limited in this application. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0121] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0122] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0123] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0124] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0125] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0126] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.
[0127] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0128] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.
[0129] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0130] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0131] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0132] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G etc. applied to the electronic device 100. The modem processor may include a modulator and a demodulator.
[0133] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0134] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.
[0135] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0136] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0137] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0138] The camera 193 is used to capture still images or videos. The object is projected onto the photosensitive element by generating an optical image through the lens.
[0139] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0140] The internal memory 121 may be used to store computer executable program codes, wherein the executable program codes include instructions. The processor 110 executes the instructions stored in the internal memory 121 so that the electronic device 100 executes the method in the embodiment of the present application.
[0141] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0142] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0143] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.
[0144] Motor 191 can generate vibration prompts.
[0145] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0146] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or removing the SIM card from the SIM card interface 195 .
[0147] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0148] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
[0149] The layered architecture of the electronic device 100 divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.
[0150] The application layer can include a series of application packages.
[0151] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, video conferencing, etc.
[0152] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0153] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0154] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0155] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0156] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0157] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).
[0158] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0159] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0160] The system library and runtime layer include the system library and Android Runtime. The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc. The 3D graphics library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the function functions that need to be called by the Java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.
[0161] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0162] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0163] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0164] A 2D graphics engine is a drawing engine for 2D drawings.
[0165] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0166] Understandably, Figure 2 The components included in the system framework layer, system library and runtime layer shown do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently.
[0167] At present, with the development of computer technology, video conferencing applications are becoming increasingly popular. Video conferencing applications can remotely connect multiple terminals located in different geographical locations and conduct video conferences.
[0168] Whether in an enterprise office scenario or a personal life scenario, users can use video conferencing applications to remotely connect with one or more other users, and each terminal connected to the conference can interact with voice and video. In addition, in the video conferencing application, the terminal connected to the conference can also share the data of the terminal (such as documents, pictures, etc.) with other terminals connected to the conference through document sharing or desktop sharing.
[0169] Figure 3 The figure is a schematic diagram of an application interface of a video conferencing application in the related art.
[0170] like Figure 3 As shown, the application interface is a conference interface 100 , which may include a sharing window (also called a main window) 101 , and a plurality of video windows (also called small windows).
[0171] like Figure 3As shown, the multiple video windows are window 102, window 103, window 104, window 105, window 106, and window 107. The users (also referred to as participants) participating in the conference are participant 1 to participant 6, and the windows 102 to 107 are used to display the video images of the participants 1 to participant 6, respectively.
[0172] Among them, each participant's terminal (such as a laptop) can be installed with a video conferencing application. After the participant operates the video conferencing application to connect the meeting number and other meeting information of the meeting to the meeting, the front camera of the participant's laptop can collect video images (such as the video images of the participants) in real time and upload them to the cloud server of the video conferencing application. The cloud server can display the video images of the participants uploaded by the video conferencing application on the terminal side of each participant on a computer such as a computer. Figure 3 In the windows 102 to 107 shown, the conference interface 100 displayed on the notebook computer side of the participant can display the video images of each participant in the above-mentioned video windows respectively.
[0173] In addition, any participant in this meeting (such as Figure 3 The participant 1 shown in the figure can act as the presenter to share the materials of the speech (here, the slides (PPT)) by operating the video conferencing application on the side of his laptop, so that each participant from participant 1 to participant 6 has a respective conference interface. Figure 3 The conference interface 100 shown in the figure can display the multimedia stream of the presentation PPT shared by the participant 1 in the main window 101 .
[0174] also, Figure 3 The small windows (eg, windows 102 to 107) in the conference may also display a microphone icon, which may be used to identify whether the microphone of a participant's terminal (eg, a laptop) is muted, and whether the corresponding participant is speaking.
[0175] In the above-mentioned related technologies, Figure 3 As shown, the video window of participant 1 who is the presenter (here is window 102) is displayed side by side with the video windows of other participants on the conference interface 100. It is difficult for other participants (such as participants 2 to 6) to quickly identify the small window where the presenter is located through the list of video windows, and it is difficult to quickly locate the video screen of the presenter during the video conference.
[0176] In another related technology, the following solution is given, such as Figure 4The conference interface 200 shown in the figure may include a sharing window 201, wherein the sharing window 201 may display the document shared by the presenter. In addition, the related technology also performs a cutout process on the video image of the presenter uploaded by the presenter's conference terminal (for example, the video image captured in real time by the front camera of the presenter's laptop computer for this conference), and displays the image of the person 202 in a superimposed manner on the conference interface, so that the video image of the presenter and the multimedia stream of the document shared by the presenter are superimposed and displayed on the same display interface. In this way, other participants do not need to search for the small window where the video image of the presenter is located from the small window, but can directly see the video image of the presenter on the conference interface 200.
[0177] Although this related technology can quickly locate the video screen of the speaker from the conference interface, this solution only cuts out the user image in the video screen of the speaker and directly superimposes the cutout character image on the conference interface 200. Then it will have the following problems: 1) There is no connection between the character of the speaker's character image displayed on the conference interface 200 and the current position of the speaker's speech in the shared window 201 (such as the position of the mouse), so that the character image and the speech material are not matched enough. 2) The superimposed character image 202 easily blocks the speech material displayed in the shared window 201, and the participants need to manually move the position of the character image 202 continuously during the meeting. 3) It is difficult for other participants other than the speaker to present their reactions and concerns about the speech materials of the speaker's speech to the speaker.
[0178] To this end, the present application provides a conference system, a conference method, a conference terminal and a cloud.
[0179] Figure 5a , Figure 5b The application scenario diagram of the conference system of the present application is exemplarily shown, and the conference system may include a cloud 300 (or a server), and one or more conference terminals, which are conference terminals 401, 402, and 403. Here, the conference terminal is a laptop computer as an example. In other embodiments, the conference terminal may be any electronic device, such as Figure 1 An electronic device 100 is shown.
[0180] The user of conference terminal 401 is speaker 400, for example, speaker 400 is a user who has shared presentation materials for this conference. The users of conference terminal 402 and conference terminal 403 are participants. Of course, in some scenarios, the identities of speaker and participant can be switched at any time, which is not limited here.
[0181] Among them, the application for implementing video conferencing in this application may include a video conferencing application (such as WeMeeting) installed on the conference terminal side, and a conference main program installed on the cloud 300 side.
[0182] The video conferencing application can be connected to the conference main program on the cloud 300 through the network, so that the video conferencing application on the conference terminal and the conference main program on the cloud 200 can exchange data through the network connection to realize the video conferencing.
[0183] The video conferencing application of the conference terminal can call the camera of the conference terminal (such as the front camera of a laptop computer, which is not limited here) to capture video images, which are video images to be displayed in the video window of the corresponding conference terminal as described above, and the video conferencing application can upload the video images to the conference main program of the cloud 300.
[0184] For example, Figure 5b As shown, the conference terminal 401 can upload the multimedia stream 2 (such as the video stream collected by the local camera, etc.) related to the first video window to the cloud 300. Among them, the first video window is a video window related to the account of the video conferencing application of the conference terminal 401.
[0185] Similarly, if Figure 5b As shown, the conference terminal 402 can upload the multimedia stream 4 (eg, the multimedia stream collected by the local camera) related to the second video window to the cloud 300. The second video window is a video window related to the account of the video conferencing application of the conference terminal 402.
[0186] Similarly, if Figure 5b As shown, the conference terminal 403 can upload the multimedia stream 5 (eg, the multimedia stream collected by the local camera) related to the third video window to the cloud 300. The third video window is a video window related to the account of the video conferencing application of the conference terminal 403.
[0187] In addition, in video conferencing, Figure 5a The presenter 400 shown can operate the video conference application on the conference terminal 401 to select a document sharing method or a desktop sharing method to share the presentation materials among the conference terminal 401 , the conference terminal 402 , and the conference terminal 403 .
[0188] Example 1: When the presenter 400 chooses to share a document, since the document is displayed in a window (referred to as the document window), the video conferencing application on the conference terminal 401 can obtain the multimedia stream to be displayed in the document window in real time and forward it to the cloud 300.
[0189] Example 2: When the presenter 400 chooses to share the document by sharing the desktop, the video conferencing application on the conference terminal 401 can obtain the multimedia stream to be displayed on the screen of the conference terminal 401 in real time, and forward it to the cloud 300 .
[0190] The multimedia stream forwarded by the conference terminal 401 to the cloud 300 is a multimedia stream related to the shared window in the conference interface (also called Figure 5b As shown in the multimedia stream 1), in the above example 1, the shared window here is the above document window, and the multimedia stream 1 here can be a multimedia stream related to the shared window; in the above example 2, the shared window here is the desktop window, and the multimedia stream 1 here can be a multimedia stream related to the desktop window.
[0191] The cloud 300 can call computing resources to process multimedia streams (such as the video windows) related to the respective video windows received by the conference main program in the cloud 300 from the respective conference terminals. Figure 5b The multimedia stream 1, multimedia stream 4, multimedia stream 5) and the multimedia stream 1 related to the shared window are subjected to image analysis, calculation and other processing to obtain the target multimedia stream, and are synchronously sent to the conference terminal 401, the conference terminal 402 and the conference terminal 403. In this way, the conference terminal 401 of the presenter and the conference terminals 402 and 403 of the multiple participants can realize data synchronization of the conference interface through the cloud 300, so that all conference terminals participating in the conference display the same conference interface.
[0192] In this embodiment, the conference interface may display a sharing window, and a video window related to the speaker 400 may be superimposed on the sharing window. The video window may display a multimedia stream corresponding to the multimedia stream 2 described above.
[0193] Optionally, the conference interface may also display a video window of a participant (eg, a user of the conference terminal 402 ), and the video window may display a multimedia stream corresponding to the multimedia stream 4 .
[0194] The system of the present application is described below in conjunction with Example 1 and Example 2.
[0195] Example 1
[0196] Combined with the above Figure 5a , Figure 5b , Figure 6a The figure is a schematic diagram showing the data processing process of the system of the present application for example.
[0197] like Figure 6a As shown, the process may include the following steps:
[0198] S101 , the conference terminal 401 may send a multimedia stream 1 related to a shared window to the cloud 300 .
[0199] The application interface of the video conferencing application of the present application may include multiple windows, which may include a shared window (also called a main window) and at least one video window.
[0200] For example, the shared window may be the largest window among the above-mentioned multiple windows, and is used to display the presentation materials (such as PPT, word, webpage, etc., without limitation) shared by the speaker 400 related to the conference terminal 401 .
[0201] Figure 7 The display interface 404 of the video conference application of the conference terminal 401 is shown as an example. The display interface may include a sharing window 406 and a video window 405 .
[0202] The conference terminal 401 may send the multimedia stream 1 (eg, the multimedia stream of the above-mentioned presentation material) to be displayed in the shared window 406 to the cloud 300 .
[0203] For example, before sending the multimedia stream 1 of the shared window 406 to the display screen of the conference terminal 401 , the conference terminal 401 may forward the multimedia stream 1 to the cloud 300 so that the cloud 300 processes the multimedia stream 1 .
[0204] Of course, in some embodiments, the display interface 404 can also display the real-time video images of participants other than the speaker through multiple video windows. The video windows of participants other than the speaker can be hidden in some conference modes (such as speaker mode). Figure 7 The conference interface in the speaker mode shown is taken as an example to illustrate the technical solution of the present application. When the conference interface is in other conference modes, the method is the same and will not be repeated here.
[0205] S102 , the conference terminal 401 may send the multimedia stream 2 related to the first video window to the cloud 300 .
[0206] For example, in Figure 7 In the first video window, Figure 7The video window 405 shown is used when the speaker enables the camera in the video conference. The video conference application of the conference terminal 401 can call the camera of the conference terminal 401 (e.g., the front camera) to obtain the real-time video image of the speaker and obtain the multimedia stream. The video conference application of the conference terminal 401 can process the multimedia stream to send the processed multimedia stream or the unprocessed multimedia stream to the cloud 300. The processed multimedia stream or the unprocessed multimedia stream sent by the conference terminal 401 to the cloud 300 can be named multimedia stream 2.
[0207] The multimedia stream 2 or the multimedia stream 2 ′ after the multimedia stream 2 is processed by the cloud 300 is the multimedia stream to be displayed in the video window 405 , so the multimedia stream 2 is the multimedia stream related to the video window 405 .
[0208] S103, the cloud 300 may determine a first target position based on the multimedia stream 1, the video window parameters and the shared window parameters, wherein the first target position is a position where the first video window is to be displayed in the shared window.
[0209] The video window parameters here are first preset window parameters about the video window set by the cloud 300 (such as preset size, preset display mode, preset aspect ratio and other parameters, which are not limited here).
[0210] Similarly, the shared window parameters here are second preset window parameters about the shared window set by the cloud 300 (such as preset size, preset display mode and other parameters, which are not limited here).
[0211] In other embodiments, the video window parameter in S103 may also be each conference terminal of this video conference, for example Figure 5a , Figure 5b Conference terminal 401, conference terminal 402, and conference terminal 403 are shown with respect to preset window parameters of their respective video windows.
[0212] In other embodiments, the shared window parameter in S103 may also be each conference terminal of this video conference, for example Figure 5a , Figure 5b Conference terminal 401 , conference terminal 402 , and conference terminal 403 are shown with respect to preset window parameters of the shared windows displayed on their respective conference interfaces.
[0213] In this way, the cloud 300 can determine the display position (eg, the first target position) of the first video window on the shared window in the conference interface of each conference terminal based on the window parameters of each conference terminal and in combination with the multimedia stream 1.
[0214] In the related art, the speaker's video window is fixedly superimposed on a fixed position on the sharing window (for example, the lower right corner), but the shared content displayed in the sharing window is changing, so that the video window displayed in the fixed position will block the speaker's speech content, which is not conducive to other participants to watch the speech content.
[0215] In the embodiments of the present application, as described above, for example Figure 7 The shared window 406 shown displays multimedia stream 1, so that the display content of the shared window 406 can change. Therefore, the present application can dynamically adjust the display position of the speaker's video window 405 on the shared window 406 according to the change of the display content of the shared window 406 to reduce the obstruction of the speaker's current speech content.
[0216] In a possible implementation, when the cloud 300 executes S103, Figure 7 The cloud 300 can determine the superimposed position (such as the first target position mentioned above) of the video window 405 related to the speaker on the shared window 405 according to the position of the blank area in the shared window 406 when the multimedia stream 1 related to the shared window 406 is displayed on the shared window 406.
[0217] For example, the cloud 300 may render (or pre-render) the multimedia stream 1 related to the shared window according to the shared window parameters (such as window size, aspect ratio, etc.) to obtain rendering data (such as rendering layer) related to the display interface of the shared window. The rendering data can indicate the display position of the elements in the multimedia stream 1 on the shared window.
[0218] In addition, the cloud 300 can analyze the rendering data to identify a blank area in the display interface of the shared window corresponding to the rendering data, or a blank area that meets preset conditions (for example, at least one of the first preset condition and the second preset condition described below), and identify the location of the blank area as the first target position for the first video window to be displayed on the shared window.
[0219] Figure 8 The example shows the display interface of the shared window 501 corresponding to the rendering data related to the display interface of the shared window acquired by the cloud 300. It should be understood that Figure 8 It is just to facilitate the reader to understand that the cloud 300 analyzes the rendering data based on the blank area determined. In the application, the cloud 300 may not generate Figure 8 The display interface shown.
[0220] Since the multimedia stream 1 is composed of multiple frames of images, the cloud 300 pre-renders the multimedia stream 1, and the obtained rendering data is the rendering data of the multiple frames of images in the multimedia stream 1. Figure 8 The display interface corresponding to the rendering data of a frame of image obtained by pre-rendering a frame of image in multimedia stream 1 is shown as an example on the shared window 501. The window parameters of the shared window 501 are the same as the above-mentioned shared window parameters.
[0221] The principle of obtaining blank areas by analyzing the rendering data of other frames of multimedia stream 1 is similar to that for Figure 8 The principles described in the related embodiments are similar and will not be repeated here.
[0222] like Figure 8 As shown, the display content of the display interface of the sharing window 501 is that the window parameters conform to the above-mentioned sharing window parameters, and the display content is the rendering data corresponding to the multimedia stream 1.
[0223] The display interface of the sharing window 501 is, for example, the rendering data corresponding to a frame of an image in the PPT stream (an example of the multimedia stream 1) shared by the speaker 400. The cloud 300 analyzes the rendering data of the frame of the image and determines the blank area A1, the blank area A2, and the blank area A3 that meet the first preset condition. The blank area A1 and the blank area A2 may have an overlapping blank area X.
[0224] In this embodiment, considering that the video window of the speaker 400 needs to occupy a certain display area to be clearly visible, when the cloud 300 identifies the blank area, the first preset condition may be a blank area whose area is greater than a preset threshold. Alternatively, the cloud 300 may set a first preset condition of minimum width and minimum height for the blank area, so as to analyze the rendering data of at least one frame of an image in the multimedia stream 1, and identify the blank area that meets the first preset condition from the rendering data of one frame of an image.
[0225] In addition, in some scenarios, the multimedia screen (e.g., the document interface of a word document) corresponding to the multimedia stream 1 shared by the presenter 400 to the sharing window may have a background color or background pattern. For example, the shared word document has a background color, and the blank area identified by the cloud 300 mentioned in this article is used to represent an area without foreground elements. For example, the blank area only has the background color or background pattern of the word document, but no text in the word.
[0226] In a possible implementation, Figure 8As shown, the cloud 300 recognizes that the display interface of the shared window 501 may include blank area A1, blank area A2, and blank area A3, which meet the first preset condition. Then, the cloud 300 may select the location of the blank area that meets the second preset condition (as the target blank area) from among the multiple blank areas that meet the first preset condition as the first target location for displaying the first video window on the shared window (such as the above-mentioned shared window 501).
[0227] For example, the second preset condition may include but is not limited to at least one of the following: the blank area with the largest area; the blank area with the smallest difference between the aspect ratio of the inscribed rectangle of the blank area and the aspect ratio of the video window (which can be determined based on the video window parameters mentioned in S103).
[0228] Optionally, when the cloud 300 determines that there are multiple blank areas that satisfy both the first preset condition and the second preset condition in the display interface of the shared window corresponding to the rendering data of a frame of image in the multimedia stream 1, a blank area can be selected from the multiple blank areas as the target blank area, wherein the position of the target blank area in the interface of the shared window is the display position of the video window on the shared window (for example, the first target position), and the specific selection strategy is not limited.
[0229] S104, the cloud 300 may determine a first position of the first conference focus in the shared window.
[0230] In a possible implementation, the cloud 300 may also determine the first position of the first conference focus in the shared window (the window parameters are the above-mentioned shared window parameters) by analyzing the rendering data of the multimedia stream 1 (the specific process of obtaining the rendering data refers to the relevant introduction of S103 and is not repeated here).
[0231] The first conference focus (also referred to as the visual focus corresponding to the user of conference terminal 401 in the shared window) is also referred to as the presentation focus hereinafter.
[0232] For example, the cloud 300 may analyze the rendering data of the multimedia stream 1 to determine the position of the mouse icon in the display interface of the sharing window as the first position of the presentation focus.
[0233] Typically, the presenter may use the mouse to indicate the content of the presentation in the document in the shared window, and the cloud 300 may determine the location of the mouse as the first location of the presentation focus.
[0234] For another example, the cloud 300 may analyze the multimedia stream 2 of the speaker 400 to analyze the visual heat map of the speaker 400 in a recent period of time to determine the first position where the focus of the speech is located in the shared window.
[0235] The present application does not impose any restrictions on the strategy of the cloud 300 in determining the first position of the presentation focus on the shared window associated with the multimedia stream 1. The cloud 300 may also use other methods to identify the first position of the presentation focus in the video screen associated with the shared window displaying the multimedia stream 1.
[0236] The present application does not limit the execution order of S103 and S104, which can be executed simultaneously. In addition, when the cloud 300 determines the first target position through S103, it can also determine the first target position in combination with the first position of the first conference focus determined through S104.
[0237] Specifically, in order to make the display position (first target position) of the first video window of the speaker 400 on the shared window closer to the current speech focus of the speaker 400 on the shared window, it is convenient for the participants to observe the video images such as the expressions and movements of the speaker 400 in real time and quickly for the current speech content of the speaker.
[0238] In a possible implementation, the cloud 300 may adjust the first target position of the first video window in the shared window in real time according to the change of the position of the presentation focus corresponding to the multimedia stream 1.
[0239] In this embodiment, the cloud 300 can adjust the first target position of the first video window in the shared window in real time when detecting that the first position of the speech focus has changed. In this way, the superimposed position of the first video window of the speaker 400 on the shared window can change with the change of the position of the speech focus.
[0240] However, as mentioned above, the video window has a large area, and a certain size of blank area is required to superimpose it on the shared window. Then, when the first video window moves with the speech focus in real time, the first video window may block the content of the speech focus. In another possible implementation, the cloud 300 can adjust the first target position of the first video window in the shared window when it detects that the area where the first position of the speech focus is located has changed.
[0241] Specifically, the cloud 300 can pre-divide the shared window into regions according to the shared window parameters mentioned in S103 to obtain a plurality of preset regions. The present application does not limit the division strategy of the preset regions.
[0242] Figure 9a, Figure 9b , Fig.9c 3 and 4 respectively show the rendering data related to the display interface of the shared window obtained by the cloud 300 and the three display interfaces of the corresponding shared window 501. Figure 8 similar, Figure 9a , Figure 9b , Fig.9c It is just to facilitate the reader to understand that the cloud 300 re-determines the first target position of the first video window based on the change of the preset area to which the first position where the speech focus is located. In the application, the cloud 300 may not generate Figure 9a , Figure 9b , Fig.9c The display interface shown.
[0243] In an example, Figure 9a As shown, the cloud 300 divides the shared window 501 into three equal parts along the height direction of the shared window 501, thereby obtaining three rectangular areas, namely, the preset area B1, the preset area B2, and the preset area B3. Figure 9a In the figure, the boundaries between the three preset areas are indicated by dotted lines.
[0244] like Figure 9a As shown, the cloud 300 determines that the first position (here, position 1) of the mouse icon 502 (an example of the presentation focus) is located in the preset area B1.
[0245] In a possible implementation, the cloud 300 may identify a target blank area (eg, Figure 9a The target blank area 601a) shown is located in the sharing window 501, which serves as the first target position for displaying the first video window on the sharing window 501.
[0246] In a possible implementation, when determining the target blank area, the cloud 300 may also combine the video window parameters mentioned in S103 to determine the target blank area in the blank areas that meet the preset conditions. Figure 9a The preset area B1 shown therein identifies a target blank area 601a whose size matches the video window parameters (eg, the same size or the same ratio).
[0247] In another possible implementation, the cloud 300 may select a blank area at least partially located in the preset area B1 as the target blank area from among the identified blank areas that meet the preset conditions.
[0248] For example, refer to Figure 8, the blank areas meeting the preset conditions identified by the cloud 300 are blank area A1, blank area A2, and blank area A3, and only part of the blank area A1 is located Figure 9a The preset area B1 shown in the figure, then the blank area A1 is located Figure 9a The partial area in the preset area B1 shown is the area where the first video window can be set. The cloud 300 can combine the video window parameters to determine the target blank area in the blank area A1 and in the blank area of the preset area B1 as the area where the first video window is located.
[0249] For the rendering data corresponding to the multimedia stream 1, the display interface of the corresponding shared window 501, Figure 9a The example shows the rendering data corresponding to a frame of multimedia stream 1 and the display interface of the corresponding shared window.
[0250] like Figure 9b As shown, it shows the rendering data of another frame of multimedia stream 1 and the display interface of the corresponding shared window. Figure 9b As shown, the cloud 300 can recognize that the first position of the speech focus (specifically the mouse icon 502) in the shared window 501 is position 2, and the position 2 belongs to the preset area B3. Figure 9a If the preset area B1 to which position 1 belongs changes, the cloud 300 can redefine the target blank area (here 601b) in the preset area B3 as the first target position for displaying the first video window on the shared window 501.
[0251] About Figure 9b The method and principle for determining the target blank area in the preset area B3 shown in FIG. Figure 9a The introduction is similar to that of , so I will not repeat it here.
[0252] In another possible implementation, when the cloud 300 determines the first target position of the video window in the shared window based on the first position of the speech focus, the first target position may not be determined in the preset area where the first position is located. On the contrary, the cloud 300 may determine the first target position in other preset areas that are located in the shared window and outside the preset area where the first position of the speech focus is located. Moreover, the first target position may be the position of the target blank area, or may not be a blank area (for example, the first video window may partially block the speech content, but will not block the speech content related to the current speech focus), so as not to affect the information display of the area where the speech focus is located, and will not cause the occlusion of the area where the speech focus is located.
[0253] For example, Figure 9a and Figure 9b In the example of FIG. 1 , the cloud 300 may determine the first target location within the preset area B2, where the preset area B2 is Figure 9a The adjacent area of the preset area B1 to which the position 1 of the speech focus in FIG. The preset area B2 is Figure 9b The adjacent area of the preset area B3 to which the speech focus is located in position 2.
[0254] In another possible implementation, when the cloud 300 determines that there is no blank area that meets the preset conditions in the area where the speech focus is located, the cloud 300 may select a target position farthest from the first position where the speech focus is located in an adjacent area of the preset area where the speech focus is located, or in other preset areas other than the preset area where the speech focus is located, and determine the first target position to display the first video window in the preset area other than the preset area where the speech focus is located based on the target position. In this way, when the first video window is displayed at the determined first target position, the first video window is less likely to affect the display of the content in the area where the speech focus is located.
[0255] For example, Fig.9c 5 shows the rendering data corresponding to a frame of image in multimedia stream 1 and the display interface of the corresponding shared window 501. Fig.9c As shown, the cloud 300 determines that the first position where the presentation focus (here is the mouse icon 502) is located in the shared window 501 is position 3, and determines that position 3 belongs to the preset area B1. However, the cloud 300 does not identify the target blank area that meets the preset conditions in the preset area B1. For example, the area of the blank area in the preset area B1 does not meet the area requirement of the first video window. Then, the cloud 300 can determine that the target position with the best distance from position 3 is position 4 (specifically, the lower right corner vertex of the preset area B2) in the preset area B2 adjacent to the preset area B1. Then, the cloud 300 can use position 4 as a reference to determine the target area 601c for window overlay shown in the gray dotted box in the preset area B2, wherein the vertex of the lower right corner of the target area 601c coincides with position 4. Among them, the size of the target area 601c is the same as the window size in the video window parameters mentioned in S103, or the same proportion.
[0256] In this way, when there is not enough blank area in the area where the speech focus is located to overlay the video window, a blank area can be selected from the adjacent area to overlay the video window, and the occlusion of the first video window on the content near the speech focus can be reduced as much as possible.
[0257] In an embodiment of the present application, when the focus area of the speech changes, the cloud 300 can dynamically adjust the display position of the video window (here is the first video window) on the shared window, so that the video window of the speaker 400 can change its position as the focus area of the speech material of the speaker 400 on the shared window changes.
[0258] The embodiment of the present application can prompt a change in the focus of the speech by dynamically changing the position of the video window.
[0259] S105, the cloud 300 may mirror at least one frame of the multimedia stream 2 based on the positional relationship between the first target position and the first position to obtain a multimedia stream 2'.
[0260] S107, the cloud 300 synthesizes the multimedia stream 1 and the multimedia stream 2' according to the first target position to obtain a first target multimedia stream.
[0261] The following is a detailed description of S105 and S107:
[0262] Among them, the first target position and the rendering data of a frame image (for example, named as the first frame image) corresponding to the first position in multimedia stream 1, and the rendering data of a frame image (for example, named as the second frame image) in multimedia stream 2 that is mirrored here correspond to each other, so each second frame image in multimedia stream 2 can also correspond to each first target position.
[0263] The first frame image and the second frame image may correspond to each other in the form of images displayed at the same time in the conference interface, or the timestamps of the first frame image and the second frame image may correspond to each other.
[0264] Among them, the conference interface includes a sharing window. When the first frame image is displayed on the sharing window, the video window of the speaker 400 displays the second frame image corresponding to the first frame image (or the third frame image after mirroring the second frame image), wherein the display position of the video window on the sharing window is the first target position.
[0265] The cloud 300 synthesizes the image of the video window of the speaker 400 (also called the video window image) above the image of the shared window (also called the shared window image) according to the first target position (for example, before layer synthesis). In some scenarios, the video screen of the multimedia stream 2 of the speaker 400 is captured by the camera of the conference terminal 401 of the speaker 400, so the image captured by the camera is mirrored to the actual subject. For example, when the speaker 400 looks at the right side of the screen of the conference terminal 401, the line of sight of the person in the image of the speaker 400 (for example, a frame of image in the multimedia stream 2) captured by the camera of the conference terminal 401 is looking at the left side of the screen.
[0266] The inventors consider that if the second frame image (also called the video window image) rendered according to the video window parameters is directly synthesized onto the first frame image rendered according to the shared window parameters for the multimedia stream 2 according to the first target position corresponding to the second frame image of each frame in the multimedia stream 2, then the synthesized image will be displayed on the first frame image. Figure 5a , Figure 5b In the conference terminal 402 and the conference terminal 403 shown in the figure, the speaker's sight line in the video window image of the speaker 400 is looking towards the left side of the screen. If the focus of the speech is located at the right side of the screen in the synthesized image, then in the conference interface, the speaker's sight line direction in the video window of the speaker 400 is opposite to the position of the focus of the speech in the shared window, thereby affecting the viewing experience of other participants on the conference interface and reducing the sense of immersion.
[0267] To this end, in this embodiment, after the cloud 300 determines the superimposed position of the video window (e.g., the first target position mentioned above) and the position of the speech focus (e.g., the first position mentioned above), the cloud 300 can determine whether to mirror the frame image in the multimedia stream 2 based on the relative position relationship between the speech focus position and the superimposed position of the video window, so that at least one frame image in the processed multimedia stream 2', when displayed at the first target position of the shared window through the video window of the speaker 400, the line of sight of the speaker 400 in the frame image is looking at the speech focus in the shared window image.
[0268] In some scenarios, when the speaker 400 is giving a speech on a shared document, his line of sight is usually towards the speech focus position (the first position mentioned above), so the cloud 300 can default the speech focus position to the viewing area of the speaker 400. However, as described above, since the video image of the speaker is captured by the front camera of the conference terminal 401 of the speaker 400, the video image and the actual object are mirrored, so that the user's line of sight in the video window may not be towards the direction where the speech focus is located, but in the opposite direction.
[0269] In one example, the cloud 300 can symmetric the presentation focus corresponding to the first frame image in multimedia stream 1 about the midline in the height direction of the shared window to obtain a mirror focus, and then determine whether to mirror the second frame image corresponding to the first frame image in multimedia stream 2 based on the relative position relationship between the mirror focus position and the superimposed position of the video window (such as the above-mentioned first target position). After mirroring at least one second frame image in multimedia stream 2, multimedia stream 2' can be obtained.
[0270] Specifically: when the cloud 300 determines that the mirror focus position and the superimposed position of the video window (such as the first target position mentioned above) are on the same side of the midline in the height direction of the shared window, the cloud 300 can perform mirror processing on the corresponding second frame. When the cloud 300 determines that the mirror focus position and the superimposed position of the video window are on different sides of the midline in the height direction of the shared window, the cloud 300 does not need to perform mirror processing on the corresponding second frame.
[0271] Combine the following Fig.10a and Fig.10b The process of the cloud 300 executing S105 and S107 is described in the scenario.
[0272] Fig.10a (1) exemplarily shows the display interface of the shared window 501 corresponding to the rendering data of the first frame image acquired by the cloud 300. It should be understood that Fig.10a (1) It is only for the convenience of readers to understand that the cloud 300 analyzes the rendering data to determine whether to mirror the second frame image corresponding to the first frame image in the multimedia stream 2. In the application, the cloud 300 may not generate Fig.10a (1) is the display interface of the sharing window 501 shown.
[0273] Fig.10a (2) exemplarily shows the rendering data of the second frame image corresponding to the first frame image obtained by the cloud 300 (rendered according to the video window parameters in S103), and the display interface of the corresponding video window 602, which is also called the video screen 6021 of the second frame image. Fig.10a As shown in (2), the line of sight of the image of the speaker 400 in the video screen 6021 is indicated by an arrow.
[0274] like Fig.10a As shown in (1), the center line of the display interface of the shared window 501 in the height direction is the center axis 0, and the center axis 0 divides the display interface of the shared window 501 into a left part and a right part.
[0275] The cloud 300 can determine the mirror position of the presentation focus D1 in the display interface of the sharing window 501, and the mirror position of the mirror focus D2 symmetrical to the central axis 0 in the sharing window 501. Fig.10a (1) and Fig.10a As shown in (2), the sight line of the person in the video screen 6021 is toward the mirror focus D2.
[0276] In addition, if Fig.10a (1) and Fig.10a As shown in (2), the superimposed position of the video window 602 on the shared window 501 (e.g., the first target position described above) is represented by a dotted frame D3. The cloud 300 can determine that the dotted frame D3 and the mirror focus D2 are located on the same side of the central axis 0 (both on the right side), then the cloud 300 can mirror the second frame image in the multimedia stream 2 corresponding to the video image 6021 to obtain a third frame image, wherein the video image corresponding to the rendering data of the third frame image after rendering according to the video window parameters is Fig.10a (3) is a display interface of the video window 602', which is also called the video screen 6021' of the third frame.
[0277] like Fig.10a As shown in (3), the cloud 300 can perform layer synthesis of the rendering data of the third frame (corresponding to the video screen 6021') and the rendering data of the corresponding first frame in the multimedia stream 1 (corresponding to the display interface of the shared window 501) according to the first target position shown in the dotted box D3, thereby obtaining synthesized rendering data. The multimedia screen corresponding to the synthesized rendering data can be shown as Fig.10a (3) as shown. Fig.10a As shown in (3), the position of the dotted frame D3 displays a video screen 6021', which is the display interface of the video window 602'. Fig.10a In (3), the sight line of the person in the video screen 6021' is looking at the speech focus D1. This makes the sight line of the speaker in the video window superimposed on the shared window correspond to the direction of the speech focus, so that the user's sight line can be directed toward the speech focus, presenting a video screen of the speaker looking at the speech focus while giving a speech.
[0278] In this way, the cloud 300 can synthesize the two corresponding frames of images in multimedia stream 1 and multimedia stream 2' according to the corresponding first target position, thereby obtaining a first target multimedia stream. The two corresponding frames of images can be images displayed at the same time, or images with the same timestamp.
[0279] In another example, Fig.10b(1) exemplarily shows the display interface of the shared window 501 corresponding to the rendering data of the first frame image acquired by the cloud 300. It should be understood that Fig.10b It is just to facilitate the reader to understand that the cloud 300 analyzes the rendering data to determine whether to mirror the second frame image corresponding to the first frame image in the multimedia stream 2. In the application, the cloud 300 may not generate Fig.10b The display interface of the sharing window 501 is shown.
[0280] Fig.10b (2) exemplarily shows the rendering data of the second frame image corresponding to the first frame image obtained by the cloud 300 (rendered according to the video window parameters in S103), and the display interface of the corresponding video window 602, which is also called the video screen 6021 of the second frame image. Fig.10b As shown in (2), the line of sight of the image of the speaker 400 in the video screen 6021 is indicated by an arrow.
[0281] like Fig.10b As shown in (1), the center line of the display interface of the shared window 501 in the height direction is the center axis 0, and the center axis 0 divides the display interface of the shared window 501 into a left part and a right part.
[0282] like Fig.10b As shown in (1), the cloud 300 can determine the mirror position of the mirror focus D2 of the presentation focus D1 symmetrical with respect to the central axis 0 in the display interface of the sharing window 501. Fig.10b (1) and Fig.10b As shown in (2), the sight line of the person in the video screen 6021 is toward the mirror focus D2.
[0283] In addition, if Fig.10b (1) and Fig.10b As shown in (2), the superimposed position of the video window 602 on the shared window 501 (e.g., the first target position described above) is represented by a dotted frame D3. The cloud 300 can determine that the dotted frame D3 and the mirror focus D2 are located on different sides of the central axis 0 (respectively on the left and right sides of the central axis 0) in the shared window 501, and the cloud 300 can not perform mirroring on the second frame image in the multimedia stream 2 corresponding to the video screen 6021.
[0284] like Fig.10b As shown in (3), the cloud 300 can perform layer synthesis of the rendering data of the second frame (corresponding to the video screen 6021) and the rendering data of the corresponding first frame in the multimedia stream 1 (corresponding to the display interface of the shared window 501) according to the first target position shown in the dotted box D3, thereby synthesizing the two frames into one frame. Fig.10bAs shown, a video screen 6021 is displayed at the position of the dotted frame D3, and the video screen 6021 is the display interface of the video window 602. Fig.10b In (3), the sight line of the person in the video screen 6021 is looking at the speech focus D1. This makes the sight line of the speaker in the video window superimposed on the shared window correspond to the direction of the speech focus, so that the user's sight line can be directed toward the speech focus, presenting a video screen in which the speaker is looking at the speech focus and giving a speech.
[0285] In this way, the cloud 300 can synthesize the two corresponding frames of images in multimedia stream 1 and multimedia stream 2' according to the corresponding first target position, thereby obtaining a first target multimedia stream. The two corresponding frames of images can be images displayed at the same time, or images with the same timestamp.
[0286] In a possible implementation, when cloud 300 synthesizes multimedia stream 1 and multimedia stream 2', cloud 300 may also perform preset processing on image frames in multimedia stream 2' to be displayed in the video window, so that the images displayed in the video window in the first target multimedia stream only include images of people.
[0287] Continue to refer to Fig.10b , 300 cloud can Fig.10b (2) The images other than the speaker in the video screen 6021 shown in FIG. 6 (e.g., foreground elements, background elements, etc.) are processed transparently, and the window border of the video window 601 is hidden. Then, the processed video screen 602" is synthesized with a first frame image in the multimedia stream 1 (e.g., rendering data after rendering the first frame image according to the shared window parameters) at the position shown in the dotted box D3 to obtain a frame image in the first target multimedia stream, which displays the video screen 602".
[0288] It should be understood that the process of cutting out the avatar of the user (eg, speaker) giving a speech in the video window is not limited to the above-mentioned scheme of setting transparency, and can be implemented by any cutting out scheme, which is not limited here.
[0289] In this way, the video screen in the video window in the conference interface of the video conference can visually display only the portrait.
[0290] In addition, when the cloud 300 determines the mirror focus position by symmetry with respect to the center line in the height direction of the shared window at the first position where the speech focus is located, the center line is not limited to Fig.10a and Fig.10bThe centerline 0 shown may be a line in the height direction of any shared window, and the line is not limited to be parallel to the height direction, and may have an angle within 45 degrees. The specific implementation principle is similar and will not be repeated here.
[0291] S108, the cloud 300 sends the first target multimedia stream to each conference terminal.
[0292] Among them, the cloud 300 can synchronously send the first target multimedia stream to the conference terminal 401, the conference terminal 402, and the conference terminal 403.
[0293] S109a, the conference terminal 401 can display a conference interface based on the first target multimedia stream.
[0294] Each frame image in the first target multimedia stream includes a shared window image and a video window image of the speaker 400. The display position of the video window image in the shared window image is the first target position, the shared window image corresponds to the frame image in multimedia stream 1, and the video window image corresponds to the frame image in multimedia stream 2'. The above two frame images in the two sets of multimedia streams are corresponding frame images, for example, the timestamps are the same or correspond to each other.
[0295] The window parameters (eg, window size) of the video window image are the video window parameters in S103 above, and the window parameters of the shared window image are the shared window parameters in S103 above.
[0296] The conference interface may be, for example, the above Fig.10a (3) The conference interface shown is in speaker mode (video windows of participants other than the speaker are not displayed), and the conference interface may include a display interface of a shared window 501, wherein the first target position (the position of the dotted box D3) in the display interface of the shared window 501 displays the display interface of the video window 602' of the speaker 400.
[0297] For example, the conference interface can be the above Fig.10b (3) or Fig.10b (4) is a display interface of the shared window 501 (also called conference interface).
[0298] In one possible implementation, the first shared window parameter of the shared window in the conference interface of the conference terminal 401 may differ from the shared window parameter referenced by the cloud 300 in S103 (for example, a different window size), and / or the first video window parameter of the speaker's video window in the conference interface of the conference terminal 401 may differ from the first video window parameter referenced by the cloud 300 in S103 (for example, a different window size). In this case, the conference terminal 401 may adjust the display ratio of at least one frame of the received first target multimedia stream (for example, a shared window image and a video window image located at the first target position) based on the first shared window parameter and / or the first video window parameter, so that the conference interface displayed by the conference terminal 401 matches the window parameters of the conference terminal 401.
[0299] S109b, the conference terminal 402 can display a conference interface based on the first target multimedia stream.
[0300] The execution principle of this step is the same as that of S109a, and will not be repeated here.
[0301] S109c, the conference terminal 403 can display a conference interface based on the first target multimedia stream.
[0302] The execution principle of this step is the same as that of S109a, and will not be repeated here.
[0303] In this way, the conference terminals 401, 402, and 403 participating in the conference can display the same conference interface.
[0304] It should be understood that in the above example 1 Figure 6a The process shown can be executed during document sharing or desktop sharing at conference terminal 401, so that the video screen of the speaker in the conference interface corresponding to the first target multimedia stream is matched in real time with the video screen corresponding to the document presented by the speaker.
[0305] In the embodiment of the present application, when the cloud 300 synthesizes the multimedia stream 1 related to the shared window and the multimedia stream 2 related to the video window of the presenter, it can refer to the positional relationship between the superimposed position of the video window on the shared window (such as the first target position mentioned above) and the position of the speech focus (such as the first conference focus mentioned above) (such as whether they are on the same side of the center line of the shared window) to determine whether to perform mirror processing on the frame image in the multimedia stream 2 to be displayed in the video window, so that in the first target multimedia stream synthesized after the processed multimedia stream 2' and the multimedia stream 1, the line of sight of the presenter in the video window can match the position of the speech focus in the shared window. The line of sight of the portrait of the person in the video window in each frame image of the conference interface is coordinated with the speech focus in the shared window, thereby enhancing the immersiveness of the video conference.
[0306] In addition, in the first target multimedia stream of the present application, the superimposed position of the video window image on the shared window image is determined by the cloud 300 based on the blank area of the video window image (excluding the area of the foreground element) and / or based on the position of the speech focus, so that the video window image will not block the current speech focus in the speech document in the shared window. When the conference terminal of the participant displays the conference interface corresponding to the first target multimedia stream, there is no need to frequently adjust the display position of the speaker's video window due to the video window blocking the speech content.
[0307] In the embodiment of the present application, based on the positional relationship between the overlay position and the speech focus, if the two are on the same side of the center line, the cloud performs mirror processing on the video frame of the video window. When the portrait's line of sight is not looking at the speech focus, the video frame in the speaker's video is mirrored to make him look at the speech focus. Thus, visually, the speaker is always looking at the position of the speech focus, reducing the sense of incoordination of the picture in the shared window and enhancing the immersion of the participants.
[0308] Example 2
[0309] Combined with the above Figure 5a , Figure 5b , Figure 6a , Figure 6b The figure is a schematic diagram showing the data processing process of the system of the present application for example.
[0310] like Figure 6b As shown, the process may include: Figure 6a S101 to S106 shown (refer to the introduction of Example 1 for details, which will not be repeated here), and include the following steps:
[0311] S201, the conference terminal 402 can send the multimedia stream 4 related to the second video window to the cloud 300.
[0312] The execution principle of this step is the same as that of S102 in Example 1. When the camera of the conference terminal 402 is started, it can also capture the video image of the user of the conference terminal 402 (referred to as user A) and send it to the cloud 300 as the multimedia stream 4 here.
[0313] Here, the display window of the video screen related to user A in the conference interface is called the second video window. In Example 1, the display window of the video screen related to speaker 400 is called the first video window. They are both video windows.
[0314] S202, the cloud 300 detects at least one first moment related to human voice in the multimedia stream 4.
[0315] In this example, user A of conference terminal 402 is used as a speaker in the conference process to illustrate the process of overlaying the video windows of participants who are speaking other than the speaker on the shared window. In some scenarios, there may be multiple speakers other than the speaker, and the example is not limited to the user of conference terminal 402 as the speaker. The specific principles are similar and will not be repeated here.
[0316] During the video conference, in addition to the above-mentioned speaker 400, other participants can also speak. For example, in a video conference, the cloud 300 can perform voice recognition on the multimedia stream 4 from the conference terminal 402 (for example, after removing noise), thereby determining at least one second moment when there is a human voice in the multimedia stream 4.
[0317] Among them, the first moment includes the second moment.
[0318] In one example, the cloud 300 may use the at least one second moment as the at least one first moment related to the human voice. In this way, the second video window related to the user A who is speaking can be displayed on the shared window during the time when the user A is speaking (e.g., the at least one second moment mentioned above). After the user A finishes speaking, the second video window is restored to its original position, for example Figure 3 The position of the video window list shown (eg, the position of the window 103 of participant 2) is not limited to the original position.
[0319] In another example, the cloud 300 may also identify at least one second moment and a preset duration adjacent to the second moment and located after the second moment as at least one first moment related to human voice. In this way, the second video window related to the speaking user A can be displayed on the shared window during the period when user A is speaking (such as the at least one second moment mentioned above) and within the preset duration after the end of the current speech of user A. After the preset duration after the end of the current speech of user A, the second video window related to user A is no longer displayed on the shared window.
[0320] S203, the cloud 300 may determine a second target position associated with the at least one first moment based on the multimedia stream 1 (received through S102), the video window parameters, and the shared window parameters.
[0321] The video window parameters and shared window parameters here are the same as the video window parameters and shared window parameters in S103.
[0322] The second target position is the position where the second video window is to be displayed in the shared window at the at least one first moment.
[0323] The execution principle of S203 is the same as the principle of the specific implementation process of S103 described in Example 1. For details, please refer to the introduction of Example 1 and will not be repeated here.
[0324] The difference between S203 and S103 is that in S203, only the rendering data of the first frame image corresponding to the above-mentioned at least one first moment in multimedia stream 1 (rendering data after rendering according to the above-mentioned shared window parameters) needs to be analyzed, and there is no need to analyze the rendering data of each frame image in multimedia stream 1, so as to determine the second target position to be displayed on the shared window at the second video window at at least one first moment.
[0325] S204, the cloud 300 may determine a second position of a second conference focus associated with the at least one first moment in the shared window.
[0326] Among them, the second conference focus is also called the visual focus of the speaker (also called the corresponding visual focus of the user of the second electronic device in the shared window), wherein the second position of the visual focus can be the first position where the above-mentioned first conference focus (also called the speech focus) is located, or, the second position of the visual focus can also be the first target position of the first video window related to the speaker to be displayed on the shared window, or, the conference terminal 402 can also move the mouse on the shared window, but the icon of this mouse will not be displayed on the conference interface of other conference terminals (such as conference terminal 401, conference terminal 403), then the position of the mouse controlled by the speaker on the shared window corresponding to the conference terminal 402 can also be the position of the visual focus of the second speaker here.
[0327] That is to say, the speaker's visual focus can be the current speech focus or the speaker's video window. There is no specific restriction on the speaker's focus position in the shared window.
[0328] For example, at least one first moment is the moment when user A speaks. Since the second video window of user A only needs to be superimposed on the shared window during the speech of user A, when determining the second conference focus, the cloud 300 only needs to analyze the rendering data of the frame image (also called the fourth frame image) related to at least one first moment in the multimedia stream 1 to determine the second position of each second conference focus of at least one first moment in the shared window corresponding to each fourth frame image.
[0329] Similar to the above example 1, the cloud 300 can also determine the second target position of the second video window on the shared window in combination with the second position of the second conference focus, so that when the position or area of the second conference focus changes, the cloud 300 can dynamically adjust the display position of the video window (here, the second video window) on the shared window, so that the video window of the speaking user A can follow the changes in the visual focus of user A (for example, the change in the area where the speaker 400's speech focus on the speech material on the shared window, or the change in the area where the speaker 400's video window is located, or the change in the area where the mouse operated by user A on the shared window belongs to). The specific principle is similar to the introduction of S103 and S104 in example 1, and will not be repeated here.
[0330] S205, the cloud 300 may mirror at least one frame of image related to the at least one first moment in the multimedia stream 4 based on the positional relationship between the second target position and the second position, to obtain a multimedia stream 4'.
[0331] Among them, the second target position and the second position related to at least one first moment, the corresponding at least one frame image in the multimedia stream 1 (for example, named as the fourth frame image), and the at least one frame image in the multimedia stream 4 that is mirrored here (for example, named as the fifth frame image) are mutually correspondingly associated through the above-mentioned first moment.
[0332] The execution principle of this step is the same as the execution principle of S105 described in Example 1.
[0333] Below Fig.11 Let's take an example to illustrate:
[0334] Fig.11 (1) exemplarily shows the display interface of the shared window 501 corresponding to the rendering data of the fourth frame image obtained by the cloud 300. It should be understood that Fig.11 (1) It is only for the convenience of readers to understand that the cloud 300 analyzes the rendering data to determine whether to mirror the fifth frame corresponding to the fourth frame in the multimedia stream 4. In the application, the cloud 300 may not generate Fig.11 (1) is the display interface of the sharing window 501 shown.
[0335] For example, Fig.10b (4), in Fig.11 In FIG. 1 , a schematic diagram of the cloud 300 superimposing and displaying the video screen 6021” related to the first video window at the superimposed position shown in the dotted frame D3 is shown. The specific process can be referred to in Fig.10b The description is not repeated here.
[0336] like Fig.11 As shown, the center line of the display interface of the sharing window 501 in the height direction is the center axis 0, and the center axis 0 divides the display interface of the sharing window 501 into a left part and a right part.
[0337] exist Fig.11 In the scenario, the second conference focus is taken as the speech focus D1 as an example.
[0338] like Fig.11 As shown in (1), the cloud 300 can determine the second position of the mirror focus D5 of the presentation focus D1 symmetrical with respect to the central axis 0 in the display interface of the sharing window 501.
[0339] In addition, if Fig.11 As shown in (1), the cloud 300 can also determine the second target position D4 where the mirror focus D5 and the second video window are superimposed on the shared window 501.
[0340] like Fig.11 (1), the cloud 300 can determine that the mirror focus D5 and the second target position D4 (shown in a dotted box) are located on the same side of the central axis 0 (both on the left side), then the cloud 300 can Fig.11 The video screen 6031 of the display interface 603 of the second video window shown in (2) is mirrored to the corresponding fifth frame in the multimedia stream 2 to obtain the sixth frame, wherein the video screen corresponding to the rendering data of the sixth frame after rendering according to the video window parameters is Fig.11 (3) is a display interface of the video window 603', which is also called the video screen 6031' of the sixth frame.
[0341] like Fig.11 As shown in (3), the cloud 300 can connect the rendering data of the sixth frame (corresponding to the video screen 6031') and the rendering data of the corresponding fourth frame in the multimedia stream 1 (corresponding to the display interface of the shared window 501) according to the following example: Fig.11 (1) The second target position D4 shown in FIG. 1 is subjected to layer synthesis, and Fig.10b The rendering data of the second frame image in the multimedia stream 2' shown (the corresponding video screen is 6021) and the rendering data of the corresponding fourth frame image in the multimedia stream 1 (corresponding to the display interface of the shared window 501) are layered according to the first target position shown in the dotted box D3, thereby synthesizing the three frames of image into one frame of image.
[0342] like Fig.11 As shown in (3), Fig.11The second target position D4 shown in (1) displays a video screen 6031' of user A who is speaking. The video screen 6031' is the display interface of the video window 603'. Fig.11 (3), the sight direction of the person (eg user A) in the video screen 6031' is looking towards the speech focus D1. Fig.11 The position of the dotted box D3 shown in (1) is Fig.11 (3) shows a video screen 602", which is the display interface of the first video window of the speaker 400. Fig.10a In (3), the sight line of the person (here, the speaker 400) in the video screen 602" is also directed toward the speech focus D1.
[0343] In this way, Fig.11 (3) In the display interface of the synthesized conference interface presented, the sight line direction of user A in the video window of user A who is speaking and the sight line direction of speaker 400 in the video window of speaker 400 who is giving a speech can both look toward the speech focus D1. Alternatively, when the cloud 300 uses the second position of the second conference focus as the first target position of the first video window, when the conference terminal displays the conference interface based on the synthesized second target multimedia stream, in the conference interface, the sight line direction of user A in the video window of user A who is speaking can be aligned with the sight line direction of speaker 400 in the video window of speaker 400 who is giving a speech, so that the sight line direction of the speaker in the video window superimposed in the shared window is opposite to the direction of the speech focus, thereby presenting a conference screen of two users in conversation.
[0344] After S105 and S205 , the cloud 300 may execute S207 .
[0345] S207, the cloud 300 may synthesize the frames related to at least one first moment in multimedia stream 1, multimedia stream 2' (obtained through S105), and multimedia stream 4' according to the first target position and the second target position to obtain a second target multimedia stream.
[0346] The execution principle of this step is similar to that of S107 in Example 1, with the only difference being that here the three frames of images of each of the three multimedia streams are synthesized (e.g., layer synthesis, hardware synthesis, etc.), and, during synthesis, only the frame images related to at least one first moment in the multimedia stream 4' need to be synthesized to realize the synthesis of the three multimedia streams and obtain a second target multimedia stream.
[0347] In this way, when the conference terminal displays the conference interface based on the second target multimedia stream, the video window of the speaker 400 continues to be displayed in the shared window within the conference interface. However, for other participants except the speaker, the shared window will only display the video window of the speaker's user (for example, user A) when the participant speaks. After the speaker finishes speaking or a period of time after the speaker finishes speaking, the user's video window will no longer be displayed on the shared window.
[0348] S208, the cloud 300 sends the second target multimedia stream to each conference terminal.
[0349] The cloud 300 may synchronously send the second target multimedia stream to the conference terminal 401 , the conference terminal 402 , and the conference terminal 403 .
[0350] S209a, the conference terminal 401 can display a conference interface based on the second target multimedia stream.
[0351] Each frame image in the second target multimedia stream includes a shared window image and a video window image of the speaker 400, and the frame image corresponding to at least one of the first moments in the second target multimedia stream may include a shared window image and a video window image of the speaker 400 and a video window image of user A who is speaking.
[0352] The display position of the video window image of the speaker 400 in the shared window image is the first target position.
[0353] The display position of the video window image of user A in the shared window image is the second target position.
[0354] The shared window image corresponds to the frame image in multimedia stream 1, the video window image of the speaker 400 corresponds to the frame image in multimedia stream 2', and the video window image of the speaking user A corresponds to the frame image related to at least one first moment in multimedia stream 4'.
[0355] The frame images in each window displayed in the same conference interface are corresponding frame images, for example, the timestamps are the same or the timestamps correspond to each other.
[0356] The window parameters (eg, window size) of the video window image are the video window parameters in S103 and S203 above, and the window parameters of the shared window image are the shared window parameters in S103 and S203 above.
[0357] S209b, the conference terminal 402 can display a conference interface based on the second target multimedia stream.
[0358] The execution principle of this step is the same as S209a and will not be repeated here.
[0359] S209c, the conference terminal 403 can display a conference interface based on the second target multimedia stream.
[0360] The execution principle of this step is the same as S209a and will not be repeated here.
[0361] In an embodiment of the present application, other participants can see the video images of the speaker and the speaker at the same time in the shared window, creating an interactive atmosphere between the speaker and the speaker, which can enhance the participation of other participants in the current question content and improve the sense of immersion.
[0362] In this example 2, the process from S103 to S105 and the process from S203 to S205 can be executed in parallel. For example, S103 and S103 can be executed in parallel, S104 and S204 can be executed in parallel, and S105 and S205 can be executed in parallel. In this way, the shared window can superimpose the video window of the speaker, and when other participants are speaking, the video window of the speaker who is speaking (i.e., the participant who is speaking) can be superimposed. The video screen of the speaker and the speaker having a conversation can be presented on the shared window, and the content of the conversation is for the document displayed on the shared window (such as the focus of the speech). In addition, the cloud 300 superimposes the video screen of the speaker on the video window. In this way, other participants can see the questioner and the speaker at the same time, and present the interactive atmosphere between the questioner and the speaker in the conference interface, which can enhance the sense of participation of other participants in the current question content and enhance the sense of immersion.
[0363] It should be noted that the above Figure 5a , Figure 5b , Figure 6a , Figure 6b The same reference numerals denote the same objects. Therefore, the reference numerals of the drawings are not explained one by one. The reference numerals not mentioned in the above drawings can be referred to in Figure 5a , Figure 5b , Figure 6a , Figure 6b The explanations of the same figure marks mentioned in the figure will not be repeated here.
[0364] It should be noted that the above Figure 8 , Figure 9a , Figure 9b , Fig.9c , Fig.10a , Fig.10b , Fig.11 The objects represented by the same reference numerals have the same or similar meanings, and are only marked for objects with the same physical meaning in different scenarios or different embodiments. Therefore, the reference numerals of the drawings are not explained one by one. The reference numerals not mentioned in the above drawings can be referred to in Figure 8, Figure 9a , Figure 9b , Fig.9c , Fig.10a , Fig.10b , Fig.11 The explanations of the same figure marks mentioned in the figure will not be repeated here.
[0365] Example 3-1
[0366] In a possible implementation, in the above-mentioned Example 1 or Example 2, the cloud 300 may also obtain information such as the user's gestures, actions, or expressions in the video window. When it is determined that a specific gesture (such as a heart gesture, etc.) or a specific action (such as raising a hand, etc.) or a specific expression (such as frowning, etc.) exists in the multimedia screen corresponding to the multimedia stream related to the video window, a process similar to that of the above-mentioned Example 2 may be adopted to superimpose the video window related to the video screen with the specific gesture, specific action, or specific expression on the shared window for display.
[0367] Example 3-2
[0368] In a possible implementation, combined with the above example 1 or example 2 or example 3-1, the cloud can also determine the target animation corresponding to the specific gesture, specific action or specific expression when determining that there is a specific gesture (such as a heart gesture, etc.) or a specific action (such as a hand-raising action, etc.) or a specific expression (such as a frown, etc.) in the multimedia screen corresponding to the multimedia stream related to a certain video window; and when the cloud 300 synthesizes the target video stream (such as the above first target video stream or the above second target video stream), it can refer to the superimposed position corresponding to the video window with the corresponding gesture, action or expression (such as the above second target position), and optionally refer to the superimposed position corresponding to the video window of the speaker 400 (such as the above first target position), to synthesize the above animation in the target video stream. In this way, when each conference terminal displays the conference interface corresponding to the target video stream, the animation can be displayed in the shared window in the conference interface, and the display position of the animation is related to the superimposed position of the video window related to the animation.
[0369] For example, if user A (but not limited to the user of the conference terminal who spoke, and can also be the user of the conference terminal who did not speak) made a heart gesture in the video conference, the cloud (or the conference terminal 402 corresponding to the user A) can determine the target animation corresponding to the heart gesture, which indicates the emission of love. Then, the cloud can set the starting display position of the target animation to the vicinity of the superimposed position where the second video window related to the user A is located, and set the animation direction of the target animation (for example, the direction of emitting love) to the direction from the second target position where the second video window is located to the first target position where the first video window of the speaker 400 is located. In this way, in the conference interface of this conference, the conference interface displayed by each conference terminal can not only display the video window of the speaker, but also display the video window corresponding to the participant who made the heart gesture, and the conference interface also displays the target animation corresponding to the heart gesture, and the heart-shaped pattern of the heart of the target animation starts to be emitted from the position of the second video window of the user A who made the heart gesture, and is emitted in the direction of the first video window of the speaker 400, so as to express the interaction between the participants in the conference interface in an animated manner.
[0370] Example 3-3
[0371] In order to make the speaker 400 understand the interest level of the participants of this conference in the current speech content of the speaker 400.
[0372] The conference system of the present application may also include an electronic device with a display screen (also called a third-party monitor), which can interact with the cloud 300. The electronic device can be any type of electronic device with a display screen, such as a computer, a mobile phone, etc., and is not limited here.
[0373] In some scenarios, the speaker 400 needs to have an overview of the conference screen of this video conference. Then the conference interface displayed by the third-party monitor of the present application may include a sharing window, which displays the video screen of the multimedia stream 1 mentioned in the above examples 1 and 2 to display the content of the speaker's speech.
[0374] In combination with the above Example 1, or Example 2, or Example 3-1, or Example 3-1, in this Example 3-3, if Figure 5b As shown, the cloud 300 can obtain multimedia streams related to the video windows of each conference terminal of the current conference. Or it can be multimedia streams related to the video windows of each conference terminal of the current conference participant other than the conference terminal of the presenter, so that in the solution described below, the conference interface displayed on the third-party monitor can only display the digital people of other participants except the presenter. The principle is similar and will not be repeated here.
[0375] For example, in this example, the cloud 300 can obtain multimedia stream 2 from the conference terminal 401, multimedia stream 4 from the conference terminal 402, and multimedia stream 5 from the conference terminal 403.
[0376] The cloud 300 can analyze the multimedia streams (multimedia stream 2, multimedia stream 4, and multimedia stream 5) of each video window obtained from each conference terminal (the above-mentioned conference terminal 401 to conference terminal 403) to determine the focus of each participant corresponding to each conference terminal on the video screen of the shared window.
[0377] For example, the cloud 300 can perform a visual heat map analysis on the multimedia streams associated with the video windows of all participants to determine in real time the focus of each participant on the display screen of the conference terminal (eg, which position on the display screen).
[0378] There is no limitation on the method of determining the focus here, and it can also be achieved through other methods, which will not be described in detail here.
[0379] Then, the cloud 300 may synthesize the visual identification of each participant and the multimedia stream 1 related to the shared window based on the third position of each participant's focus on the shared window to obtain a third target multimedia stream.
[0380] Among them, the synthetic position of the visual identification of each participant in the corresponding frame image in the multimedia stream 1 can be determined by the cloud 300 based on the third position.
[0381] For example, the visual identifier may be a digital human (Digital Human / Meta Human) or other identifiers, which are not limited here.
[0382] Then, the cloud 300 may send the third target multimedia stream to the third-party monitor, and the third-party monitor may display the conference interface based on the third target multimedia stream.
[0383] In this way, the conference interface displayed by the third-party monitor may include a shared window, which displays the video screen of multimedia stream 1. In addition, the third position corresponding to the focus of each participant on the shared window may also display the digital person of the corresponding participant. Then the speaker can browse the conference interface of the third-party monitor to get an overview (equivalent to a bird's-eye view) of the focus of each participant, so as to determine the participation of each participant in the current speech content of the speaker.
[0384] For example, the digital human can indicate the focus of the corresponding participant through different states or display positions.
[0385] Fig.12It is a schematic diagram of a conference interface 501 (also referred to as a display interface of a sharing window 501 ) displayed on a third-party monitor for exemplary purposes.
[0386] The conference interface 501 displays the document content of the document shared by the presenter, such as text E1, text E2, etc. For example, text E1 is specifically "Office enters the 3.0 era". Other document contents displayed on the conference interface 501 are not described in detail here.
[0387] Fig.12 The display content of the digital human in the conference interface 501 is shown in different scenes.
[0388] Scenario 1: If the participant's visual focus falls on a certain position in the sharing window, the participant's digital figure is added to the position and is set to be displayed with its back facing the screen. Fig.12 In the "Looking at Materials" scene, Digital Human 2 is displayed on the screen with his back facing away.
[0389] When the cloud 300 determines that the visual focus of multiple participants is located in the same area, the digital humans of the multiple participants can overlap and be displayed distributed around the area to form an effect of gathering to watch the area.
[0390] For example, Fig.12 In the "viewing materials" scenario, the visual focus of the three participants is the area where the text E1 is located. Then, in the conference interface 501, digital human 1, digital human 2, and digital human 3 all face the text E1 and are displayed in a distributed manner near the text E1 to form an effect of gathering to view the text E1.
[0391] Scenario 2: When a participant is speaking, the digital human of the participant can be displayed in the shared window facing the screen, and the display position of the digital human can be anywhere in the shared window, for example, the position can be the third position where the visual focus of the participant is located.
[0392] For example, in Fig.12 In the “Speech: Facing the Host” scenario shown, the visual focus of participant 1 is on the position of text E2, so the cloud 300 can display the digital human 1 of participant 1 next to the text E2 that he is focusing on.
[0393] Scenario 3: When a participant inputs chat content in the chat window of the video conference application, the participant's side status is displayed in the shared window, and the chat content input by the participant can be displayed in the shared window in the form of bubbles, etc. (the specific display method is not limited). The display position of the participant in the shared window can be any position, a preset position, or a third position where the visual focus is located, which is not limited here.
[0394] For example, in Fig.12 In the "public chat" scenario shown, participant 1 enters chat content in the chat window within the video conferencing application of his conference terminal, and the cloud 300 can display the digital human 1 of participant 1 in a sideways state on the conference interface 501, and the two chat contents of the digital human 1 are shown as bubbles 701 and bubble 702.
[0395] Scenario 4: Multiple participants input chat content in the video conferencing application of their respective conference terminals through private chat. The cloud 300 can display the digital humans of the participants related to each conference terminal conducting the private chat in a specific position in the conference interface, and configure animation parameters for the digital humans of the participants conducting the private chat, so that the digital humans of the participants conducting the private chat can be displayed in an animated manner in the conference interface of the third-party monitor.
[0396] For example, the animation may be a digital person's mouth facing another digital person's ear to indicate that the two are whispering.
[0397] For example, in Fig.12 In the "private chat" scenario shown, participant 1 and participant 2 conduct a private chat in this video conference through their respective conference terminals. Then, the conference interface 501 of the third-party monitor can display digital human 1 and digital human 2 in the lower right corner of the conference interface 501, and the mouth of digital human 1 is facing the ear of digital human 2, and the digital human 1 and digital human 2 are displayed in an animated manner of whispering.
[0398] so, Fig.12 The conference interface 501 of the third-party monitor shown can present different states of different participants. The speaker 400 can then browse the participation level of the participants in the current speech content by browsing the conference interface 501 of the third-party monitor. This solution can provide feedback to the speaker on the participation status of each participant through the conference interface of the third-party monitor, so that the speaker can view the interest level of the participants in the current speech content through the conference interface.
[0399] It should be understood that Fig.12 The images of the corresponding digital humans in the "reading materials" scene, "speaking: facing the host" scene, "public chat" scene, and "private chat" scene shown are the display contents on the conference interface 501 at different times. The digital human-related images of the above four scenes are not displayed on the conference interface 501 at the same time.
[0400] Of course, in other embodiments, some of the above scenarios 1 to 4 may occur at the same time, and the principles are similar, which will not be repeated here.
[0401] It is understandable that the above examples 1, 2, 3-1, 3-2, and 3-3 all take the cloud 300 as the execution subject of the above method of the present application as an example to describe the processing process of the method of the present application. In other embodiments, the execution subject of the method of each of the above embodiments of the present application may also be an electronic device (such as the above Figure 5a If the conference terminals 401, 402, and 403 are shown in the figure, the cloud 300 can only forward the data processed by different conference terminals, or simultaneously implement part of the processing work in the above method (for example, determining the overlay position of at least one video window, determining the second target position of the conference focus, determining the target animation, etc.).
[0402] In a possible implementation, the conference terminal may analyze and determine the superimposed position of the video window of the participant of its own conference terminal in the shared window. Optionally, each conference terminal may forward the superimposed position of its own video window determined by itself to other conference terminals via the cloud 300. In some embodiments, in the speaker mode, the determination of the superimposed position of the participant's video window in the shared window may also be completed by the speaker's conference terminal.
[0403] In a possible implementation, the conference terminal may also analyze and determine the position of the conference focus corresponding to the participant of its own conference terminal in the shared window. Optionally, the conference terminal may forward the position of the conference focus corresponding to the participant of its own conference terminal in the shared window to other conference terminals via the cloud 300.
[0404] In a possible implementation, the conference terminal may determine whether to mirror at least one frame of the multimedia stream related to the video window of the conference terminal in combination with the corresponding overlay position of the video window of the participant of its own conference terminal and the corresponding conference focus position, and forward the processed multimedia stream related to the video window of the conference terminal to other conference terminals through the cloud 300.
[0405] In one possible implementation, the conference terminal may determine whether to mirror at least one frame of an image in a multimedia stream (e.g., a video stream) of the video window received from the cloud 300 from other conference terminals in combination with the overlay position of video windows related to other conference terminals forwarded through the cloud 300 and the position of the conference focus.
[0406] In addition, the operation of synthesizing the multimedia stream of the shared window and the multimedia stream of the video window in combination with the superimposed position of the video window can also be performed by the conference terminal. Moreover, when synthesizing the multimedia stream, the conference terminal can synthesize not only the multimedia stream of the video window of the conference terminal (such as the first video window of the speaker), but also the multimedia stream of the video window of other conference terminals (such as the second video window of the speaker).
[0407] It should be understood that the above-mentioned embodiments are merely examples of partial embodiments when the methods in the above-mentioned Examples 1, 2, 3-1, 3-2, and 3-3 are executed by electronic devices, and may also include embodiments of the above-mentioned methods that can be executed by other electronic devices not listed. The specific execution principle of the method is the same as the principle of the process of the cloud 300 executing the corresponding operations introduced in the above-mentioned Examples 1, 2, 3-1, 3-2, and 3-3. The only difference is that the cloud can perform the corresponding operations on the corresponding electronic devices and forward the processed data to other electronic devices participating in the meeting.
[0408] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.
[0409] In one example, Fig.13 A schematic block diagram of a device 300 according to an embodiment of the present application is shown. The device 300 may include: a processor 301 and a transceiver / transceiver pin 302 , and optionally, a memory 303 .
[0410] The components of the device 300 are coupled together via a bus 304, wherein the bus 304 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 304 in the figure.
[0411] Optionally, the memory 303 may be used for instructions in the aforementioned method embodiment. The processor 301 may be used to execute instructions in the memory 303, and control the receiving pin to receive a signal, and control the sending pin to send a signal.
[0412] The apparatus 300 may be the electronic device or a chip of the electronic device in the above method embodiment.
[0413] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0414] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the conference interface display method in the above-mentioned embodiment.
[0415] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method for displaying a conference interface in the above-mentioned embodiment.
[0416] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor may execute the computer execution instructions stored in the memory so that the chip executes the conference interface display method in the above-mentioned method embodiments.
[0417] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.
[0418] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0419] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0420] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0421] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0422] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.
[0423] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0424] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0425] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device. Of course, the processor and the storage medium can also be present in a network device as discrete components.
[0426] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.
[0427] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for displaying a conference interface, It is characterized in that Applied to a first electronic device, the method includes: Display the conference interface; The conference interface includes a sharing window and n video windows displayed in the sharing window, wherein n≥1, and n is an integer; Wherein, the n video windows are located at n first positions within the shared window; The shared window is used to display a multimedia screen related to the shared first multimedia stream; The n video windows are respectively used to display multimedia pictures related to n third multimedia streams respectively related to the n second multimedia streams, wherein the n second multimedia streams include multimedia streams collected by n electronic devices respectively, and the n electronic devices include the first electronic device; For any one of the n video windows, there exists a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream related to the any one video window; Whether there is a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream is related to the positional relationship between the first position corresponding to any one of the video windows and the corresponding second position; The second position corresponding to any one of the video windows is: the position of the visual focus of the user of the electronic device corresponding to the any one of the video windows in the shared window.
2. The method according to claim 1, It is characterized in that The second frame images that are mirror images of each other between the third multimedia stream and the second multimedia stream are frame images that are mirror images of each other in a height direction relative to the corresponding video windows.
3. The method according to claim 2, It is characterized in that When the first position corresponding to any one of the video windows and the corresponding second position are located on the same side of the center line, the third multimedia stream includes a frame image obtained by performing mirror processing on the corresponding frame image in the second multimedia stream; The center line is a line related to the height direction of any one of the video windows.
4. The method according to claim 3, It is characterized in that When the first position corresponding to any one of the video windows and the corresponding second position are located on different sides of the center line, the third multimedia stream includes the corresponding original frame image in the second multimedia stream.
5. The method according to claim 3 or 4, It is characterized in that The angle between the center line and the height direction of the corresponding video window is greater than or equal to 0 degrees and less than or equal to 45 degrees.
6. The method according to any one of claims 1 to 5, It is characterized in that The corresponding first position of the same video window in the shared window is related to the multimedia picture related to the first multimedia stream currently displayed in the shared window.
7. The method according to any one of claims 1 to 6, It is characterized in that A first position of at least one video window among the n video windows in the shared window is located in a target area of a multimedia picture related to the first multimedia stream displayed in the shared window; The target area does not include a foreground element related to the first multimedia stream.
8. The method according to any one of claims 1 to 7, It is characterized in that The n first positions of the n video windows in the shared window change as the second positions corresponding to the corresponding visual focuses change in the preset area in the shared window.
9. The method according to claim 8, It is characterized in that At least one of the n first positions is located in the preset area where the corresponding second position is located.
10. The method according to claim 8, It is characterized in that At least one of the n first positions is located in an adjacent area of the preset area where a corresponding second position is located in the shared window.
11. The method according to any one of claims 1 to 10, It is characterized in that The conference interface is an interface displayed in response to a received first user operation, wherein the first user operation is used to instruct an operation of sharing a first window or a screen.
12. The method according to any one of claims 1 to 11, It is characterized in that The n video windows include a first video window, wherein the first video window is used to display a video screen of a third multimedia stream related to the first electronic device, and the method further includes: Determining, based on the first multimedia stream associated with the shared window, a first position of the first video window to be displayed within the shared window; Determine a second position of a visual focus corresponding to a user of the first electronic device associated with the first video window in the sharing window; Based on the positional relationship between the first position and the second position corresponding to the first video window, processing at least one frame of the second multimedia stream related to the first video window to obtain a third multimedia stream related to the first video window; synthesizing the first multimedia stream and a third multimedia stream related to the first video window according to the first position related to the first video window to obtain a target multimedia stream; The conference interface includes a multimedia screen of the target multimedia stream.
13. The method according to any one of claims 1 to 12, It is characterized in that The n electronic devices further include at least one second electronic device, the n video windows include a second video window associated with the second electronic device, and the second video window is used to display a video screen of a third multimedia stream associated with the second electronic device; wherein the second video window is displayed at a corresponding first position within the shared window at at least one first moment; The at least one moment includes a moment when the user voice exists in the second multimedia stream related to the second electronic device.
14. The method according to claim 13, It is characterized in that The method further comprises: Acquire the second multimedia stream related to the second electronic device; Obtaining a first position of the second video window to be displayed in the shared window; Acquire a second position of a visual focus corresponding to a user of the first electronic device associated with the second video window in the sharing window; Based on the positional relationship between the first position and the second position corresponding to the second video window, processing at least one frame of the second multimedia stream related to the second video window to obtain a third multimedia stream related to the second video window; According to the first position related to the second video window, the first multimedia stream and the third multimedia stream related to the second video window are synthesized to obtain the target multimedia stream.
15. A method for displaying a conference interface, It is characterized in that Applied to a second electronic device, the method includes: In response to the received user voice information, displaying a conference interface; The conference interface includes a sharing window and n video windows displayed in the sharing window, wherein n≥2, and n is an integer; Wherein, the n video windows are located at n first positions within the shared window; The shared window is used to display a multimedia screen related to a first multimedia stream shared with the first electronic device; The n video windows are respectively used to display multimedia pictures related to n third multimedia streams respectively related to the n second multimedia streams, wherein the n second multimedia streams include multimedia streams respectively collected by n electronic devices, and the n electronic devices include the first electronic device and the second electronic device; For any one of the n video windows, there exists a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream related to the any one video window; Whether there is a second frame image that is a mirror image of each other between the third multimedia stream and the second multimedia stream is related to the positional relationship between the first position corresponding to any one of the video windows and the corresponding second position; The second position corresponding to any one of the video windows is: the position of the visual focus of the user of the electronic device corresponding to the any one of the video windows in the shared window.
16. The method according to claim 15, It is characterized in that The n video windows include a second video window associated with the second electronic device, and the second video window is used to display a video screen of a third multimedia stream associated with the second electronic device; The second video window is displayed at a corresponding first position within the shared window at at least one first moment; The at least one moment includes a moment when the user voice exists in the second multimedia stream related to the second electronic device.
17. The method according to claim 15 or 16, It is characterized in that The n video windows include a first video window, and the first video window is used to display a video screen of a third multimedia stream related to the first electronic device. The method further includes: Acquire the second multimedia stream related to the first electronic device; Obtaining a first position of the first video window to be displayed in the shared window; Acquire a second position of a visual focus corresponding to a user of the first electronic device associated with the first video window in the sharing window; Based on the positional relationship between the first position and the second position corresponding to the first video window, processing at least one frame of the second multimedia stream related to the first video window to obtain a third multimedia stream related to the first video window; According to the first position associated with the first video window and the first position associated with the second video window, the first multimedia stream, the third multimedia stream associated with the first video window, and the third multimedia stream associated with the second video window are synthesized to obtain a target multimedia stream.
18. A method for processing conference data, applied to a server, It is characterized in that The method comprises: Acquire a first multimedia stream associated with the shared window; Obtain n second multimedia streams associated with n video windows, where n≥1 and n is an integer; Determining n first positions where the n video windows are to be displayed in the shared window; Determine n second positions where the visual focus of the users of the electronic devices corresponding to the n video windows is located in the shared window; Based on the positional relationship between the first position and the second position corresponding to any one video window of the n video windows, at least one frame of the second multimedia stream related to the any one video window is processed to obtain a third multimedia stream related to the any one video window; According to the n first positions, synthesizing n third multimedia streams related to the corresponding n video windows and the first multimedia stream to obtain a target multimedia stream; The conference interface corresponding to the target multimedia stream includes the sharing window and the n video windows displayed at the n first positions in the sharing window; The shared window is used to display a multimedia screen related to the first multimedia stream; The n video windows are respectively used to display multimedia pictures related to n third multimedia streams.
19. A conference interface display device, applied to a first electronic device, It is characterized in that The device is used to execute the conference interface display method as described in any one of claims 1 to 14.
20. A conference interface display device, applied to a second electronic device, It is characterized in that The device is used to execute the conference interface display method as described in any one of claims 15 to 17.
21. An electronic device, It is characterized in that include: a memory and a processor, the memory and the processor being coupled; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method for displaying a conference interface as described in any one of claims 1 to 14, or executes the method for displaying a conference interface as described in any one of claims 15 to 17.
22. A computer-readable storage medium, It is characterized in that It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the method for displaying a conference interface as described in any one of claims 1 to 14, or the method for displaying a conference interface as described in any one of claims 15 to 17.
23. A computer program product comprising instructions, It is characterized in that When the computer program product runs on a computer, the computer executes the method for displaying a conference interface as described in any one of claims 1 to 14, or executes the method for displaying a conference interface as described in any one of claims 15 to 17.
24. A chip, It is characterized in that It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method for displaying a conference interface as described in any one of claims 1 to 14, or executes the method for displaying a conference interface as described in any one of claims 15 to 17.
25. A conference interface display system, It is characterized in that The system includes a first electronic device, at least one second electronic device, and a server, wherein the first electronic device is used to execute the method for displaying a conference interface as described in any one of claims 1 to 14, the second electronic device is used to execute the method for displaying a conference interface as described in any one of claims 15 to 17, and the server is used to execute the method for processing conference data as described in claim 18.
Citation Information
Patent Citations
Method and system for adapting a continuous presence layout according to interaction between conferees
CN102209228A
Immersive and interactive video conference room environment
CN104469256A
Video conference implementation method and device and storage medium
CN113784080A
Method and system for providing video conference using screen mirroring
KR1020160069271A
Method and apparatus for self-selection by participant to display a mirrored or unmirrored video feed of the participant in a videoconferencing platform
US11546385B1
