Conference processing method and related device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BOE TECHNOLOGY GROUP CO LTD
- Filing Date
- 2024-10-31
- Publication Date
- 2026-06-30
Smart Images
Figure CN122319652A_ABST
Abstract
Description
A meeting processing method and related equipment Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a conference processing method and related equipment. Background Technology
[0002] With the diversification of meeting needs, the concept of hybrid online and offline meeting scenarios has emerged. In hybrid online and offline meetings, some participants attend in a physical meeting room, while others attend from other locations. Video conferencing typically requires devices such as cameras and microphones. However, in practical applications, meeting rooms often lack these devices. A common approach is to place a computer with a camera in the physical meeting room, connecting it to a large screen to project the video conference content. However, the image quality of a single camera capturing all participants in the meeting room is poor.
[0003] Summary of the Invention
[0004] This application provides a meeting processing method and related equipment to improve the quality of meeting videos in meeting scenarios and increase the flexibility of meeting participants.
[0005] A first aspect includes a conference processing method, wherein the method is applied to a conference system, the conference system comprising a first device and at least one second device; the method includes:
[0006] The first device acquires images of participants from multiple second devices;
[0007] The first device generates a first conference video frame based on the acquired conference images, and the first conference video frame includes at least part or all of the content in the acquired conference images;
[0008] The first device displays and / or sends the first conference video frame.
[0009] In practice, the first device can acquire images of the participants through the second device and fuse these images to generate a first conference video frame. The first device displays or sends the first video frame so that users can see the participants' images and enjoy a better viewing experience.
[0010] In one possible design, the meeting processing method provided in this application embodiment includes the first device acquiring meeting images from multiple second devices, including:
[0011] The first device acquires a preset image corresponding to the target second device.
[0012] In practice, the first device can acquire a preset image corresponding to the target second device and use the preset image to replace the meeting image of the target second device to improve the viewing effect of the fused image.
[0013] In one possible design, the conference processing method provided in this application embodiment includes a plurality of second devices, including a first type of second device and / or a second type of second device, wherein the first type of second device is a second device that provides video frames to the first device, and the second type of device is a second device that does not provide video frames;
[0014] The first device acquires a preset image corresponding to the target second device, including:
[0015] Determine the preset image corresponding to the second type of second device; and / or,
[0016] If any of the second devices in the first type of second devices does not have a human figure in the video frame provided by any of the second devices, then the preset image corresponding to any of the second devices is determined.
[0017] In specific implementation, the target second device may include a first type of second device and / or a second type of second device. A preset image from the second device that does not provide video frames to the first device is used as the meeting image for that second device, as well as a preset image from the second device that provides video frames to the first device where no human figure is present in the video frames, is used as the meeting image for that second device. This ensures that the first meeting video frame includes the preset image of the target second device, improving the viewing effect of the merged image.
[0018] In one possible design, the preset image in the meeting processing method provided in this application embodiment includes at least one of the following:
[0019] User avatar, user-configured image, default image.
[0020] In one possible design, the conference processing method provided in this application embodiment includes a first type of second device, wherein the first type of second device is a second device that provides video frames to the first device;
[0021] The participation image of at least one of the plurality of second devices is determined based on video frames provided by the at least one second device.
[0022] In one possible design, the meeting processing method provided in this application embodiment,
[0023] The first device acquires images of multiple second devices participating in the meeting, and also includes:
[0024] The first device performs human image detection on the video frames provided by the first type of second device;
[0025] If the first device detects a human figure, it will use that human figure as the participant image for the second device of the first type.
[0026] In practice, the first device can perform human image detection on the video frames provided by the second device. If a human image is detected, it is used as the second device's image of the meeting, thus including the human image of the second device in the first meeting video frame. Since the second device provides its own captured video frames, which have high shooting quality, including the human image captured by the second device in the first meeting video frame improves the shooting quality of the first meeting video frame.
[0027] In one possible design, in the conference processing method provided in this application embodiment, the proportion of the participating images of each second device in the first conference video frame is the same.
[0028] In practice, when the first device fuses the images from each of the second devices, the size of each second device's image within the first conference video frame can be the same, or the fused area corresponding to the second device's image can occupy the same proportion within the first conference video frame. Alternatively, when a user views the first conference video frame, the viewing effect of each second device's image is such that its proportion within the first conference video frame is the same.
[0029] In one possible design, in the meeting processing method provided in this application embodiment, the proportion of the participant image of the presenting second device in the first meeting video frame is greater than the proportion of the participant image of the audience second device in the first meeting video, wherein the audience second device is a second device other than the presenting second device among the plurality of second devices;
[0030] The second device for speaking can be a preset second device, or it can be determined based on the current speaking audio, or it can be determined based on a second device that provides video frames with mouth speaking features to the first device.
[0031] In practice, among the multiple secondary devices, there can be a main speaker device and audience devices. The main speaker device can be understood as the secondary device used by the speaker or current speaker, while the audience devices can be understood as any secondary device other than the main speaker device. In the first conference video frame, the proportion of the image from the main speaker device is greater than that from the audience devices. This design highlights the speaker's image, making it easier for users to understand the current speaker's situation.
[0032] Mouth speech characteristics can include the mouth being in an open state, such as the distance between the upper and lower lips being greater than or equal to a preset distance threshold. Another example is that the shape of the mouth matches the actual mouth shape during speech.
[0033] In one possible design, the conference processing method provided in this application embodiment includes the first device acquiring video frames from multiple second devices, comprising:
[0034] The first device receives video frames from the second device through the first network;
[0035] The method further includes:
[0036] The first device transmits the first conference video frame to at least one third device via a second network, wherein the second network is a different communication network from the first network; and / or,
[0037] The first device receives a second conference video frame from at least one third device via the second network.
[0038] In practical implementation, the relationship between the first device and the second device can be understood as a conference group, where the first device and the second device are located within the same spatial range. The first device can form a conference group with the second device within the corresponding spatial range, and the first device can access the camera function and / or microphone function of the second device within the conference group. The first device and the second device can interact through a first network.
[0039] The relationship between the first and third devices can be understood as independent conference devices. The first and third devices can interact through the second network, but the third device and the second device do not interact. Alternatively, the first device belongs to the first conference group, and the third device can be part of the second conference group. The second conference group can include only the third device, or it can include the third device and other devices.
[0040] In one possible design, in the conference processing method provided in this application embodiment, the first network is a local area network and the second network is a wide area network.
[0041] In one possible design, the conference processing method provided in this application embodiment further includes the first device generating target conference access information in response to a conference establishment operation, so that the second device establishes a communication connection with the first device based on the target conference access information.
[0042] In one possible design, the first device displays the first conference video frame, including:
[0043] The first device displays the first conference video and each of the second video conference frames on the target interface, wherein the display area of the first conference video is the same size as the display area of the second video conference frames.
[0044] In practice, the first device can receive the second conference video frame provided by the third device. When the first device displays the conference video frame, the size of the first conference video frame and the second conference video frame are the same.
[0045] In one possible design, the meeting processing method provided in this application embodiment further includes the meeting information corresponding to the second device, and / or the meeting information corresponding to the third device;
[0046] The meeting participation information includes at least one of the following:
[0047] User ID, user avatar, user status.
[0048] In one possible design, the meeting processing method provided in this application embodiment further includes,
[0049] The first device adjusts the volume of the playing audio in response to a volume adjustment command from any of the second devices.
[0050] In one possible design, the meeting processing method provided in this application embodiment, the first device generates a first meeting video frame based on the acquired meeting participant images, including:
[0051] Based on the video frames captured by the first device and the obtained meeting images, a first meeting video frame is generated.
[0052] In practice, the first device can have a shooting function to capture video. The first device can merge the video frames it captures with the images from the participating devices in the meeting to generate the first meeting video frame, thus achieving the effect of merging the images captured by the second devices with its own images into a single frame.
[0053] Secondly, embodiments of this application provide a meeting processing method, comprising:
[0054] The second device establishes a communication connection with the first device;
[0055] The second device captures video frames and sends them to the first device.
[0056] In one possible design, the meeting processing method provided in this application embodiment further includes:
[0057] The second device, based on the fact that the face in the captured video frame meets preset conditions, performs audio acquisition and sends the acquired target audio to the first device. The preset conditions include that the mouth in the face is open; or...
[0058] The second device performs decibel detection on the collected audio based on the fact that the face in the collected video frame does not meet the preset condition; and sends a volume adjustment command to the first device based on the detection result.
[0059] Thirdly, embodiments of this application also provide a conference system, including a first device and at least one second device;
[0060] The first device can be used to perform steps such as those provided in the first aspect and its possible designs;
[0061] The second device can be used to perform steps of the methods provided in the second aspect and its possible designs.
[0062] Fourthly, embodiments of this application also provide an electronic device, which includes: a processing module and a display module;
[0063] The display module is used to display content;
[0064] The processing module is used to perform the steps performed by the first device in the method provided by the first aspect and any of its designs.
[0065] Fifthly, embodiments of this application also provide an electronic device, including a processing module and an image acquisition module;
[0066] The image acquisition module is used for image acquisition;
[0067] The processing module is used to perform the steps performed by the second device in the method provided by the first aspect and any of its designs.
[0068] In a sixth aspect, embodiments of this application also provide a computer storage medium, wherein when instructions in the computer storage medium are executed by a processor, the processor performs steps performed by a first device in a method provided by the first aspect and any design thereof, or performs steps performed by a second device in a method provided by the first aspect and any design thereof. Attached Figure Description
[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0070] Figure 1 is a schematic diagram of a conference system provided in an embodiment of this application;
[0071] Figure 2A is a schematic flowchart of the meeting processing method provided in an embodiment of this application;
[0072] Figure 2B is a schematic flowchart of the meeting processing method provided in the embodiment of this application;
[0073] Figure 3 is a schematic diagram of a conference video frame;
[0074] Figure 4 is a schematic diagram of a conference video frame;
[0075] Figure 5 is a schematic diagram of a conference video frame;
[0076] Figure 6 is a schematic diagram of a user interface;
[0077] Figure 7 is a schematic diagram of a user interface;
[0078] Figure 8 is a schematic flowchart of the meeting processing method provided in an embodiment of this application;
[0079] Figure 9 is a schematic diagram of an electronic device provided in an embodiment of this application;
[0080] Figure 10 is a schematic diagram of an electronic device provided in an embodiment of this application;
[0081] Figure 11 is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0082] The embodiments of this application involve at least one, including one or more; wherein, multiple means two or more. Furthermore, it should be understood that in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. In the description of this application, "A and / or B" includes three options: including A; including B; and including both A and B.
[0083] Figure 1 is a schematic diagram of a conference system. The conference system provided in this application embodiment may include a first device and a second device. The first device can serve as a display device, used to display conference video footage. The second device can be a device with camera functionality. Optionally, the number of second devices in the conference system can be one or more. This application does not impose excessive limitations on this.
[0084] In some possible application scenarios, the first device can be a large-screen device such as a monitor, all-in-one computer, tablet, personal computer, television, or game console. The second device can be a portable electronic device, such as a portable camera, mobile phone, tablet, portable computer, wearable device with wireless communication capabilities (such as a smartwatch, smart glasses, smart bracelet, or smart helmet), or in-vehicle equipment. Exemplary embodiments of portable electronic devices include, but are not limited to, devices equipped with… Or other portable electronic devices with different operating systems. Portable camera devices may include, but are not limited to, handheld camera devices, fixed camera devices, and small or micro aircraft (or drones) with camera capabilities. This application does not impose further limitations on this.
[0085] The second device has a camera function, which may include, but is not limited to, image capture capabilities, capturing video frames. Optionally, the second device may also have an audio capture function.
[0086] In some examples, the conferencing system may also include a server (or cloud). The first device can communicate and connect to the server. In some application scenarios, the conferencing system may also include one or more third devices. The third device can communicate and connect to the server, and the server can provide the third device with conferencing video or audio. The third device can be understood as an electronic device participating in the meeting remotely. Optionally, the third device can conduct long-chain data communication with the first device through the server.
[0087] In one possible design, the third device may include, but is not limited to, a fourth device, and / or a fifth device. In some examples, the fourth device may be a monitor, all-in-one computer, tablet, personal computer, television, game console, etc. The fourth device can act as a display device, used to display the conference video feed. Optionally, the fourth device can be the same device as the first device. As shown in Figure 1, the fourth device can be understood as a remote conferencing device. The fourth device can provide conference video to the server. The fourth device can obtain conference video or audio provided by other devices through the server.
[0088] In some examples, the fifth device can be a portable electronic device, such as a portable camera, mobile phone, tablet computer, laptop computer, wearable device with wireless communication capabilities (such as smartwatch, smart glasses, smart bracelet, or smart helmet), or in-vehicle device. Exemplary embodiments of portable electronic devices include, but are not limited to, those equipped with... Alternatively, it can be a portable electronic device operating a different operating system. Portable camera devices may include, but are not limited to, handheld camera devices, fixed camera devices, and small or micro-sized aircraft (or drones) with camera capabilities. Optionally, the fifth device can be the same as the second device. As shown in Figure 1, the fifth device can be understood as a remote personal device. The fifth device can provide video frames and / or audio to the server. The fifth device can obtain conference video or audio provided by other devices through the server.
[0089] Figure 2A illustrates an exemplary conference processing method, which can be implemented by a first device in a conference system. The conference system may include the first device and one or more second devices. The method may include the following steps:
[0090] S201A, the first device acquires the participation images of multiple second devices.
[0091] S202A, the first device generates a first conference video frame based on the acquired conference images, the first conference video frame including at least part or all of the content in the acquired conference images.
[0092] S203A, the first device displays and / or sends the first conference video frame.
[0093] In practice, a participant image from a second device can be understood as an image used by the user of the second device to participate in the meeting. The participant image from the second device can be a preset image corresponding to the second device or determined based on video frames from the second device.
[0094] The first device can fuse the images from each of the second devices to generate a first conference video frame. The first conference video frame may include some or all of the content from each of the images.
[0095] The first device can display the first conference video frame so that users can view the meeting proceedings through the first device. Alternatively, the first device can send the first conference video frame to other devices so that those devices can display it. Therefore, using the participant image provided by the second device in the conference video frame can improve the viewing experience.
[0096] In one possible design, the first device and the second device can form a conference group. The first video frame generated by the first device can reflect the conference situation within this group. For example, the first device and the second device may be located in the same spatial area.
[0097] In the conference processing method provided in this application embodiment, the video frame provided by the second device can be implemented as any frame image in the video stream provided by the second device. The first device generating the first conference video frame can be implemented as any frame image in the video stream generated by the first device. The first device can receive the video stream provided by the second device, and the first device can execute the conference processing method provided in this application embodiment based on each video stream.
[0098] Optionally, the first device may initiate the process of establishing a conference group. Figure 2B illustrates an exemplary conference processing method.
[0099] S201B, the first device responds to the conference establishment operation by generating target conference access information.
[0100] S202B, the second device establishes a communication connection with the first device based on the target conference access information.
[0101] In some examples, the first device and the second device can communicate wirelessly. Optionally, the first device and the second device can achieve wireless communication based on any existing wireless communication technology. In some possible ways, the first device and the second device can connect wirelessly based on Wi-Fi technology. Both the first device and the second device are connected within the same local area network. In other possible ways, one of the first device or the second device has a wireless hotspot function, and the other device can connect to the hotspot to achieve a wireless communication connection between the first device and the second device. In some possible ways, the first device and the second device can communicate based on a video transmission protocol. In some possible implementations, the first device and the second device can communicate based on the Bluetooth communication protocol.
[0102] In some implementations, the target conference access information can be used to establish a communication connection with the first device. Optionally, the target conference access information can be used for authentication to participate in a specified conference.
[0103] In some examples, the target conference access information can be implemented as information that can be written into an NFC device. In some examples, the target conference access information can be, but is not limited to, at least one of numbers, characters, text, and graphics. For example, the target conference access information can be implemented as a graphic code or a QR code. In some examples, the target conference access information can include a network link. The second device can establish a communication link with the first device via a network connection.
[0104] In one possible design, the first device can display the target conference access information. The user can input the target conference access information into the user interface of the second device to establish a communication connection with the first device. In another possible design, the first device can display the target conference access information. The user can use the second device to capture a picture of the target conference access information. The second device can recognize the image including the target conference access information and establish a communication connection with the first device.
[0105] In one possible design, the second device can acquire the target conference access information via a network. For example, the first device generates the target conference access information and can transmit it to other electronic devices via the network. The second device can then receive the target conference access information via the network.
[0106] In one possible implementation, the first device and the second device interact through a first network, and the first device can receive video frames provided by the second device through the first network.
[0107] The first device interacts with at least one third device via a second network. In some examples, the first device can send first conference video frames to each of the third devices via the second network. In other examples, the first device can receive second conference video frames provided by the third devices via the second network.
[0108] In one possible application scenario, the first network is a local area network (LAN) and the second network is a wide area network (WAN).
[0109] Based on the meeting processing method provided in any of the above embodiments, the first device, upon acquiring the meeting images of multiple second devices, can perform the following operations:
[0110] The first device acquires a preset image corresponding to the target second device.
[0111] The preset image includes at least one of the following: user avatar, user-configured image, and default image.
[0112] In some examples, the plurality of second devices may include a first type of second device and / or a second type of second device, wherein the first type of second device is a second device that provides video frames to the first device, and the second type of device is a second device that does not provide video frames.
[0113] The target second device may include a second type of second device, and a first type of second device that provides video frames without human figures.
[0114] Among multiple second devices, some may send the captured video frames to the first device; that is, some of the multiple second devices may be of the first type. After establishing a communication connection with the first device, the second device can activate its camera function to capture images or videos. In some application scenarios, the second device can send the captured video frames to the first device in real time. Alternatively, the second device can send the captured video frames to the first device at fixed intervals. Those skilled in the art can configure the device according to the actual application scenario. In one possible implementation, the first device can instruct the second device to capture video stream data. The video stream data includes video frames. The second device can encode and compress the video frames according to a preset resolution using an encoder before sending them to the first device.
[0115] Among multiple second devices, there may be second devices that do not provide video frames to the first device; that is, there may be a second type of second device. After the second device and the first device form a conference group, the second device may not provide video frames to the first device.
[0116] In the operation of the first device acquiring the preset image corresponding to the target second device, the preset image corresponding to the second type of second device can be determined; and / or, if any second device in the first type of second device provides a video frame without a human image portion, then the preset image corresponding to the second device is determined.
[0117] In some examples, if there is a second type of second device among multiple second devices, the first device can use a preset image of the second type of second device as the participation image of that second device.
[0118] In some examples, if there is a first type of second device among multiple second devices, the first device can detect the video frames provided by each first type of second device. If a video frame provided by a second device does not have a human image, then the preset image of that second device is used as the participation image of that second device.
[0119] Based on the meeting processing method provided in any of the above embodiments, the first device acquires meeting images from multiple second devices, and further includes:
[0120] The first device performs human image detection on the video frames provided by the first type of second device;
[0121] If the first device detects a human figure, it will use that human figure as the participant image for the second device of the first type.
[0122] The first device can detect video frames provided by each type of second device. If a human figure is detected in a video frame of a second device, the human figure is used as the image of that second device. The first device can perform human figure segmentation processing on the video frames provided by the second device using a preset model. The first device can adjust the size of the human figure to achieve a better display effect.
[0123] Based on the conference processing method provided in any of the above embodiments, the first device combines the participant images and virtual backgrounds of each second device using OpenGL to draw them, and then renders them on the display layer canvas to display the first conference video frame. In some examples, the first device can encode the first conference video frame into H.264 using MediaCodec and then transmit it to a server or the cloud to send the first conference video frame.
[0124] In some examples, based on the conference processing method provided in any of the above embodiments, the first device may be configured with a first mode. In the first mode, in the first conference video frame generated by the first device based on the acquired images of the second devices participating in the conference, the proportion of the images of each second device participating in the conference video frame is the same.
[0125] In one possible implementation, as shown in Figure 3, the first conference video frame generated by the first device includes proportionally arranged sub-regions, which can achieve a split-screen effect for the first conference video. Optionally, the number of sub-regions can be the same as the number of second devices. Alternatively, the relationship between the number of sub-regions 'a' and the number of second devices 'n' can be a = n / 2 + n%2, where n%2 represents the remainder when n is divided by 2.
[0126] In some examples, based on the conference processing method provided in any of the above embodiments, the first device may be configured with a second mode. In the second mode, in the first conference video frame generated by the first device based on the acquired images of the second devices participating in the conference, the proportion of the image of the presenting second device in the first conference video is greater than the proportion of the image of the audience second devices in the first conference video, wherein the audience second devices are second devices other than the presenting second device among the plurality of second devices.
[0127] The second device for the main speaker can be understood as the second device used by the main speaker, or the second device used by the current speaker, or the second device that can be a second device that is pre-set by the first device.
[0128] In one possible design, the main speaker second device is a preset second device.
[0129] In one possible design, the second speaking device is determined based on the current speaking audio. The first device can receive audio provided by the second device, and the first device can determine the current speaking audio based on the audio provided by the second device, thereby determining the second speaking device.
[0130] In one possible design, the second speaking device is determined based on a second device providing the first device with video frames exhibiting mouth-speech characteristics. The first device can detect whether the video frames provided by the second device contain mouth-speech characteristics. These characteristics could include the mouth being open, for example, the distance between the upper and lower lip being greater than or equal to a preset distance threshold. Another example is that the shape of the mouth matches the actual mouth shape during speech.
[0131] In one possible design, each second device is equipped with the ability to detect whether there are mouth speech features in the video frame. If mouth speech features are detected, the collected audio is sent to the first device, and the first device can determine the main speaker second device based on the received audio.
[0132] In one possible design, the second presenter device can activate its microphone to capture audio. For example, the second presenter device can use MediaRecorder to capture or record audio and obtain audio metadata. Optionally, the second presenter device can also perform echo cancellation, noise reduction, and other processing on the captured audio. The second presenter device can then send the audio to the first device. The first device can then play the audio or send it to a server.
[0133] Among the multiple second devices, any second device other than the main presenting second device can serve as an audience second device. In the first conference video frame, the proportion of the participating image of the main presenting second device in the first conference video is greater than the proportion of the participating image of the audience second device in the first conference video, wherein the audience second device refers to any second device other than the main presenting second device among the multiple second devices.
[0134] In one possible implementation, as shown in FIG4, the first conference video frame generated by the first device may include a first display area and a second display area. The first display area includes the participating image of the presenting second device. The second display area may be located below the first display area, and the second display area includes the participating images of other second devices. For example, the second display area may include equally spaced sub-regions. Each sub-region includes the participating image of one other second device.
[0135] Optionally, the first display area may occupy three-quarters of the first conference video frame, and the second display area may occupy one-quarter of the first conference video frame.
[0136] In some examples, the first device may be configured with a third mode. In the third mode, the operation of the first device displaying the first conference video frame based on the acquired participant images may perform the following process:
[0137] The participation images of each second device are displayed in multiple first areas, wherein one second device's participation image is displayed in one first area.
[0138] As shown in Figure 5, the first device can display the participation images of each second device in multiple first areas of the user interface. Optionally, the first device can receive at least one second conference video frame provided by a third device from the server. The first device can display the second conference video frame provided by the third device in a second area of the user interface. The number of second areas can be the same as the number of third devices. The shape and size of each first area in the user interface are the same as the shape and size of the second areas.
[0139] In the conference processing method provided by any of the above embodiments, in step S202A, the first device can generate a first conference video frame based on the acquired participant images and target mode. The first device can be configured with multiple modes. The multiple modes may include, but are not limited to, the aforementioned first mode, second mode, and third mode. The target mode can be a default mode or a user-configured mode. The first device can display the first conference video frame in the manner corresponding to the target mode. The first device can adjust the method of generating the first conference video frame as the mode is switched.
[0140] In the conference processing method provided based on any of the above embodiments, the conference system may further include a third device. The first device may send the first conference video frame to the third device via a second network. The first device may receive at least one second conference video frame provided by a third device via the second network.
[0141] In some examples, referring to Figure 6, the first device displays the first conference video and each of the second video conference frames on the target interface, wherein the display area of the first conference video is the same size as the display area of the second video conference frames.
[0142] Optionally, as shown in Figure 7, the target interface may also include the meeting information corresponding to the second device, and / or the meeting information corresponding to the third device;
[0143] The meeting participation information includes at least one of the following:
[0144] User ID, user avatar, user status.
[0145] In the conference processing method provided in any of the above embodiments, after the first device and the second device establish a communication connection, the second device can provide the first device with the acquired video frames.
[0146] In one possible design, the first device can obtain the camera identifier (cameraAid) of the second device. Based on the camera identifier, the first device can obtain the data corresponding to that camera, i.e., the video frames captured by the second device. The first device can convert the camera data into YUV data for rendering and display. The first device can also convert the YUV data into RGB data to generate the first conference video frame.
[0147] In another possible design, the first device can be configured with first software. Each second device can be configured with second software. The first software in the first device can be configured to use the camera data of the second devices. The first device can select the desired second device through the first software. The second device selected by the first device can establish a communication connection with the first device.
[0148] The second software in the second device can obtain camera data through the system interface of the second device. For example, the second software in the second device controls the camera on / off state by accessing the camera API of the underlying framework. The second software in the second device can use the MediaCodec encoder to perform H.264 encoding at a resolution of 1920x1080 for video compression, so that the first device can provide the camera data of the second device.
[0149] The first device can decode H.264 using the MediaCodec decoder and provide the second device's camera data to the first device's system using the binder mechanism. The first device can then determine the second device's participation image based on the second device's camera data. The first software can call OpenGL functions to generate the first conference video frame based on the second device's participation image.
[0150] Based on the conference processing method provided in any of the above embodiments, after receiving audio from any second device, the first device can indicate the video frame corresponding to the second device in the first conference video frame to prompt the current speaker or speaker. The indication method may include, but is not limited to, a static method and a dynamic method. Dynamic methods may include, but are not limited to, magnification, focusing, dithering, and light waves.
[0151] Figure 8 illustrates an exemplary conference processing method that can be executed by a second device, the method including the following steps:
[0152] S801, the second device establishes a communication connection with the first device.
[0153] S802, the second device captures video frames and sends them to the first device.
[0154] In one possible design, the conference processing method may further include: the second device performing audio acquisition based on the mouth speech features in the acquired video frame, and sending the acquired target audio to the first device; or, the second device performing decibel detection on the acquired audio based on the fact that the face in the acquired video frame does not meet the preset conditions; and sending a volume adjustment command to the first device based on the detection result.
[0155] In some examples, the second device's camera function is enabled to capture video frames and perform face detection.
[0156] In one possible scenario, the second device captures audio based on mouth speech features in the acquired video frames and sends the captured target audio to the first device. The mouth speech features can indicate that the mouth in the face is open. Optionally, the second device can determine the mouth's aspect ratio in the video frame. If the aspect ratio is greater than or equal to a preset threshold, it indicates the mouth is open. If the aspect ratio is less than the preset threshold, it indicates the mouth is closed.
[0157] The second device detects mouth-like speech features in the video frame and initiates the recording function to capture audio. Alternatively, the second device simultaneously captures video frames and audio; if the second device detects mouth-like speech features in the video frame, it can send the captured audio to the first device.
[0158] The second device can use MediaRecorder to record audio and obtain audio metadata. The second device can also perform audio processing such as echo cancellation and noise reduction on the captured audio.
[0159] In one possible scenario, either of the second devices performs decibel detection on the acquired audio based on the absence of mouth speech features in the acquired video frames; and based on the detection result, sends a volume adjustment command to the first device; the first device adjusts the volume of the target audio playback based on the volume adjustment command.
[0160] In some examples, after the second device's camera function is activated, it captures video frames and performs face detection. If no facial features are detected (e.g., mouth gestures), the second device initiates audio recording to capture audio. The second device can perform decibel detection on the captured audio. If the decibel value detected by the second device is less than a first decibel threshold, it sends a volume increase command to the first device. The first device then increases the volume of the target audio based on the volume increase command. If the decibel value detected by the second device is greater than a second decibel threshold, it sends a volume decrease command to the first device, where the second decibel threshold is greater than the first decibel threshold. The first device then decreases the volume of the target audio based on the volume increase command.
[0161] Based on the same concept, Figure 9 is a schematic diagram of the structure of an electronic device 1800 provided in an embodiment of this application. The electronic device 1800 can be either the first device or the second device mentioned above. As shown in Figure 9, the electronic device 1800 may include: one or more processors 1801; one or more memories 1802; a communication interface 1803; and one or more computer programs 1804. These devices can be connected via one or more communication buses 1805. The one or more computer programs 1804 are stored in the memory 1802 and configured to be executed by the one or more processors 1801. The one or more computer programs 1804 include instructions. The communication interface 1803 is used to enable communication between the electronic device and other devices; for example, the communication interface can be a transceiver.
[0162] Figure 10 illustrates a schematic diagram of an electronic device. The electronic device can be implemented as the aforementioned first device. The electronic device may include a processing module and a display module. The display module is used to display content, such as displaying conference video frames. The processing module is used to perform the steps executed by the first device in any of the aforementioned conference processing methods provided in the embodiments.
[0163] Figure 11 illustrates a schematic diagram of an electronic device. The electronic device can be implemented as the aforementioned second device. The electronic device may include the image acquisition module for image acquisition, such as a camera. Optionally, the electronic device may also include an audio acquisition module for audio acquisition. The processing module is used to perform the steps executed by the second device in the conference processing method provided in any of the foregoing embodiments.
[0164] This application also provides a conference system, which may include the first device from any of the conference processing methods provided in the foregoing embodiments, and at least one second device. Optionally, the conference system may further include at least one third device.
[0165] In the embodiments provided above, the methods provided by the present application are described from the perspective of an electronic device (e.g., a first device or a second device) as the executing entity. To implement the functions of the methods provided in the embodiments of the present application, the electronic device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.
[0166] Additionally, embodiments of this application provide a computer-readable storage medium for storing a computer program that, when executed on a computer, causes the computer to perform steps in any of the above-described screen projection methods, such as executing the screen projection method executed by a first device, and / or, such as executing the screen projection method executed by a second device.
[0167] In the above embodiments, the terms "when..." or "after..." can be interpreted, depending on the context, as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, the phrases "when..." or "if (the stated condition or event) is detected" can be interpreted, depending on the context, as meaning "if...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)". Furthermore, in the above embodiments, relational terms such as "first" and "second" are used to distinguish one entity from another, without limiting any actual relationship or order between these entities.
[0168] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0169] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0170] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)). Where there is no conflict, the solutions of the above embodiments can be combined.
[0171] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0174] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0175] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.
Claims
1. A meeting processing method, wherein, Applied to a first device, the method includes: The first device acquires images of participants from multiple second devices; The first device generates a first conference video frame based on the acquired conference images, and the first conference video frame includes at least part or all of the content in the acquired conference images; The first device displays and / or sends the first conference video frame.
2. The method as described in claim 1, wherein, The first device acquires images of multiple second devices participating in the meeting, including: The first device acquires a preset image corresponding to the target second device.
3. The method as described in claim 2, wherein, The plurality of second devices include a first type of second device and / or a second type of second device, wherein the first type of second device is a second device that provides video frames to the first device, and the second type of device is a second device that does not provide video frames; The first device acquires a preset image corresponding to the target second device, including: Determine the preset image corresponding to the second type of second device; and / or, If any of the second devices in the first type of second devices does not have a human figure in the video frame provided by any of the second devices, then the preset image corresponding to any of the second devices is determined.
4. The method of claim 2, wherein, The preset image includes at least one of the following: User avatar, user-configured image, default image.
5. The method as described in claim 1 or 2, wherein, The plurality of second devices includes a first type of second device, wherein the first type of second device is a second device that provides video frames to the first device; The participation image of at least one of the plurality of second devices is determined based on video frames provided by the at least one second device.
6. The method of claim 5, wherein, The first device acquires images of multiple second devices participating in the meeting, and also includes: The first device performs human image detection on the video frames provided by the first type of second device; If the first device detects a human figure, it will classify the human figure as part of the first category. Images of the two participating devices.
7. The method as described in any one of claims 1-4, wherein, In the first conference video frame, the proportion of the participating images of each second device in the first conference video is the same.
8. The method as described in any one of claims 1-4, wherein, In the first conference video frame, the proportion of the presentation image of the second device in the first conference video is greater than the proportion of the audience image of the second device in the first conference video, wherein the audience second device is the second device other than the presentation second device among the plurality of second devices; The second device for speaking can be a preset second device, or it can be determined based on the current speaking audio, or it can be determined based on a second device that provides video frames with mouth speaking features to the first device.
9. The method of claim 1, wherein, The first device acquires images of multiple second devices participating in the meeting, including: The first device receives video frames from the second device through the first network; The method further includes: The first device transmits the first conference video frame to at least one third device via a second network, wherein the second network is a different communication network from the first network; and / or, The first device receives a second conference video frame from at least one third device via the second network.
10. The method of claim 9, wherein, The first network is a local area network (LAN), and the second network is a wide area network (WAN).
11. The method of claim 10, wherein, The first device displays the first conference video frame, including: The first device displays the first conference video and each of the second video conference frames on the target interface, wherein the display area of the first conference video is the same size as the display area of the second video conference frames.
12. The method of claim 10, wherein, The target interface also includes the meeting information corresponding to the second device, and / or the meeting information corresponding to the third device; The meeting participation information includes at least one of the following: User ID, user avatar, user status.
13. The method of claim 1 or 10, wherein, The method further includes: In response to a conference establishment operation, the first device generates target conference access information so that the second device can establish a communication connection with the first device based on the target conference access information.
14. The method of claim 1 or 10, wherein, The method further includes: The first device adjusts the volume of the playing audio in response to a volume adjustment command from any of the second devices.
15. The method of claim 1, wherein, The first device generates a first meeting video frame based on the acquired meeting images, including: Based on the video frames captured by the first device and the obtained meeting images, a first meeting video frame is generated.
16. A meeting processing method, wherein, Applied to a second device, the method includes: The second device establishes a communication connection with the first device; The second device captures video frames and sends them to the first device.
17. The method of claim 16, wherein, The method further includes: The second device captures audio based on mouth-related speech features in the acquired video frames and sends the captured target audio to the first device; or... The second device performs decibel detection on the collected audio based on the fact that the face in the collected video frame does not meet the preset condition; and sends a volume adjustment command to the first device based on the detection result.
18. A conference system, wherein, Includes a first device and at least one second device; The first device is used to perform the steps performed by the first device in any of the methods of claims 1-15; The second device is used to perform the steps performed by the second device as claimed in claim 16 or 17.
19. An electronic device, wherein, include: Processing module and display module; The display module is used to display content; The processing module is used to perform the steps executed by the first device in any of the methods described in claims 1-15.
20. An electronic device, wherein, Includes a processing module and an image acquisition module; The image acquisition module is used for image acquisition; The processing module is used to perform the steps performed by the second device in any of the methods described in claims 1-15.
21. A computer storage medium, wherein, When the instructions in the computer storage medium are executed by the processor, the processor performs the steps performed by the first device in any one of the methods described in claims 1-15, or the processor performs the steps performed by the second device in the method described in claim 16 or 17.