Remote conference method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a remote conference method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: receiving a position selection image which is sent by conference access equipment and aims at a conference scene, and enabling the position selection image to be displayed on a virtual display screen of augmented reality equipment; in response to a selection instruction of a target user, determining a target position corresponding to the selection instruction in the position selection image; the target position is sent to the conference access device, the conference access device generates playing content according to the target position and attitude information, and the attitude information is acquired through an attitude acquisition module of the augmented reality device; and receiving the playing content sent by the conference access device, and playing the playing content on the virtual display screen. According to the remote conference method provided by the embodiment of the invention, the audiovisual feeling of the conference site can be brought to the target user who uses the augmented reality device to perform remote conference participation, and the conference participation experience of the remote user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to a remote conference method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of extended reality devices, users can conduct virtual reality meetings through VR devices. By constructing virtual spaces and character images, simulated real meeting scenes can be displayed on the display screen of the VR device. They can also conduct augmented reality meetings through AR devices. By superimposing the video conferencing interface on the display screen of the AR device, the meeting screen can be displayed in the user's real front field of vision.

[0003] In the prior art, when users use existing extended reality devices for remote conferences, most of them present the live conference recording at the conference site to the participants of the remote conference from the perspective of the recorder.

[0004] Therefore, even if existing remote conference participants wear extended reality devices, they do not have the experience of being in the meeting site, which cannot meet the user's immersive usage needs, resulting in a poor user experience. Summary of the invention

[0005] The embodiments of the present application provide a remote conference method, apparatus, electronic device, and computer-readable storage medium, which can provide target users who use extended reality devices to remotely participate in a conference with an audio-visual experience of a conference site, thereby improving the remote user's conference experience.

[0006] In a first aspect, an embodiment of the present application provides a remote conference method, which is applied to an extended reality device, and the method includes:

[0007] Receiving a location selection image for a conference scene sent by a conference access device, wherein the location selection image is displayed on a virtual display screen of the extended reality device;

[0008] In response to a selection instruction of a target user, determining a target position corresponding to the selection instruction in the position selection image;

[0009] Sending the target position to the conference access device, the conference access device generating playback content according to the target position and posture information, the posture information being acquired by a posture acquisition module of the extended reality device;

[0010] The playback content sent by the conference access device is received, and the playback content is played on the virtual display screen.

[0011] Optionally, in some embodiments of the present application, in response to a selection instruction of the target user, determining a target position corresponding to the selection instruction in the conference scene includes:

[0012] Determine the coordinate information of the selection instruction on the position selection image;

[0013] The target position in the conference scene is determined according to the coordinate information.

[0014] Optionally, in some embodiments of the present application, the extended reality device includes: a first shooting module, the first shooting module is used to obtain an image in front of the extended reality device, and before responding to the selection instruction of the target user, the method further includes:

[0015] Acquire a gesture image of the target user through the first shooting;

[0016] Determine selection information corresponding to the gesture image in the position selection image;

[0017] The selection instruction is generated according to the selection information.

[0018] Optionally, in some embodiments of the present application, the extended reality device includes: a posture acquisition module, and before receiving the playback content sent by the conference access device, the method further includes:

[0019] Acquiring the posture information of the target user through the posture acquisition module;

[0020] The posture information is sent to the conference access device.

[0021] In a second aspect, an embodiment of the present application provides a remote conference method, which is applied to a conference access device, wherein the conference access device includes a plurality of second shooting modules, and the method includes:

[0022] Determine a position selection image corresponding to a conference scene by using the multiple second shooting modules, wherein the multiple second shooting modules are arranged at different positions of the conference scene;

[0023] Sending the location selection image to an extended reality device;

[0024] receiving a target position sent by the extended reality device, wherein the target position is generated in the position selection image according to a selection instruction of a target user;

[0025] The playback content is generated according to the target position, and the playback content is sent to the extended reality device, and the playback content is played on the extended reality device.

[0026] Optionally, in some embodiments of the present application, determining the position selection image corresponding to the conference scene by using the multiple second shooting modules includes:

[0027] Acquire a plurality of scene images of the conference scene through the plurality of second shooting modules;

[0028] The position selection image is determined based on the plurality of scene images.

[0029] Optionally, in some embodiments of the present application, determining the position selection image according to the multiple scene images includes:

[0030] Determine a first location area and a second location area in the conference scene according to the multiple scene images, wherein the first location area is a location area where conference users already exist and cannot be selected, and the second location area is a location area where no conference users exist;

[0031] The position selection image is determined according to the first position area and the second position area.

[0032] Optionally, in some embodiments of the present application, the conference access device includes a plurality of audio acquisition modules, and the generating of playback content according to the target location includes:

[0033] Acquire a plurality of scene images of the conference scene through the plurality of second shooting modules;

[0034] Acquire multiple audio data at different angles in the conference scene through the multiple audio acquisition modules, wherein the multiple audio acquisition modules are arranged at different positions of the conference scene;

[0035] Determine a target picture according to the multiple scene images and the target position, and determine audio content corresponding to the target picture according to the multiple audio data and the target position;

[0036] The playback content is generated according to the target picture and the audio content.

[0037] Optionally, in some embodiments of the present application, generating the playback content according to the multiple scene images, the multiple audio data and the target position information includes:

[0038] Receiving the posture information of the target user sent by the extended reality device;

[0039] The target picture is determined according to the posture information and the multiple scene images, and the audio content corresponding to the target picture is determined according to the posture information and the multiple audio data.

[0040] In a third aspect, an embodiment of the present application further provides a remote conference device, which is applied to an extended reality device, and the device includes:

[0041] A first processing module, configured to receive a location selection image for a conference scene sent by a conference access device, wherein the location selection image is displayed on a virtual display screen of the extended reality device;

[0042] The first processing module is further configured to determine, in response to a selection instruction of a target user, a target position corresponding to the selection instruction in the position selection image;

[0043] A first sending module, configured to send the target position to the conference access device, wherein the conference access device generates playback content according to the target position and posture information, wherein the posture information is acquired by a posture acquisition module of the extended reality device;

[0044] The first processing module is further configured to receive the playback content sent by the conference access device and play the playback content on the virtual display screen.

[0045] Optionally, in some embodiments of the present application, the first processing module is used to:

[0046] Determine the coordinate information of the selection instruction on the position selection image;

[0047] The target position in the conference scene is determined according to the coordinate information.

[0048] Optionally, in some embodiments of the present application, the first processing module is further used to:

[0049] Acquire a gesture image of the target user through the first shooting;

[0050] Determine selection information corresponding to the gesture image in the position selection image;

[0051] The selection instruction is generated according to the selection information.

[0052] Optionally, in some embodiments of the present application, the first sending module is used to:

[0053] Acquiring the posture information of the target user through the posture acquisition module;

[0054] The posture information is sent to the conference access device.

[0055] In a fourth aspect, an embodiment of the present application further provides a remote conference device, which is applied to a conference access device, and the device includes:

[0056] A second processing module, configured to determine a position selection image corresponding to a conference scene through the plurality of second shooting modules, wherein the plurality of second shooting modules are arranged at different positions of the conference scene;

[0057] A second sending module, used for sending the position selection image to an extended reality device;

[0058] The second processing module is further used to receive a target position sent by the extended reality device, wherein the target position is generated in the position selection image according to a selection instruction of a target user;

[0059] The second processing module is further configured to generate playback content according to the target location, and send the playback content to the extended reality device, so that the playback content is played on the extended reality device.

[0060] Optionally, in some embodiments of the present application, the second processing module is used to:

[0061] Acquire a plurality of scene images of the conference scene through the plurality of second shooting modules;

[0062] The position selection image is determined based on the plurality of scene images.

[0063] Optionally, in some embodiments of the present application, the second processing module is further used to:

[0064] Determine a first location area and a second location area in the conference scene according to the multiple scene images, wherein the first location area is a location area where conference users already exist and cannot be selected, and the second location area is a location area where no conference users exist;

[0065] The position selection image is determined according to the first position area and the second position area.

[0066] Optionally, in some embodiments of the present application, the second processing module is further used to:

[0067] Acquire a plurality of scene images of the conference scene through the plurality of second shooting modules;

[0068] Acquire multiple audio data at different angles in the conference scene through the multiple audio acquisition modules, wherein the multiple audio acquisition modules are arranged at different positions of the conference scene;

[0069] Determine a target picture according to the multiple scene images and the target position, and determine audio content corresponding to the target picture according to the multiple audio data and the target position;

[0070] The playback content is generated according to the target picture and the audio content.

[0071] Optionally, in some embodiments of the present application, the second processing module is further used to:

[0072] Receiving the posture information of the target user sent by the extended reality device;

[0073] The target picture is determined according to the posture information and the multiple scene images, and the audio content corresponding to the target picture is determined according to the posture information and the multiple audio data.

[0074] In a fifth aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned remote conferencing method when executed by the processor.

[0075] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned remote conferencing method are implemented.

[0076] In a seventh aspect, the embodiments of the present application further provide a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations described in the embodiments of the present application.

[0077] In summary, the embodiment of the present application receives a position selection image for a conference scene sent by a conference access device, and the position selection image is displayed on a virtual display screen of the extended reality device; in response to a selection instruction of a target user, determines a target position corresponding to the selection instruction in the position selection image; sends the target position to the conference access device, and the conference access device generates playback content according to the target position and posture information, and the posture information is acquired by a posture acquisition module of the extended reality device; receives the playback content sent by the conference access device, and plays the playback content on the virtual display screen, returns the corresponding target position to the conference access device according to the user's selection instruction, and then receives and plays the playback content generated by the conference access device according to the target position, thereby achieving the technical effect of bringing the audio-visual experience of the conference site to the target users who use the extended reality device to remotely participate in the meeting, and improving the remote users' meeting experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 A schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs a remote conference method;

[0080] Figure 2 It is a schematic diagram of a scenario in which an extended reality device provided in an embodiment of the present application executes the remote conference method;

[0081] Figure 3 is a schematic diagram of the XR glasses structure corresponding to the remote conferencing method provided in an embodiment of the present application;

[0082] Figure 4 It is a schematic diagram of a scenario in which a conference access device according to an embodiment of the present application performs a remote conference method;

[0083] Figure 5 It is a schematic diagram of a scenario in which a conference access device provided in an embodiment of the present application executes the remote conference method;

[0084] Figure 6 It is a schematic diagram of a conference scene provided in an embodiment of the present application;

[0085] Figure 7 is a schematic diagram of target position selection results provided in an embodiment of the present application;

[0086] Figure 8 It is a schematic diagram of the overall structure of the remote conference method provided in the embodiment of the present application;

[0087] Fig. 9 It is a structural schematic diagram of a remote conference device of an extended reality device provided in an embodiment of the present application;

[0088] Fig.10 It is a structural schematic diagram of a remote conference device of a conference access device provided in an embodiment of the present application;

[0089] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0091] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the above features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0092] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0093] First, the terms involved in this application are explained:

[0094] Augmented Reality (AR) is a technology that creates an enhanced perceptual environment by combining virtual information with real-world scenes.

[0095] Extended reality devices: used to integrate virtual content with the real world to provide an enhanced visual experience. These devices usually use head-mounted displays (HMD), smart glasses, or other forms of wearable devices.

[0096] The embodiment of the present application provides a remote conference method, device, electronic device and computer readable storage medium. Specifically, the embodiment of the present application provides a remote conference device suitable for the remote conference method, the remote conference device includes a remote conference device and a main control device of the remote conference device.

[0097] In the existing technology, with the rapid development and increasing maturity of extended reality display technology, the support of augmented reality display and artificial intelligence large models has enabled extended reality devices to have rich and colorful functional applications. More and more wearable extended reality devices (such as AR, VR, XR glasses, etc.) have been launched on the market, and there are more and more immersive remote conferencing scenarios using wearable extended reality devices.

[0098] However, when users use existing extended reality devices for remote meetings, most of them adopt a method of presenting the live meeting recorded at the meeting site to the participants of the remote meeting from the perspective of the recorder.

[0099] However, this type of remote conferencing method does not provide the experience of being in the meeting room, even if the user wears an extended reality device. It cannot meet the user's immersive usage needs, resulting in a poor user experience.

[0100] Therefore, the existing remote conferencing methods using extended reality devices lack the real experience effect. The playback content is not presented to remote online users from the audio-visual perspective of the users at the meeting site, resulting in the remote online users being unable to be in the on-site meeting and unable to provide users with an immersive conference experience.

[0101] The embodiment of the present application provides a remote conference method, device, electronic device and computer-readable storage medium. The method adopts a technical solution of receiving a location selection image for a conference scene sent by a conference access device, displaying the location selection image on a virtual display screen of an extended reality device; responding to a selection instruction of a target user, determining a target location corresponding to the selection instruction in the conference scene, and acting on the location selection image; sending the target location to the conference access device; receiving playback content sent by the conference access device, and playing the playback content on a virtual display screen, wherein the playback content is generated by the conference access device according to the target location.

[0102] In summary, the remote conference method in the embodiment of the present application can return the corresponding target location to the conference access device according to the user's selection instruction, and then receive and play the playback content generated by the conference access device according to the target location, thereby achieving the technical effect of bringing the audio-visual experience of the conference site to the target users who use extended reality devices to participate in the remote meeting, thereby improving the remote users' meeting experience.

[0103] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0104] See also Figure 1 , Figure 1 The scene diagram of the remote conference method provided in the embodiment of the present application is as follows. The remote conference system may include an extended reality device 100 and a main control device 200. The extended reality device 100 and the main control device 200 may be connected to each other in any manner, including but not limited to signal communication through electronic circuits, communication through wireless signals, and the wireless signals may be computer network communications of the TCP / IP Protocol Suite (TCP / IP) and the User Datagram Protocol (UDP). The extended reality device 100 may receive control signals from a remote control or a control panel, and the extended reality device 100 may also receive command information sent by the main control device 200. The extended reality device 100 may perform corresponding operations according to the corresponding command information, such as the remote conference method in the present application.

[0105] In the embodiment of the present application, the extended reality device 100 includes but is not limited to a head-mounted display (Head-Mounted Display, HMD), smart glasses (Smart Glasses) or other forms of wearable devices.

[0106] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or less of the extended reality devices shown in Figure 1 Only one extended reality device is shown, and no specific limitation is given here.

[0107] In addition, if Figure 1 As shown, the main control device 200 may include any hardware device capable of data processing and instruction sending, such as a CPU or a single-chip microcomputer embedded in the extended reality device, and the specific details are not limited here.

[0108] It should be noted that Figure 1The scenario diagram of the remote conference system shown is only an example. The remote conference system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of remote conference systems and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0109] For details, please refer to Figure 2 , Figure 2 : is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs the remote conference method, wherein the specific execution process of the extended reality device performing the remote conference method is as follows:

[0110] S201: receiving a location selection image for a conference scene sent by a conference access device.

[0111] In an embodiment of the present application, the position selection image is displayed on a virtual display screen of an extended reality device, and the virtual display screen is generated after the physical display screen undergoes multiple propagation conversions through multiple optical components.

[0112] As an optional embodiment, the position selection image may be a plurality of position selection images with different viewing angles, so that the user can select a desired position within a selectable area in the image.

[0113] It should be noted that the selection options may include: you can choose if there are vacancies, you can add seats by yourself if there are no vacancies, and you can also add seats by yourself if there are vacancies.

[0114] In the embodiment of the present application, the meeting scene is an offline meeting scene, such as an offline meeting room.

[0115] As an optional embodiment, after the conference access device at the offline conference room obtains the on-site image, the generated location selection image is sent to the extended reality device, and the extended reality device receives the location selection image and displays it on the virtual display screen.

[0116] In the embodiments of the present application, the extended reality device may be a variety of wearable extended reality devices such as XR glasses. The XR terminal electronic device refers to a wearable device for the human body that applies XR technology (including VR, AR, and MR), namely, XR smart glasses and XR head-mounted displays. In this application, XR glasses are taken as an example.

[0117] Alternatively, if Figure 3As shown in the schematic diagram of the XR glasses structure, the XR smart glasses have the natural form of ordinary glasses. The main body of the glasses is composed of a frame and temples. In addition to the components contained in these ordinary glasses, they also have a computing processor (that is, the above-mentioned main control device 200), sensors, displays, shooting modules, optical components, speakers, communication components, etc.

[0118] The computing processor is used for information data processing, logical operations, etc. of XR smart glasses, including units such as CPU, GPU, NPU, etc. It can be a general-purpose chip system or a chip system designed specifically for XR business, such as Qualcomm Snapdragon's AR1 and AR2 chip systems.

[0119] Sensors are used by smart XR glasses to collect and observe information about the environment around the glasses, including the user himself. They include cameras for collecting video image information, gyroscopes for measuring user posture and movements, accelerometers for measuring the position of the user and device, magnetic sensors for measuring the magnetic declination of the device, eye trackers for tracking the user's line of sight, and microphones for collecting user voice information.

[0120] The display is used to display the images rendered by the computing processor, including self-luminous active devices such as micro-led and micro-oled, liquid crystal displays LCD and LCOS that require external light sources, as well as micro-electromechanical systems MEMS and digital mirror arrays DMD.

[0121] Optical components are used to form images and transmit image light to the human eye, including lenses, optical films, optical waveguides, etc.

[0122] The speaker is used to play sound.

[0123] The communication component is used for communication between XR glasses and other external devices such as platform cloud servers, conference site controllers, XR glasses used by other participants, etc. It can be a cellular communication module, WiFi module, Bluetooth module, etc.

[0124] In addition, XR smart glasses also include a power supply, which can be set inside the XR glasses or separated from the glasses body and electrically connected. XR glasses may also include supporting devices such as rings, watches, bracelets, consoles, headphones, etc., which can be electrically connected to the glasses body when needed. It can be understood that the aforementioned computing processor, sensor, speaker, and communication component can also be set outside the glasses body and electrically connected to the glasses body.

[0125] In an embodiment of the present application, by determining the user's virtual meeting location in the offline on-site conference room, that is, the target location, the offline on-site conference room scene information observed by the virtual meeting location is generated, and the conference room scene information is presented to the user.

[0126] Optionally, in the meeting preparation stage before the formal start of the meeting, the XR smart glasses communicate with the conference access device installed at the offline conference room. The conference access device is an electronic device that integrates environmental perception and communication functions, and may also integrate some computing and processing functions. The XR smart glasses send instructions to the conference access device, instructing the conference access device to capture photos of the conference room scene before the meeting starts through the camera. The conference room scene photos at least include information such as the conference room scene layout, participants, and the seating arrangement of participants. The smart glasses present the photos of the conference room scene in front of the user in a virtual display.

[0127] Optionally, the conference room scene photo, i.e., the location selection image, may be a photo with additional information added, such as personal information and location information of other participants of the meeting who appear in a remote online form. The user may directly select his or her own meeting location on the location selection image.

[0128] It should be noted that the location of the conference access device is not specifically limited, and in the embodiment of the present application, installation in a conference scene is taken as an example.

[0129] S202: In response to a selection instruction of a target user, determining a target position corresponding to the selection instruction in the conference scene, and the selection instruction acts on a position selection image;

[0130] In an embodiment of the present application, after the location selection image appears on the virtual display screen of the extended display device, the user can select the desired location in the selectable area of ​​the location selection image, and the extended display device determines the corresponding target location in the conference scene based on the selection instruction.

[0131] As an optional embodiment, the location selection situation may include: you can select a seat if there is a vacant seat, you can add a seat if there is no vacant seat, and you can also add a seat if there is a vacant seat. For example: there is a vacant seat in the on-site conference room, and the user selects a vacant seat as the location for his on-site meeting; another example: there is no vacant seat in the on-site meeting, and the user adds a seat as the location for his on-site meeting; another example: there is a vacant seat in the on-site conference room, but the vacant seat is the seat of the conference speaker, and the user can add a seat as the location for his on-site meeting.

[0132] Optionally, the glasses obtain data (scene, personnel, seats, etc.) transmitted from the server, and the user selects a position based on the scene and seat data.

[0133] In some embodiments of the present application, the extended reality device includes: a first shooting module. Before responding to the selection instruction of the target user, optionally, the remote conferencing method provided in the embodiment of the present application also includes: obtaining a gesture image of the target user through the first shooting; determining the selection information corresponding to the gesture image in the position selection image; and generating a selection instruction based on the selection information.

[0134] In an embodiment of the present application, the XR smart glasses can accept user gestures and hand movements as input instructions for selecting a meeting location.

[0135] As an optional embodiment, when the virtual display screen of the XR smart glasses is a meeting location selection interface, the camera of the XR smart glasses takes photos of the user's gestures to determine the user's selection instructions.

[0136] Optionally, the remote conference method provided in the embodiment of the present application also includes: determining coordinate information of the selection instruction on the position selection image; and determining the target position in the conference scene according to the coordinate information.

[0137] In an embodiment of the present application, the XR smart glasses receive and parse the user's instructions for selecting a meeting location, calculate the meeting location coordinates corresponding to the meeting location in the virtual screen at the conference site, and send the meeting location corresponding to the conference site to the conference access device.

[0138] As an optional embodiment, after the camera of the XR smart glasses takes a photo of the user's gesture, the main control device interprets the gesture to determine the position selected by the user in the virtual screen (i.e., the position selection image), queries the specific position coordinates of the selected position in the virtual screen, and determines the specific position coordinates of the meeting position selected in the virtual screen in the real conference room through the coordinate conversion relationship between the virtual screen and the real conference room.

[0139] Optionally, after selecting the user's virtual meeting location in the real conference room, the XR smart glasses generate an initial screen of the on-site conference room.

[0140] S203: Sending the target location to the conference access device;

[0141] In the embodiment of the present application, after the user's target position is determined, the target position is sent to the conference access device so that the conference access device generates playback content according to the target position.

[0142] As an optional embodiment, the target location may also be processed and confirmed by the conference access device. For example, after obtaining the user selection instruction, the selection instruction and the corresponding image are sent to the conference access device so that the conference access device can determine the target location.

[0143] In some embodiments of the present application, the extended reality device includes: a posture acquisition module. Before receiving the playback content sent by the conference access device, optionally, the remote conference method provided in the embodiment of the present application also includes: obtaining the posture information of the target user through the posture acquisition module; and sending the posture information to the conference access device.

[0144] In an embodiment of the present application, after the remote conference user officially enters the meeting, the on-site conference room information will be synchronously presented to the user in real time. Therefore, after confirming the target location, the posture acquisition module can obtain the user's posture information in real time and send the posture information to the conference access device.

[0145] Optionally, after the user debugs and confirms the effect of the initial screen of the conference room, the user officially joins the meeting. The XR smart glasses calculate the user's head posture through gyroscopes, accelerometers, etc., and measure the user's line of sight through an eye tracker.

[0146] Optionally, the method for generating and updating the conference room screen according to the user's head posture parameters and the user's sight point parameters can be: the conference access device is equipped with more than three cameras at different locations of the conference site, and these cameras take photos of the conference room scene in real time. The conference access device sends the position coordinates of multiple cameras and the photos taken at the same time to the XR smart glasses. The XR smart glasses obtain the user's current sight point through the eye tracker and calculate the user's observation angle. According to the multiple conference room screen photos taken at the same time, the user's meeting position, and the sight point, the XR smart glasses generate the conference room scene observed by the user's meeting position from the perspective of attention based on this information.

[0147] Optionally, the method for generating and updating the conference room screen according to the user's head posture parameters and the user's sight point parameters can also be: the conference access device is equipped with more than three cameras at different locations of the conference site, and these cameras take real-time photos of the conference room scene, and build a 3D scene based on these photos. After determining the observer's position and head posture, the screen photos from the observer's perspective are generated. The XR smart glasses control the micro display optical components to present them in front of the user in an augmented reality manner.

[0148] In an embodiment of the present application, the XR smart glasses receive the conference site sounds collected by the conference access device through a microphone, and convert the conference site sounds collected by the microphone into conference site sounds heard at the user's virtual meeting location.

[0149] Optionally, the sound conversion generation method may be: determining the location of the sound source, determining the location of the collection microphone, determining the user's virtual meeting location, and based on the above relative position parameters, converting the characteristic parameters of the sound collected by the microphone into the characteristic parameters of the sound received at the user's virtual meeting location through an algorithm. The computing processor of the XR smart glasses controls the acoustic modules such as the speakers of the XR smart glasses to present the live sound of the conference room around the user. During the user's speaking stage, the microphone of the XR smart glasses collects the user's voice and transmits it to the conference access device at the meeting site.

[0150] S204: receiving the playback content sent by the conference access device, and playing the playback content on the virtual display screen, where the playback content is generated by the conference access device according to the target location.

[0151] In an embodiment of the present application, after the conference access device generates playback content according to the target location, the playback content is sent to the extended reality device. The extended reality device receives the playback content sent by the conference access device and plays the playback content on a virtual display screen.

[0152] As an optional embodiment, the XR smart glasses send instructions to the conference access device, and the conference access device can be moved to the meeting location selected by the user. The camera and microphone of the conference access device observe and receive the current scene image information and sound information of the conference room at the meeting location.

[0153] Optionally, another method for XR smart glasses to generate conference room scene information is: for sound information, the microphone of the conference access device receives and records the sound information of the conference room scene at its original position, and sends the recorded sound information including sound source, timbre, pitch, loudness, etc., the position of the conference access device, and the user's meeting position to the XR smart glasses. The XR smart glasses generate the conference scene sound information received and observed at the user's meeting position based on this information, and the XR smart glasses control the speaker to play the sound according to this conference scene sound information and present it to the user.

[0154] Optionally, for visual information, the conference access device is equipped with more than three cameras at different locations on the conference site, and these cameras take photos of the conference room scene in real time. The conference access device sends the position coordinates of multiple cameras and photos taken at the same time to the XR smart glasses. The XR smart glasses obtain the user's current line of sight through the eye tracker and calculate the user's observation angle. Based on the multiple conference room screen photos taken at the same time, the user's meeting position, and the line of sight, the XR smart glasses generate the conference room scene observed by the user's meeting position from the attention perspective based on this information. The XR smart glasses control the display screen and optical components to present this conference room scene to the user in a virtual display manner.

[0155] In this scenario, the visual information received by the user through the XR smart glasses will switch as the user's attention angle switches, and the user's attention angle generally switches as the conference speaker switches. In this scenario, the XR smart glasses can also receive and record the sound information emitted by the user and transmit it to the conference access device. The conference access device controls the speaker to emit sound at the user's location at the conference site, creating the effect that the user is speaking at the conference site.

[0156] See also Figure 4 , Figure 4The scene diagram of the remote conference method provided in the embodiment of the present application, the remote conference system may include a conference access device 400 and a processing device 500, the conference access device 400 and the processing device 500 may be connected to each other in any manner, including but not limited to signal communication through electronic circuits, communication through wireless signals, the wireless signal may be a computer network communication of the TCP / IP protocol suite (TCP / IP Protocol Suite, TCP / IP), the User Datagram Protocol (User Datagram Protocol, UDP), etc. The conference access device 400 may receive control signals from a remote control or a control panel, and the conference access device 400 may also receive instruction information sent by the processing device 500, and the conference access device 400 may perform corresponding operations according to the corresponding instruction information, such as the remote conference method in the present application.

[0157] In the embodiment of the present application, the conference access device 400 includes but is not limited to a remote conference access device or other forms of conference access devices at an offline conference room.

[0158] Those skilled in the art will understand that Figure 4 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 4 More or fewer conference access devices as shown in Figure 4 Only one conference access device is shown, and no specific limitation is given here.

[0159] In addition, if Figure 4 As shown, the processing device 500 may include any hardware device capable of performing data processing and instruction sending, such as a CPU or a single-chip microcomputer embedded in the conference access device, and the specific details are not limited here.

[0160] It should be noted that Figure 4 The scenario diagram of the remote conference system shown is only an example. The remote conference system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of remote conference systems and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0161] For details, please refer to Figure 5 , Figure 5 : is a schematic diagram of a scenario in which a conference access device according to an embodiment of the present application performs the remote conference method, wherein the specific execution process of the conference access device performing the remote conference method is as follows:

[0162] S501: Determine a position selection image corresponding to a conference scene through a plurality of second shooting modules, wherein the plurality of second shooting modules are arranged at different positions of the conference scene.

[0163] In the embodiment of the present application, the number and installation position of the second shooting module are not determined according to the scene, and the actual test scene can achieve the effect achieved by the three-dimensional reconstruction technology.

[0164] Optionally, a second shooting module performs a three-dimensional reconstruction technique to construct a one-to-one meeting scene. The second shooting module obtains information and transmits it to a processing device for unified processing to obtain relevant information such as the meeting scene, personnel, and seats. The processing device transmits the processed information to the extended reality device.

[0165] Optionally, the remote conference method provided in the embodiment of the present application also includes: acquiring multiple scene images of the conference scene through multiple second shooting modules; and determining a position selection image based on the multiple scene images.

[0166] In the embodiments of the present application, Figure 6 As shown in the conference scene diagram, several second shooting modules (Camera) arranged in advance in the conference room will fully capture the scene of the conference place and transmit the corresponding pictures to the server. The server can be a processing device of on-site conference access devices, smart glasses, cloud servers and other equipment.

[0167] It should be noted that Figure 6 The shared large screen shown in the figure is only an example, which can be the main screen in the conference room, etc. The positions selected by users A, B, C, D, E, and F and the installation positions of multiple second shooting modules are also examples and can be adjusted according to actual application conditions.

[0168] As an optional embodiment, the server may use 3D reconstruction technology to generate a 3D model of the conference scene that is the same as the current conference scene.

[0169] It should be noted that 3D reconstruction technology involves the following steps, and finally generates a 3D model that is the same as the scene captured by the picture: 1. Data acquisition: Use multiple cameras to take pictures of the room from different angles. These pictures need to cover every corner of the room so that the complete appearance of the room can be captured. 2. Feature extraction: Extract key feature points such as corners and edges from each picture. 3. Feature matching: Find the same feature points between different pictures and determine the correspondence between them. 4. Camera calibration: Determine the camera's intrinsic parameters (focal length, principal point, etc.) and extrinsic parameters (camera position and orientation), which is usually done by taking pictures of known patterns. 5. 3D point cloud generation: Using feature matching and camera parameters, convert the points on the 2D image into points in 3D space to form a point cloud. 6. Surface reconstruction: Recover the surface of the object from the point cloud, which may involve techniques such as mesh generation and surface fitting. 7. Texture mapping: Map the texture of the original picture to the reconstructed 3D model to obtain a realistic appearance. 8. Post-processing: May include steps such as model smoothing, noise removal, and optimization.

[0170] Optionally, the remote conference method provided by the embodiment of the present application also includes: determining a first position area and a second position area in the conference scene based on multiple scene images, wherein the first position area is a position area that cannot be selected because there are already participating users, and the second position area is a position area where there are no participating users; determining a position selection image based on the first position area and the second position area.

[0171] As an optional embodiment, selectable position areas and non-selectable position areas in the conference scene are determined based on a plurality of scene images, and a position selection image is generated based on the selectable position areas and non-selectable position areas.

[0172] S502: Sending the location selection image to the extended reality device.

[0173] In an embodiment of the present application, a seat selection interface is generated based on the location data of the on-site meeting, and a location selection image corresponding to the seat selection interface is sent to the extended reality device for display so that the user can select a seat.

[0174] S503: Receive the target location sent by the extended reality device.

[0175] In the embodiments of the present application, Figure 7 The target position selection result schematic diagram shown receives the user selection result, i.e., the target position, sent by the extended reality device, and generates corresponding playback content based on the target position.

[0176] It should be noted that Figure 7The rectangle at the selected position on the glasses end shown in the figure may be the direction of the user's field of view of the extended reality device, that is, the field of view seen by the remote user in the extended reality device is the field of view facing the shared large screen according to the selected position.

[0177] S504: Generate playback content according to the target location, and send the playback content to the extended reality device.

[0178] In an embodiment of the present application, based on the seat selected by the user and the three-dimensional model of the on-site meeting reconstructed by the processing device, the corresponding system automatically moves, rotates, scales, and performs other operations on the three-dimensional model of the meeting scene to try to match the location of the remote meeting scene, and presents the three-dimensional model of the meeting scene on the glasses end.

[0179] Optionally, a corresponding 3D character model will also be generated and placed in the 3D model of the conference site. The 3D model will also be bound to some common basic animation states (for example; Idle, talking, standing up, sitting down, turning head, clapping). In addition, the presentation of more advanced and complex actions can be achieved by using motion capture technology to drive the corresponding model.

[0180] In an embodiment of the present application, according to the position selected by the glasses, a three-dimensional model scene is generated at the conference site, and the scene is moved, rotated, and placed as shown in the figure, so that the participants at the glasses can be immersed in the on-site meeting. The glasses can combine the 3dof (or 6dof) of the glasses to rotate and observe the scene, or roam the scene.

[0181] It should be noted that the server also obtains on-site information in real time and transmits it to the glasses, which displays the real-time situation of different conference scenes and drives the expressions and movements of the characters in the three-dimensional model scene.

[0182] As an optional embodiment, by inputting the virtual meeting location and the initial observation location, such as facing the large screen of the meeting room, the conference room screen seen by the user at the virtual meeting location (target location) can be obtained. As an example, the process of generating a three-dimensional model of the conference scene through three-dimensional reconstruction and then generating the scene screen at a specified location and specified observation angle can be a NERF rendering process.

[0183] It should be noted that NERF (Neural Radiance Field) is an emerging rendering technology, and its core goal is to solve how to generate images from new perspectives from some captured images. Unlike traditional 3D reconstruction methods, NERF does not represent the scene as an explicit form such as point cloud, grid, voxel, etc., but models the scene as a continuous 5D radiation field, which is implicitly stored in the neural network. NERF inputs sparse multi-angle images to train a neural radiation field model, and then renders clear photos from any perspective based on this model. The rendering process is to input the position and direction of the light emitted from a certain perspective, as well as the corresponding coordinates, and then send them to NERF to obtain the volume density and color, and finally obtain the final image through volume rendering.

[0184] Optionally, the remote conferencing method provided in the embodiment of the present application also includes: receiving posture information of the target user sent by the extended reality device; determining the target screen based on the posture information and multiple scene images, and determining the audio content corresponding to the target screen based on the posture information and multiple audio data.

[0185] In an embodiment of the present application, after the remote conference user officially enters the meeting, the on-site conference room information will be synchronously presented to the user in real time. Therefore, after confirming the target location, the posture acquisition module can obtain the user's posture information in real time and send it to the conference access device, and the conference access device receives the posture information sent by the extended reality device in real time.

[0186] As an optional embodiment, the conference room scene information mainly refers to visual picture information and auditory sound information, and if possible, it can also include olfactory information. Visual picture information of the on-site conference room observed at the virtual conference location (ie, the target location) is generated.

[0187] In some embodiments of the present application, the conference access device includes multiple audio acquisition modules. Optionally, the remote conference method provided by the embodiments of the present application also includes: acquiring multiple scene images of the conference scene through multiple second shooting modules; acquiring multiple audio data from different angles in the conference scene through multiple audio acquisition modules, wherein the multiple audio acquisition modules are arranged at different positions of the conference scene; determining the target picture according to the multiple scene images and the target positions, and determining the audio content corresponding to the target picture according to the multiple audio data and the target positions; generating playback content according to the target picture and the audio content.

[0188] In an embodiment of the present application, surrounding environment data can be collected through the gyroscope, magnetic sensor, IMU, etc. carried by the smart glasses, and the 6Dof tracking technology can be used to obtain the user's current head posture parameters, thereby obtaining the user's current perspective of the conference room, and responding to the user's instruction to select the current meeting location on the virtual screen, determining the user's virtual meeting location in the on-site conference room, and updating the user's field of view in the previous three-dimensional model of the meeting scene, and the meeting scene screen is also updated.

[0189] It should be noted that 6DOF (six degrees of freedom) provides a more complete description of spatial motion, including three translational degrees of freedom and three rotational degrees of freedom: X-axis translation: movement along the X-axis. Y-axis translation: movement along the Y-axis. Z-axis translation: movement along the Z-axis. Rotation around the X-axis: usually called pitch, the rotation of an object around the X-axis. Rotation around the Y-axis: usually called yaw, the rotation of an object around the Y-axis. Rotation around the Z-axis: usually called roll, the rotation of an object around the Z-axis. 6DOF allows objects to perform more complex movements in space, including aircraft in flight simulators, precise manipulation of robots, and full range of movement of users' heads and hands in virtual reality.

[0190] Optionally, the user's line of sight can be tracked by an eye tracker on the smart glasses to obtain the user's current viewing angle of the conference room, update the user's field of view in the three-dimensional model of the conference scene, and update the conference scene screen at the same time. The computing processor controls the display screen to display the generated screen, and after imaging and transmission by the optical components, the virtual screen of the conference room is presented to the user.

[0191] Through the above steps, the virtual image presented to the user changes with the observation line of sight and the location of the remote participant, and the virtual image is equivalent to the on-site conference room image.

[0192] In an embodiment of the present application, for the generation of auditory sound information of the on-site conference room observed at the virtual meeting location, the conference site sounds collected by the conference access device through the microphone array (i.e., audio acquisition module) arranged in the on-site conference room can be received by XR smart glasses, and the conference site sounds collected by the microphone can be converted into conference site sounds heard at the user's virtual meeting location.

[0193] Optionally, the method for converting and generating sound may be: determining the location of the sound source, determining the location of the collection microphone, determining the user's virtual meeting location, and based on the above relative position parameters, converting the characteristic parameters of the sound collected by the microphone into the characteristic parameters of the sound received at the user's virtual meeting location through an algorithm. Using spatial audio technology, the computing processor of the XR smart glasses controls the acoustic modules such as the XR speakers to present the live sound of the conference room around the user.

[0194] In an embodiment of the present application, the second shooting module on site can shoot the on-site conditions in real time and transmit it to the server; the server can capture and analyze the status of the on-site participants, who is speaking, actions and other information in real time, and then generate a corresponding data packet based on the identity and location of the speaker and the corresponding information of the three-dimensional model of the conference site, and transmit it to the glasses end.

[0195] After the glasses obtain and analyze the data, they drive the corresponding character model in the three-dimensional model of the conference scene on the glasses to drive the lip movements and movements, and at the same time move the position of the sound source to the corresponding character model position in the three-dimensional model of the conference scene.

[0196] It should be noted that the user of the glasses can also look at the person who is speaking by rotating.

[0197] Optionally, by continuously repeating the above steps, the glasses can see a conference effect similar to the actual scene, characters, movements, and sound positions of the conference.

[0198] Through the embodiments of the present application, the sound heard by the remote conference user changes with the position selected by the user, and this sound is equivalent to the on-site sound. It is achieved that the remote user can arbitrarily switch the observation and listening position and angle in the on-site conference room in the form of virtual conference attendance through XR smart glasses. The smart glasses control the camera to take pictures of the user's image, control the microphone to collect the user's voice, and transmit it to the on-site conference access device. The conference access device can control the on-site optical instruments to holographically display the user's image at the virtual conference location selected by the user, and can control the on-site speaker array to play the sound equivalent to the user's sound at the conference location in the conference room.

[0199] In addition, if the on-site participants use the PC client to participate in the meeting, the PC client can directly obtain the identity and voice of the speaker. If the video conference is turned on, the speaker's video information can be directly obtained.

[0200] In the embodiments of the present application, Figure 8 The overall structural diagram of the remote conference method shown in the figure shows that the main terminal can have a server terminal (processing device of the conference access device) that provides related services, which is responsible for the management of the online conference of the XR glasses. Different server services will be obtained when logging into the client in different scenarios, and one or more second shooting modules at the conference site will be controlled to obtain the current environment and scene conditions, which are used to analyze the location layout of the conference seats, whether there are people at the location, etc., and transmit them to the server in real time. If the participants participate in the meeting on the computer side (PC side), the computer side will synchronize the shared information of the meeting, and the camera module on the computer side can obtain the information of capturing the half-length head portrait and transmit it to the server in real time; at the same time, the virtual image of the participants on the glasses side can be holographically projected.

[0201] Specifically, the on-site participants speak through the PC client and transmit the corresponding identity, sound, and video information to the server. The server obtains the 3D model of the conference site and the character model information based on the corresponding identity, and then uses the video information to obtain the character's actions and expressions. These are packaged into information data packets and transmitted to the glasses. After obtaining and analyzing the data, the glasses drive the corresponding character model in the 3D model of the conference site on the glasses to drive the lip shape and action, and at the same time move the position of the sound source to the corresponding character model position in the 3D model of the conference site.

[0202] In the embodiment of the present application, when the XR glasses log in to the meeting, they will first generate a corresponding meeting scene on the glasses according to the on-site environment scanned by the server and the actual layout of the venue, and then preliminarily allocate a position to the glasses according to the vacant positions on the scene; the scene and position displayed on the glasses are basically consistent with the on-site meeting scene. Combined with the slam of the glasses, the customers on the glasses can also roam in the scene, or turn their heads to look at someone or the person who is speaking while sitting. In addition, when the customers on the glasses hear the conversations in different meetings, the sound they hear is similar to the real scene, which is the sound effect of the 3D scene, depending on the location; according to the on-site character information, the server transmits relevant information, and builds the corresponding on-site character model in the 3D scene on the glasses, which can synchronize the on-site character movements and expressions and other information in real time.

[0203] Through the embodiment of the present application, a technical solution is adopted in which a position selection image corresponding to a conference scene is determined by multiple second shooting modules, and the multiple second shooting modules are arranged at different positions of the conference scene; the position selection image is sent to an extended reality device; a target position sent by the extended reality device is received; playback content is generated according to the target position, and the playback content is sent to the extended reality device. According to the user's selection instruction, the corresponding target position is returned to the conference access device, and then the playback content generated by the conference access device according to the target position is received and played, thereby achieving the technical effect of bringing the audio-visual experience of the conference site to the target users who use the extended reality device to remotely participate in the meeting, and improving the remote users' meeting experience.

[0204] In order to facilitate better implementation of the remote conference method of the present application, the present application also provides a remote conference device based on the remote conference method. The meanings of the terms are the same as those in the remote conference method, and the specific implementation details can refer to the description in the method embodiment.

[0205] See also Fig. 9 , Fig. 9 : is a schematic diagram of the structure of a remote conference device provided in an embodiment of the present application, wherein the remote conference device 900 is applied to an extended reality device, which can be specifically as follows:

[0206] The first processing module 901 is used to receive a location selection image for a conference scene sent by a conference access device, and the location selection image is displayed on a virtual display screen of an extended reality device;

[0207] The first processing module 901 is further used to determine the target position corresponding to the selection instruction in the position selection image in response to the selection instruction of the target user;

[0208] A first sending module 902 is used to send the target position to the conference access device, and the conference access device generates playback content according to the target position and posture information, and the posture information is obtained by the posture acquisition module of the extended reality device;

[0209] The first processing module 901 is further configured to receive the playback content sent by the conference access device and play the playback content on the virtual display screen.

[0210] Optionally, in some embodiments of the present application, the first processing module 901 is used to:

[0211] Determine the coordinate information of the selection instruction on the position selection image;

[0212] Determine the target location in the conference scene based on the coordinate information.

[0213] Optionally, in some embodiments of the present application, the first processing module 901 is further configured to:

[0214] Acquire a gesture image of the target user through a first shooting;

[0215] Determine selection information corresponding to the gesture image in the position selection image;

[0216] A selection instruction is generated according to the selection information.

[0217] Optionally, in some embodiments of the present application, the first sending module 902 is configured to:

[0218] Acquire the target user's posture information through the posture collection module;

[0219] Send posture information to the conference access device.

[0220] In the embodiment of the present application, the first processing module 901 first receives a position selection image for a conference scene sent by a conference access device, and the position selection image is displayed on a virtual display screen of an extended reality device. Next, the first processing module 901 responds to a selection instruction of a target user, determines a target position corresponding to the selection instruction in the position selection image, and then, the first sending module 902 sends the target position to the conference access device. The conference access device generates playback content based on the target position and posture information, and the posture information is obtained by a posture acquisition module of the extended reality device. Then, the first processing module 901 receives the playback content sent by the conference access device, and plays the playback content on the virtual display screen.

[0221] Among them, the embodiment of the present application receives a position selection image for a conference scene sent by a conference access device, and the position selection image is displayed on a virtual display screen of an extended reality device; in response to a selection instruction of a target user, determines a target position corresponding to the selection instruction in the position selection image; sends the target position to the conference access device, and the conference access device generates playback content according to the target position and posture information, and the posture information is obtained by a posture acquisition module of the extended reality device; receives the playback content sent by the conference access device, and plays the playback content on a virtual display screen, and returns the corresponding target position to the conference access device according to the user's selection instruction, and then receives and plays the playback content generated by the conference access device according to the target position, thereby achieving the technical effect of bringing the audio-visual experience of the conference site to the target users who use the extended reality device to remotely participate in the meeting, and improving the remote users' meeting experience.

[0222] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of a remote conference device provided in an embodiment of the present application, wherein the remote conference device 1000 is applied to a conference access device, which can be specifically as follows:

[0223] The second processing module 1001 is used to determine the position selection image corresponding to the conference scene through multiple second shooting modules, and the multiple second shooting modules are arranged at different positions of the conference scene;

[0224] The second sending module 1002 is used to send the location selection image to the extended reality device;

[0225] The second processing module 1001 is further used to receive a target position sent by the extended reality device, and the target position is generated in the position selection image according to the selection instruction of the target user;

[0226] The second processing module 1001 is further used to generate playback content according to the target location, and send the playback content to the extended reality device, and the playback content is played on the extended reality device.

[0227] Optionally, in some embodiments of the present application, the second processing module 1001 is used to:

[0228] Acquire multiple scene images of the conference scene through multiple second shooting modules;

[0229] An image is selected based on a plurality of scene images.

[0230] Optionally, in some embodiments of the present application, the second processing module 1001 is further configured to:

[0231] Determine a first location area and a second location area in the conference scene according to the multiple scene images, wherein the first location area is a location area where conference users already exist and cannot be selected, and the second location area is a location area where no conference users exist;

[0232] A position selection image is determined based on the first position area and the second position area.

[0233] Optionally, in some embodiments of the present application, the second processing module 1001 is further configured to:

[0234] Acquire multiple scene images of the conference scene through multiple second shooting modules;

[0235] Acquire multiple audio data from different angles in the conference scene through multiple audio acquisition modules, wherein the multiple audio acquisition modules are set at different positions of the conference scene;

[0236] Determining a target picture according to the multiple scene images and the target positions, and determining audio content corresponding to the target picture according to the multiple audio data and the target positions;

[0237] Generate playback content based on target image and audio content.

[0238] Optionally, in some embodiments of the present application, the second processing module 1001 is further configured to:

[0239] Receiving the target user's posture information sent by the extended reality device;

[0240] A target picture is determined according to the posture information and the multiple scene images, and audio content corresponding to the target picture is determined according to the posture information and the multiple audio data.

[0241] In the embodiment of the present application, the second processing module 1001 first determines the position selection image corresponding to the conference scene through multiple second shooting modules, and the multiple second shooting modules are set at different positions of the conference scene. Then, the second sending module 1002 sends the position selection image to the extended reality device. Subsequently, the second processing module 1001 receives the target position sent by the extended reality device, and then the second processing module 1001 generates playback content according to the target position and sends the playback content to the extended reality device. The playback content is played on the extended reality device.

[0242] Among them, the embodiment of the present application determines the position selection image corresponding to the conference scene through multiple second shooting modules, and the multiple second shooting modules are set at different positions of the conference scene; sends the position selection image to the extended reality device; receives the target position sent by the extended reality device, and the target position is generated in the position selection image according to the selection instruction of the target user; generates playback content according to the target position, and sends the playback content to the extended reality device. The technical solution for playing the playback content on the extended reality device, returns the corresponding target position to the conference access device according to the user's selection instruction, and then receives and plays the playback content generated by the conference access device according to the target position, thereby achieving the technical effect of bringing the audio-visual experience of the conference site to the target users who use the extended reality device to remotely participate in the meeting, and improving the remote user's meeting experience.

[0243] In addition, the present application also provides an electronic device, such as Fig.11 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0244] The electronic device may include one or more processors 1101 of processing cores, one or more computer-readable storage media memories 1102, a power supply 1103, an input unit 1104, a camera module 1105 and other components. Those skilled in the art will appreciate that Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0245] The processor 1101 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1102, and calling data stored in the memory 1102, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 1101 may include one or more processing cores; preferably, the processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1101.

[0246] The memory 1102 can be used to store software programs and modules. The processor 1101 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1102 may also include a memory controller to provide the processor 1101 with access to the memory 1102.

[0247] The electronic device also includes a power supply 1103 for supplying power to each component. Preferably, the power supply 1103 can be logically connected to the processor 1101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 1103 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0248] The electronic device may further include an input unit 1104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0249] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 1101 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 1102 according to the following instructions, and the processor 1101 will run the application program stored in the memory 1102, thereby implementing the steps in any remote conference method provided in the embodiments of the present application.

[0250] The embodiment of the present application receives a position selection image for a conference scene sent by a conference access device, and the position selection image is displayed on a virtual display screen of an extended reality device; in response to a selection instruction of a target user, determines a target position corresponding to the selection instruction in the conference scene, and the selection instruction acts on the position selection image; sends the target position to the conference access device; receives playback content sent by the conference access device, and plays the playback content on the virtual display screen. The playback content is generated by the conference access device according to the technical solution of the target position, and according to the user's selection instruction, the corresponding target position is returned to the conference access device, and then the playback content generated by the conference access device according to the target position is received and played, thereby achieving the technical effect of bringing the audio-visual experience of the conference site to the target users who use the extended reality device to remotely participate in the meeting, and improving the remote users' meeting experience.

[0251] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0252] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0253] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any remote conferencing method provided in the present application.

[0254] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0255] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0256] Since the instructions stored in the computer-readable storage medium can execute the steps in any remote conferencing method provided in the present application, the beneficial effects that can be achieved by any remote conferencing method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0257] The above is a detailed introduction to a remote conferencing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A remote conference method, characterized in that: Applied to an extended reality device, the method comprises: Receiving a location selection image for a conference scene sent by a conference access device, wherein the location selection image is displayed on a virtual display screen of the extended reality device; In response to a selection instruction of a target user, determining a target position corresponding to the selection instruction in the position selection image; Sending the target position to the conference access device, the conference access device generating playback content according to the target position and posture information, the posture information being acquired by a posture acquisition module of the extended reality device; The playback content sent by the conference access device is received, and the playback content is played on the virtual display screen.

2. The method according to claim 1, characterized in that In response to a selection instruction of a target user, determining a target position corresponding to the selection instruction in the conference scene includes: Determine the coordinate information of the selection instruction on the position selection image; The target position in the conference scene is determined according to the coordinate information.

3. The method according to claim 1, characterized in that The extended reality device includes: a first shooting module, the first shooting module is used to obtain an image in front of the extended reality device, and before responding to the selection instruction of the target user, the method further includes: Acquire a gesture image of the target user through the first shooting; Determine selection information corresponding to the gesture image in the position selection image; The selection instruction is generated according to the selection information.

4. The method according to claim 1, characterized in that The extended reality device includes: the posture acquisition module, and before receiving the playback content sent by the conference access device, the method further includes: Acquiring the posture information of the target user through the posture acquisition module; The posture information is sent to the conference access device.

5. A remote conference method, characterized in that: Applied to a conference access device, the conference access device includes a plurality of second shooting modules, and the method includes: Determine a position selection image corresponding to a conference scene by using the multiple second shooting modules, wherein the multiple second shooting modules are arranged at different positions of the conference scene; Sending the location selection image to an extended reality device; receiving a target position sent by the extended reality device, wherein the target position is generated in the position selection image according to a selection instruction of a target user; The playback content is generated according to the target position, and the playback content is sent to the extended reality device, and the playback content is played on the extended reality device.

6. The method according to claim 5, characterized in that The step of determining the position selection image corresponding to the conference scene by using the plurality of second shooting modules includes: Acquire a plurality of scene images of the conference scene through the plurality of second shooting modules; The position selection image is determined based on the plurality of scene images.

7. The method according to claim 6, characterized in that The step of determining the position selection image according to the plurality of scene images comprises: Determine a first location area and a second location area in the conference scene according to the multiple scene images, wherein the first location area is a location area where conference users already exist and cannot be selected, and the second location area is a location area where no conference users exist; The position selection image is determined according to the first position area and the second position area.

8. The method according to claim 5, characterized in that The conference access device includes a plurality of audio acquisition modules, and the generating of playback content according to the target location includes: Acquire a plurality of scene images of the conference scene through the plurality of second shooting modules; Acquire multiple audio data at different angles in the conference scene through the multiple audio acquisition modules, wherein the multiple audio acquisition modules are arranged at different positions of the conference scene; Determine a target picture according to the multiple scene images and the target position, and determine audio content corresponding to the target picture according to the multiple audio data and the target position; The playback content is generated according to the target picture and the audio content.

9. The method according to claim 8, characterized in that Determining a target picture according to the multiple scene images and the target position, and determining audio content corresponding to the target picture according to the multiple audio data and the target position, includes: Receiving the posture information of the target user sent by the extended reality device; The target picture is determined according to the posture information and the multiple scene images, and the audio content corresponding to the target picture is determined according to the posture information and the multiple audio data.

10. A remote conference device, characterized in that: Applied to an extended reality device, the device comprises: A first processing module, configured to receive a location selection image for a conference scene sent by a conference access device, wherein the location selection image is displayed on a virtual display screen of the extended reality device; The first processing module is further configured to determine, in response to a selection instruction of a target user, a target position corresponding to the selection instruction in the position selection image; A first sending module, configured to send the target position to the conference access device, wherein the conference access device generates playback content according to the target position and posture information, wherein the posture information is acquired by a posture acquisition module of the extended reality device; The first processing module is further configured to receive the playback content sent by the conference access device and play the playback content on the virtual display screen.

11. A remote conference device, characterized in that: Applied to a conference access device, the conference access device includes a plurality of second shooting modules, and the method includes: A second processing module, configured to determine a position selection image corresponding to a conference scene through the plurality of second shooting modules, wherein the plurality of second shooting modules are arranged at different positions of the conference scene; A second sending module, used for sending the position selection image to an extended reality device; The second processing module is further used to receive a target position sent by the extended reality device, wherein the target position is generated in the position selection image according to a selection instruction of a target user; The second processing module is further configured to generate playback content according to the target position, and send the playback content to the extended reality device, so that the playback content is played on the extended reality device.

12. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the remote conference method according to any one of claims 1 to 9 when executing the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the remote conference method according to any one of claims 1 to 9 are implemented.

Images