Communication system and method

Abstract

The invention relates to a communication system and method. Communication systems and methods are described herein. In some examples, a plurality of participants are detected in an area, and a participant feature for each participant of the plurality of participants is obtained. A plurality of participant tiles are generated. A group interface for the communication session is generated based on the plurality of tiles and rendered for display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 537,042, entitled “Communication Systems and Methods,” filed on September 7, 2023, which is hereby incorporated by reference in its entirety. Technical Field

[0003] Aspects of the present disclosure relate to communication systems and methods, and more particularly to systems and methods for conducting a group conference with a plurality of participants disposed in an area. Background Art

[0004] The communication system can be used to conduct a group conference over a network. In some examples, the conference can include multiple participants co-located in an area. The participants can be located at different locations in the area and in various orientations relative to the camera. Summary of the invention

[0005] The specific implementations described and claimed herein provide systems and methods for group communication. In some specific implementations, multiple participants of a communication session are detected. The multiple participants are co-located in an area. Participant features of each of the multiple participants are obtained, and the participant features are captured using at least one image sensor. Multiple participant tiles are generated. Each participant tile in the participant tiles corresponds to the participant features of a corresponding one of the multiple participants. A group interface for the communication session is generated based on the multiple tiles. The group interface includes the participant features of each of the multiple participants. The group interface is rendered for display using at least one display. Therefore, even when one or more of the participants are blocked by structures in the area, each of the participants in the area can be clearly displayed to the recipient.

[0006] In some implementations, multiple participants of a communication session are detected. The multiple participants are co-located in an area. Participant features of each of the multiple participants are extracted from image data of the area. Multiple participant tiles are generated. Each participant tile in the participant tiles corresponds to a participant feature of a corresponding one of the multiple participants. A group interface for the communication session is generated based on the multiple tiles. The group interface includes the participant features of each of the multiple participants. The group interface is output for display during the communication session.

[0007] In some implementations, at least one image sensor is configured to capture image data of a plurality of participants of a communication session co-located in an area. At least one processor is configured to generate a group interface for the communication session based on the plurality of participant tiles. Each participant tile in the participant tiles is generated based on the image data and corresponds to a participant feature of a corresponding one of the plurality of participants. The group interface includes the participant features of each of the plurality of participants. The display is configured to display the group interface during the communication session.

[0008] Other specific implementations are also described and listed herein. In addition, although multiple specific implementations are disclosed, other specific implementations of the technology disclosed in the present invention will become apparent to those skilled in the art from the following detailed description showing and describing exemplary specific implementations of the technology disclosed in the present invention. As will be appreciated, the technology disclosed in the present invention may be modified in various aspects without departing from the spirit and scope of the technology disclosed in the present invention. Therefore, the drawings and specific embodiments are to be regarded as illustrative rather than restrictive in nature. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An exemplary environment for group communications is shown.

[0010] Figure 2 Generation of an exemplary group interface is shown.

[0011] Figure 3 Exemplary operations for group communications are shown.

[0012] Figure 4 An exemplary mobile device is shown.

[0013] Figure 5 An example computing system is shown. DETAILED DESCRIPTION

[0014] Various aspects of the currently disclosed technology relate to systems and methods for facilitating communication among groups. In some aspects, a communication session is initiated with multiple participants co-located in an area. Typically, a camera displays a single view of an area. In many cases, it is challenging to clearly observe each participant due to differences in position, orientation, distance, obstacles, lighting conditions, and / or other differences among the participants and / or parts of the area. The camera can detect a specific participant who is speaking and modify the view to focus on the specific participant. However, when focusing on a specific participant, non-verbal communications (e.g., facial expressions, gestures, etc.) from other participants may be missed. Therefore, one or more cameras capture image data of the area, and the face of each participant is obtained from the image data. Multiple tiles are generated, each tile corresponding to one of the participant faces, and a group interface is generated based on the multiple tiles so that the face of each participant is presented to one or more recipients involved in the communication session. In some instances, based on the corresponding tiles, the face of each recipient (including any recipient located outside the area) in the recipient is further presented in the group interface.

[0015] The tiles can be homogenized in various ways to account for differences among participants and / or corresponding areas in which each participant is located, thereby providing a consistent view of all participants in the group interface during a communication session. In some examples, each participant in the area is detected, and in some instances, an identification of each participant is determined. The identification of the participants can be used, for example, to homogenize the tiles with increased accuracy and / or to show portions of the participant's face that are uncertain in the image data, such as portions that are blocked and / or have low visibility (e.g., due to lighting conditions, sensor quality, etc.). Therefore, the currently disclosed technology facilitates communication among groups by providing a homogenized view of all participants throughout the duration of the communication session, which accounts for and addresses differences in viewing quality among the participants and / or portions of the areas in which the participants are located.

[0016] To begin a detailed description of an exemplary environment 100 for group communications, refer to Figure 1 to Figure 2. In some implementations, the sensor system 104 may be configured to capture image data (e.g., video), depth data, and other sensor data of the area 102. The sensor system 104 has one or more fields of view of the area 102. The sensor system 104 may have one or more image sensors corresponding to the field of view of the area 102. For example, the sensor system 104 may include multiple cameras placed at various locations within the area 102 and having various fields of view of the area 102. As another example, the sensor system 104 includes a single camera. The single camera may include multiple lenses, each lens having a separate image sensor, thereby using a single camera to provide multiple fields of view of the area 102. For example, the sensor system 104 may include a stereo camera with multiple fields of view of the area 102. In an example where the sensor system 104 includes multiple image sensors, the image sensors may have overlapping fields of view of the area 102, different fields of view of the area 102, and / or a combination thereof. The sensor system 104 may include other sensors for capturing sensor data corresponding to the area 102 and / or one or more participants. In some examples, sensor system 104 includes a camera corresponding to each of the participants, either as part of an integrated device or as a separate device.

[0017] A plurality of participants (e.g., a first participant 106, a second participant 108, a third participant 110, and a fourth participant 112) may be co-located in the area 102. In some implementations, the area 102 is in a structure 114, such that the participants 106 to 112 are co-located in the structure 114. For example, the area 102 may be within the interior of the structure 114. The structure 114 may be stationary, such that the structure 114 is fixed in place, or mobile, such that the structure 114 is configured to move along a movement path 116 from a starting point toward a destination. For example, the structure 114 may be a mobile device that transports the participants 106 to 112 along the movement path 116, wherein the area 102 is disposed within the interior of the mobile device. In some instances, one or more second participants may be located in the second area or move between the area 102 and other locations. The second area may be separate from the area 102, outside the structure 114, away from the structure 114, and / or outside the field of view of a sensor in the sensor system 104.

[0018] A communication session may be initiated for a participant. For example, participants 106 to 112 may be co-located in region 102 and communicate with the one or more second participants via the communication session. The communication session may be initiated in various ways, such as using displayed options, voice commands, gesture commands, user commands, and / or identifiers. Participants may similarly be added to or join a communication session. For example, a communication session 106 may be initiated between a remote participant and a first participant. Based on user preferences and consent, participants 108 to 112 may be detected within region 102, and an option to add participants 108 to 112 to the communication session may be presented. Upon approval, participants 108 to 112 may be added to the communication session. If any of participants 108 to 112 refuses to join the communication session, sensor data corresponding to the participant may be excluded from the communication session. In some examples, a communication session is initiated between a group placed in region 102 and one or more second participants in one or more corresponding regions. In conjunction with initiation of the communication session, each of the participants in the group (e.g., participants 106-112) may be detected in area 102, while a second participant is similarly detected in a corresponding area. Participants may be detected using sensor data captured by sensor system 104, user input, and / or other data corresponding to participants and / or areas.

[0019] In some implementations, the sensor system 104 captures participant features of each of the participants 106 to 112 in the area 102 according to user consent. The participant features may include facial features of the participants 106 to 112. A participant tile is generated for each of the participants 106 to 112 based on the participant features. More specifically, a first participant tile 200 is generated based on the participant features corresponding to the first participant 106, a second participant tile 202 is generated based on the participant features corresponding to the second participant 108, a third participant tile 204 is generated based on the participant features corresponding to the third participant 110, and a fourth participant tile 206 is generated based on the participant features corresponding to the fourth participant 112. For example, each of the participant tiles 200 to 206 may include the face of the corresponding participant 106 to 1112. A group interface 208 is generated for the communication session based on the participant tiles 200 to 206. The group interface 208 may be similarly generated based on the participant tiles corresponding to the participants located in the second area. In one example, the group interface 208 is generated by snapping together the participant tiles 200 - 206 .

[0020] Group interface 208 is displayed for region 102 and the second region. In some examples, group interface 208 displayed to recipients in the second region corresponds to participants in region 102, and a different group interface displayed to participants in region 102 corresponds to participants in the second region. In other examples, group interface 208 corresponds to all participants regardless of the region in which the participants are located.

[0021] Based on the participant tiles 200-206, the participants 106-112 can be displayed in an order in the group interface 208. The order can include the participants 106-112 arranged on the group interface 208 based on the corresponding positions within the area 102, and the order can change if the participants 106-112 move or change positions within the area 102. The order can be updated or otherwise determined based on the level of participation of each of the participants 106-112 in the communication session. For example, the participant 112 can be present in the area 102, but not actively participating in the communication session, and is arranged on the group interface 208 accordingly relative to the participants 106-110.

[0022] Regardless of any differences in viewing quality of sensor system 104 among participants 106-112 and / or portions of area 102, group interface 208 may provide a unified presentation of each of participant tiles 200-206 such that the faces of each of participants 106-112 are displayed simultaneously with equal effectiveness. Figure 2 It will be appreciated that there may be differences in positions, orientations, distances, obstructions, lighting conditions, and / or other differences among participants 106-112 and / or portions of area 102. Figure 2 In the example shown, the first participant 106 and the second participant 108 are placed at a greater distance from the sensor system 104 than the third participant 110 and the fourth participant 112, so that the participants 106 to 108 appear smaller in size relative to the participants 110 to 112. In addition, the participants 106 to 112 face different directions or are otherwise oriented differently from each other, so that the participants 106 to 112 appear different and / or portions of their faces are not discernible. Similarly, the second participant 108 is in a darker lighting condition than the first participant 106 and the third participant 110, and the fourth participant 112 is partially in a darker lighting condition. The first participant 106 is also partially blocked from the field of view of the sensor system 104 by the object 210. To account for such differences, the participant tiles 200 to 206 can be homogenized.

[0023] In some examples, the participant tiles 200 to 206 may be homogenized by one or more optimization actions, such as tile optimization actions and / or feature optimization actions. The optimization actions may be applied in two dimensions and / or three dimensions, and the group interface 208 may be rendered for two-dimensional or three-dimensional display. The tile optimization actions may modify one or more of the participant tiles 200 to 206. For example, the tile optimization actions may include translating, scaling, tilting, straightening, rotating, cropping, and / or transforming the participant tiles 200 to 206. The tile optimization actions form the participant tiles 200 to 206 of uniform size, shape, and orientation for arrangement in the group interface 208. In addition, the participant features of the corresponding participants 106 to 112 are similarly positioned relative to each other within the participant tiles 200 to 206. For example, the participant feature of the first participant 106 may be positioned at the center of the first participant tile 200 with a predetermined amount of spacing around the participant feature within the first participant tile 200. Tile optimization actions may be applied to the participant tiles 202 - 206 to create the same positioning and spacing as the first participant tile 200 to provide the appearance of uniform size and spacing of the participants 106 - 112 in the group interface 208 .

[0024] The tile optimization action may further include applying one or more image adjustments to the participant tiles 200 to 206. For example, the image adjustments may include adjusting the exposure, luminance, highlights, shadows, contrast, brightness, black point, saturation, natural saturation, color temperature, hue, sharpness, clarity, noise reduction, and / or vignetting of the participant tiles 200 to 206. Similarly, background content may be removed from the region 102 when generating the group interface 208, such that content other than participant features is omitted from the participant tiles 200 to 206. Through image adjustments and content removal, participant tiles 200 to 206 are further generated with a uniform appearance regardless of any differences in lighting conditions, sensor quality, and / or visibility of different participants 106 to 112 within the region 102.

[0025] The feature optimization action may modify the participant features corresponding to one or more of the participants 106 to 112 when generating the participant tiles 200 to 206. The sensor system 104 may capture depth data corresponding to the participant features. The feature optimization action may be applied based on the depth data. For example, the feature optimization action may include modifying the perspective of one or more of the participants 106 to 112, modifying the orientation of one or more of the participants 106 to 112, removing obstacles that block the participant features of one or more of the participants 106 to 112, and / or increasing the visibility of one or more of the participants 106 to 112.

[0026] When removing obstacles, the blocking of a portion of the participant feature can be detected, and a heuristic action can be used to generate an unobstructed view of the corresponding participant. For example, artificial intelligence and / or machine learning algorithms can be used to determine how the participant feature of the corresponding participant looks under various participant conditions (e.g., orientation, position, emotion, operation, etc.). The identity and participant condition of the blocked corresponding participant can be determined. Using artificial intelligence and / or machine learning algorithms, a rendering of a portion of the participant feature is generated, and the rendering is mixed with the remaining participant features when generating the participant tile to provide an unobstructed view of the corresponding participant in the group interface 208. In doing so, the expression of the corresponding participant can be determined, and the rendering of the portion of the participant feature is matched to the expression of the corresponding participant. Thus, for example, if the blocked portion of the participant feature is a mouth and the corresponding participant is depressed, the rendering of the portion includes a mouth with a matching depressed expression, which is different from an expression associated with a different emotion (e.g., a smile). Any portion of the participant feature with low visibility can be similarly detected and corrected for those participant features using heuristic actions through feature optimization actions. Feature optimization actions are applied to the participant features when generating the participant tiles 202 - 206 as needed to provide a complete, high-quality view of the participants 106 - 112 in the group interface 208 from a unified perspective and / or viewpoint.

[0027] In this way, participants 106-112 can be detected in area 102 with corresponding participant features extracted from image data (e.g., video) of area 102 captured using sensor system 104. The participant features are separated to create separate participant tiles 200-206, each of which corresponds to a particular participant. The participant tiles 200-206 are homogenized and constructed together in group interface 208 for uniform presentation of participants 106-112, regardless of varying participant conditions, area conditions, environmental conditions, and / or other differences among participants 106-112 and / or area 102.

[0028] Refer to the example Figure 2, a tile optimization action and / or a feature optimization action may be applied to the participant tile 200 to normalize the size differences of the first participant 106 relative to the other participants 106 to 112 based on the distance of each of the participants 106 to 112 from the sensor system 104 and / or the location of the participants 108 to 112 within the area 102. Similar actions may be applied when generating the participant tiles 202 to 206 to provide a uniform size of the participants 106 to 112 in the group interface 208 that reflects the true proportions of the participants' faces in the real-world environment. The different perspectives of the sensor system 104 relative to the front of each of the faces of the participants 106 to 112 and the different orientations of the participant features relative to the sensor system 104 may be further normalized by the tile optimization action and / or the feature optimization action when generating the participant tiles 200 to 206. In doing so, heuristic actions and / or other sensor data may be used to create any indiscernible, blocked, and / or low-visibility portions of participant features to generate the participant tiles 200 to 206 and thereby create a group interface 208 with an unobstructed, high-quality view of the participants 106 to 112 from a uniform perspective. For example, the first participant tile 200 is generated such that the first participant 106 is shown with the object 210 removed, and the effects of the different directions faced by the participants 106 to 112 and the different orientations of the participants 106 to 112 on the rendering of the participant features are addressed when generating the participant tiles 200 to 206. Finally, the different lighting conditions of the participants 106 to 112 in the area 102 are normalized to provide uniform image quality and presentation across the group interface 208. In this way, as shown in FIG. Figure 2 As shown, group interface 208 includes the full faces of participants 106-112, with a uniform size, perspective, image quality, and / or presentation that is consistent with the real-world appearance of participants 106-112, regardless of any differences in conditions among participants 106-112 within area 102. Furthermore, in some examples, one or more of participants 106-112 may move within area 102, between area 102 and a second area, and / or the conditions of participants 106-112 and / or area 102 may continue to change, such that differences due to the changing conditions are addressed in real time through dynamic optimization actions to maintain group interface 208 with a uniform size, perspective, image quality, and / or presentation of participants 106-112.

[0029] The group interface 208 may be further optimized for rendering in various ways. As previously described, background content from the region 102 may be removed from the participant tiles 200 to 206. Alternatively, sensor data corresponding to the background content may be discarded so that the participant tiles 200 to 206 without the background content are generated. The background content may refer to any portion of the region 102 and / or the participant other than the participant features. The participant features may correspond to the head of each participant in the participants 106 to 112, the body of each participant in the participants 106 to 112, and / or a combination or portion thereof. The group interface 208 may be generated and rendered so that the boundaries between the participant tiles 200 to 206 are visible or invisible. For example, a frame may be shown around each participant in the participants 106 to 112 in the group interface 208 based on the participant tiles 200 to 206, or the participants 106 to 112 may be shown adjacent to each other in the group interface 208 without a frame or similar boundary. The participants 106-112 may be arranged on the group interface 208 based on the location of the participants within the area 102, the level of participation in the communication session, and / or otherwise according to user preferences. The participants 106-112 may be rendered in two or three dimensions in the group interface 208, where the group interface 208 is presented using an interface system associated with the area 102 and / or the structure 114, a plurality of electronic devices such as user devices (e.g., wearable devices, smart phones, etc.) associated with one or more of the participants 106-112, and / or a combination thereof.

[0030] The participant tiles 200 to 206 may be arranged in the group interface 208 to indicate to the second participant (e.g., outside the area 102) how each of the participants 106 to 112 is positioned within the area 102. For example, as discussed herein, in some examples, the area 102 is located within the interior of a structure 114, which may be a mobile device (e.g., a vehicle). The first and second participants 106 to 108 may be located in the back seat of the mobile device, and the third and fourth participants 110 to 112 may be located in the front seat of the mobile device. The fourth participant 112 may be located in the driver's seat. In some instances, the sensor system 104 may have a camera positioned at the front of the mobile device so that the third and fourth participants 110 to 112 are closer to the camera than the first and second participants 106 to 108. For example, the camera may be fixed at the top center of the front windshield of the mobile device (e.g., near the rearview mirror). In this way, the camera will capture the first and second participants 106 to 108 as having a smaller size relative to the third and fourth participants 110 to 112. As described herein, these differences are homogenized when generating the group interface 208 to provide a normalized presentation of the participants 106-112. In addition, the participants 106-112 do not have to be close together to appear together in the video, but rather, each of the participants 106-112 can be comfortably seated in a separate seat, with participant features of each of the participants 106-112 extracted from the video to form separate participant tiles 200-206 for homogenization and combination into the group interface 208. The participant tiles 200-206 can be arranged in the group interface 208 to indicate to a second participant (e.g., a participant outside the mobile device) how each of the participants 106-112 is seated within the mobile device. For example, the group interface 208 can have the participants 106-112 displayed in an arrangement representing driver-side and passenger-side occupants. Optimization actions can be appropriately applied relative to participants 106-112 so that if one of participants 106-112 turns his head to look at one of the other participants 106-112 or otherwise moves away from the camera, the perspective of participants 106-112 remains unchanged and the participant's head turn is not reflected in the group interface 208 during the communication session.

[0031] In addition, in conjunction with or separate from the communication session, participant characteristics of one or more of the participants 106 to 112 may be extracted and analyzed in conjunction with the attitudes of the participants 106 to 112 toward the operation or characteristics of the mobile device. For example, the participant characteristics may be compared with participant characteristics that represent the participant's readiness for a particular operation mode of the mobile device, a particular navigation action of the mobile device, a particular operation action of the mobile device, and / or autonomous operation. The operation mode of the mobile device may include an autonomous operation mode and a manual operation mode. In some examples, the participant characteristics may be used to detect an operation mode transition trigger of the mobile device, wherein the mobile device transitions between the autonomous operation mode and the manual operation mode based on the operation mode transition trigger. The operation mode transition trigger may correspond to the participant's readiness for the transition (e.g., the driver is alert to take manual control of the mobile device).

[0032] Steering Figure 3 , shows an exemplary operation 300 for group communication. Operation 302 detects multiple participants of a communication session. The multiple participants may be co-located in an area and / or located in different areas. In some examples, the area is the interior of a vehicle.

[0033] Operation 304 obtains participant characteristics for each of the plurality of participants. The participant characteristics may be captured using at least one image sensor. The at least one image sensor may be configured to capture image data of an area, such as a video. In some examples, operation 304 extracts participant characteristics for each of the plurality of participants from the image data of the area. The at least one image sensor may include a plurality of image sensors having overlapping fields of view of the area. The at least one image sensor and the display may be associated with a vehicle (e.g., positioned within an interior of the vehicle).

[0034] Operation 306 generates a plurality of participant tiles. Each participant tile in the participant tiles corresponds to a participant feature of a corresponding one of the plurality of participants. The plurality of participant tiles may be homogenized to account for the effects of differences in conditions among the participants and / or the regions in which the participants are located. For example, homogenizing the participant tiles may include applying a tile optimization action to at least one participant tile in the plurality of participant tiles, applying a feature optimization action to a participant feature of at least one participant tile in the plurality of participant tiles, and the like. The tile optimization action may modify the at least one participant tile in two dimensions, and / or the feature optimization action may modify the participant feature in three dimensions. For example, the feature optimization action may modify the perspective of one or more of the plurality of participants.

[0035] In addition, optimization actions (such as tile optimization actions applied to one or more participant tiles in the participant tiles and / or feature optimization actions applied to participant features corresponding to one or more participant tiles in the participant tiles) can address the effects of blocking or reduced visibility of a portion of a participant feature of a particular participant. For example, blocking of a portion of a participant feature of a particular participant in the plurality of participants can be detected, and heuristic actions can be used to generate an unobstructed view of the particular participant. The unobstructed view includes a rendering of a portion of the participant feature. In doing so, an expression of the particular participant can be determined, and a rendering of a portion of the participant feature corresponding to the particular participant can be rendered using a rendering generated based on the expression. In this way, the rendering of the portion of the participant feature can match the expression of the particular participant.

[0036] Operation 308 generates a group interface for the communication session based on the plurality of tiles. The group interface includes participant features for each of the plurality of participants. In some examples, the group interface includes participant features for each of the plurality of participants arranged according to corresponding positions of each of the plurality of participants within an area (e.g., an interior of a vehicle). In addition, second participant features for each of the one or more second participants located in a second area may be obtained, the second area being outside one or more fields of view of the at least one image sensor. One or more second participant tiles may be generated, and a group interface for the communication session may be further generated based on the one or more second participant tiles.

[0037] Operation 310 renders a group interface for display during the communication session. The group interface may be displayed using at least one display. In some examples, the group interface removes and / or omits background content from an area. Additionally, the at least one image sensor may include a depth sensor, and the image data includes depth data for participant features. In this example, the participant features may be rendered in three dimensions in the group interface based on the depth data. In conjunction with or independent of the communication session, the participant features may be used to detect an operating mode transition trigger for the vehicle. The vehicle may transition between an autonomous operating mode and a manual operating mode based on the operating mode transition trigger.

[0038] Steering Figure 4, an exemplary mobile device 400 is shown, which can be structure 114. In some specific implementations, mobile device 400 includes one or more sensor systems 402 and device systems 404. It should be understood that any of perception system 406, lidar sensor 408, camera 410, positioning system 412, other sensors 414, planning system 416, control system 418, subsystem 420, interface system 422, and / or communication system 424 can be part of device system 406, separate from the device system, or not included in the device system. In addition, interface system 422 and communication system 424 can be separate from subsystem 420 and / or be part of the subsystem.

[0039] The sensor system 402 includes one or more sensors configured to capture sensor data, including, but not limited to: data (e.g., one or more images) of a field of view inside (e.g., area 102) and / or outside the mobile device 400; positioning data corresponding to the position, heading, and / or orientation of the mobile device 400 and / or a participant; movement data corresponding to the movement of the mobile device 400, area 102, and / or a participant (e.g., along the movement path 116); mobile device information, participant information (based on user preferences and consent), and / or data corresponding to a communication session. The one or more sensors of the sensor system 402 may include, but are not limited to, a 3D sensor configured to capture a 3D image, a 4D sensor configured to capture a 4D image, a radar sensor, an infrared (IR) sensor, an optical sensor, and / or a visual detection and ranging (ViDAR) sensor. For example, one or more 3D sensors may include a lidar sensor 408 (e.g., a scanning lidar sensor) or other depth sensors, and one or more 4D sensors may include a camera 410 (e.g., an RGB camera). The camera 410 may capture color images, grayscale images, and / or other 4D images. Positioning system 412 may capture positioning data. Positioning systems may include, but are not limited to, GNSS, inertial navigation systems (INS), inertial measurement units (IMU), global positioning systems (GPS), attitude and heading reference systems (AHRS), compasses, and / or accelerometers. Other sensors 414 may be used to capture positioning data, movement data, participant data, and / or other authorized and relevant sensor data.

[0040] The perception system 404 may generate perception data that may use sensor data to detect, identify, classify, and / or determine one or more objects, such as the location of one or more participants. The perception data may be used by the planning system 416 to determine one or more actions, such as generating a navigation plan with at least one movement action for autonomously navigating the mobile device 400 from a starting point toward a destination along the movement path 116 and / or adjusting the operation of one or more of the device systems 406. The control system 418 may be used to control various operations of the mobile device 400, including but not limited to detecting participants, identifying participants, generating and homogenizing participant tiles, generating group interfaces, facilitating communication sessions, performing navigation planning, and / or other operations. The navigation plan may include various operating instructions for the subsystem 420 of the mobile device 400 to autonomously perform navigation actions and other actions based on the content, so that the mobile device 400 moves according to its own planning and decision-making. Instructions for operating the mobile device 400 in view of the movement path 116 may be executed by the planning system 416, the control system 418, the subsystem 420, and / or other components of the mobile device 400. Instructions may be modified (eg, using interface system 422 ) prior to execution by mobile device 400 , and in some cases mobile device 400 may ignore instructions, for example, based on sensor data captured by sensor system 402 .

[0041] In some specific implementations, the interface system 422 includes a presentation system and an input system. The interface system 422 may form a part of a computing device such as a smart phone, a mobile device (e.g., mobile device 400), a robot, a wearable device, a home system, etc. The input system of the interface system 422 may include one or more input devices configured to capture various forms of user input. For example, the interface system 422 may be configured to capture content from various sources and / or user input from a user. Similarly, the presentation system of the interface system 422 may include one or more output devices configured to present various forms of content (including visual, audio, and / or tactile content) in two and / or three dimensions. The interface system 422 may include various software and / or hardware for input and presentation. The input system and the presentation system may be integrated into one system in whole or in part, or may be separate. For example, the input system and the presentation system of the interface system 422 may be provided in the form of a touch screen.

[0042] In some implementations, the interface system 422 provides an interactive interface. The interactive interface may be deployed in the mobile device 400. For example, the interactive interface may be deployed in the interior of the mobile device 400. The communication system 424 may include, but is not limited to, one or more antennas, receivers, transponders, transceivers, and / or communication ports for providing a communication session. In some cases, the communication system 424 is configured to communicate via different types of wireless networks associated with the mobile group communication session and communicating with other devices. In some examples, the communication system 424 is configured for long-range communication (e.g., via a cellular network, a satellite network, radio, etc.), short-range communication (e.g., Bluetooth, Wi-Fi, UWB, etc.), and / or otherwise communicate with other devices and data sources associated with the communication session.

[0043] See also Figure 5 , which provides a detailed description of an example computing device 500 having one or more computing units that can implement the various systems and methods discussed herein. The various components of the computing device 500 can be formed into specific, unconventional, and non-universal arrangements to implement the various technical solutions discussed herein. Thus, the computing device 500 and / or components of the computing device 500 can be applicable to the structure 114, the various systems and subsystems of the mobile device 400, the various communication and interface systems described herein, and other computing or network devices. In some examples, the structure 114 and / or the mobile device 400 is a vehicle, but it should be understood that these devices can be various types of devices, such as robots, user devices, electronic devices, household devices, etc. It should be understood that specific implementations of these devices may have different possible specific computing architectures, not all of which are specifically discussed herein, but will be understood by those of ordinary skill in the art.

[0044] The computing device 500 may be a computing system capable of executing a computer program product to perform a computer process. Data and program files may be input into the computing device 500, which reads the files and executes the programs therein. Figure 5 500, including one or more hardware processors 502, one or more data storage devices 504, one or more memory devices 506, and / or one or more ports 508 to 512. In addition, other elements that those skilled in the art will recognize may be included in the computing device 500, but are not limited to the following. Figure 5 The various elements of computing device 500 may be connected via one or more communication buses, point-to-point communication paths, or Figure 5 The devices communicate with each other by other communication means not explicitly shown.

[0045] Processor 502 may include, for example, a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processor (DSP), and / or one or more internal levels of cache. There may be one or more processors 502, such that processor 502 includes a single central processing unit or multiple processing units capable of executing instructions and performing operations in parallel with each other (commonly referred to as a parallel processing environment).

[0046] Computing device 500 may be a conventional computer, a distributed computer, or any other type of computer, such as one or more external computers available via a cloud computing architecture. The techniques described herein are optionally implemented in software stored on data storage device 504, stored on memory device 506, and / or communicated via one or more of ports 508 to 512, thereby communicating Figure 5 The computing device 500 in the embodiment is converted into a special-purpose machine for implementing the operations described herein. Examples of the computing device 500 include personal computers, servers, special-purpose autonomous processors, terminals, workstations, mobile phones, tablet computers, laptop computers, and the like.

[0047] The one or more data storage devices 504 may include any non-volatile data storage device capable of storing data generated or employed within the computing device 500, such as computer executable instructions for executing computer processes, which may include instructions for both application programs and operating systems (OS) that manage various components of the computing device 500. The data storage device 504 may include, but is not limited to, a magnetic disk drive, an optical disk drive, a solid-state drive (SSD), a flash drive, and the like. The data storage device 504 may include removable data storage media, non-removable data storage media, and / or external storage devices available via a wired or wireless network architecture using such computer program products, including one or more database management products, network server products, application server products, and / or other additional software components. Examples of removable data storage media include compact disk read-only memory (CD-ROM), digital versatile disk read-only memory (DVD-ROM), magneto-optical disk, flash drive, and the like. Examples of non-removable data storage media include internal magnetic hard disks, SSDs, and the like. One or more memory devices 506 may include volatile memory (eg, dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and / or nonvolatile memory (eg, read only memory (ROM), flash memory, etc.).

[0048] A computer program product containing mechanisms for implementing systems and methods according to the techniques described herein may reside in data storage devices 504 and / or memory devices 506, which may be referred to as machine-readable media. It should be understood that a machine-readable medium may include any tangible, non-transitory medium capable of storing or encoding instructions to perform any one or more of the operations of the present disclosure for execution by a machine, or capable of storing or encoding data structures and / or modules used by or associated with such instructions. A machine-readable medium may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more executable instructions or data structures.

[0049] In some implementations, computing device 500 includes one or more ports for communicating with other computing devices, network devices, or vehicle devices, such as input / output (I / O) port 508, communication port 510, and subsystem port 512. It should be understood that ports 508-512 may be combined or separated, and more or fewer ports may be included in computing device 500.

[0050] I / O port 508 may be connected to I / O devices or other devices through which information is input to or output from computing device 500. Such I / O devices may include, but are not limited to, one or more input devices, output devices, and / or environmental transducer devices.

[0051] In one specific implementation, the input device converts human-generated signals such as human voice, physical movement, physical touch or pressure into electrical signals as input data into the computing device 500 via the I / O port 508. Similarly, the output device can convert electrical signals received from the computing device 500 via the I / O port 508 into signals that can be sensed by a person as output, such as sound, light and / or touch. The input device can be a mixed alphanumeric input device, including alphanumeric keys and other keys for transmitting information and / or command selections to the processor 502 via the I / O port 508. The input device can be another type of user input device, including but not limited to: direction and selection control devices, such as mice, trackballs, cursor direction keys, joysticks and / or wheels; one or more sensors, such as cameras, microphones, position sensors, orientation sensors, gravity sensors, inertial sensors and / or accelerometers; and / or touch-sensitive display screens ("touch screens"). Output devices can include but are not limited to displays, touch screens, speakers, tactile and / or tactile output devices, etc. In some specific implementations, for example, in the case of touch screens, the input device and the output device can be the same device.

[0052] The environmental transducer device converts one form of energy or signal into another form of energy or signal for input to or output from the computing device 500 via the I / O port 508. For example, an electrical signal generated within the computing device 500 can be converted into another type of signal, and vice versa. In one specific implementation, the environmental transducer device senses a characteristic or aspect of the environment local or remote to the computing device 500. In addition, the environmental transducer device can generate a signal to exert some effect on the environment local or remote to the example computing device 500.

[0053] In a specific implementation, the communication port 510 is connected to the network, and the computing device 500 can receive network data through the port, which can be used to perform the methods and systems described herein and transmit the information and network configuration changes determined thereby. In other words, the communication port 510 connects the computing device 500 to one or more communication interface devices, which are configured to transmit and / or receive information between the computing device 500 and other devices through one or more wired or wireless communication networks or connections. Examples of such networks or connections include, but are not limited to, universal serial bus (USB), Ethernet, Wi-Fi, Bluetooth, near field communication (NFC), cellular, etc. One or more such communication interface devices can be used to communicate with one or more other machines directly through a point-to-point communication path, through a wide area network (WAN) (e.g., the Internet), through a local area network (LAN), through a cellular (e.g., a third generation (3G), fourth generation (4G) network, or fifth generation (5G)) network, or through another communication means via the communication port 510. In addition, the communication port 510 can communicate with an antenna for electromagnetic signal transmission and / or reception. In some examples, an antenna may be employed to receive Global Positioning System (GPS) data to facilitate determining the location of the device.

[0054] The mobile devices discussed herein may include vehicles. The computing device 500 may include a subsystem port 512 for communicating with one or more systems to control the operation of the vehicle and / or to exchange information between the computing device 500 and one or more subsystems of the vehicle. Examples of such subsystems include, but are not limited to, imaging systems, radars, lidars, motor controllers and systems, battery controls, fuel cells or other energy storage systems or controls, processors and controllers, steering systems, braking systems, lighting systems, navigation systems, environmental controls, entertainment systems, and the like for such vehicles having hybrid or electric motor systems.

[0055] The present disclosure recognizes that participation in communication sessions and operations based on participant feature extraction can be used to benefit users. Entities implementing the technology of the present invention should comply with established privacy policies and / or practices that meet or exceed industry or government requirements for maintaining the privacy and security of transmitted data. The present disclosure expects that any device participating in communication sessions and participant feature extraction will provide an input interface for specifying when and where what type of communication, extraction and operation will occur, thereby allowing users to customize their expected functions. Devices participating in these services can also provide indications that a communication session is being requested and / or that the communication session is active. In addition, users should be allowed to opt in or opt out of allowing devices to participate in such services, including by turning off cameras and muting microphones. In addition, specific information being transmitted (such as messages and recipients) can be encrypted, structured and / or encoded to further maintain privacy and security. Third parties can evaluate these implementers to confirm that they comply with established privacy policies and practices.

[0056] Figure 5 The system shown is only one possible example of a computer system that can employ or be configured according to aspects of the present disclosure. It should be understood that other non-transitory tangible computer-readable storage media that store computer-executable instructions for implementing the disclosed technology on a computing system can be used.

[0057] In the present disclosure, the disclosed method may be implemented as an instruction set or software readable by a device. In addition, it should be understood that the specific order or hierarchical structure of the steps in the method disclosed in the present invention is an example of an example method. Based on design preferences, it should be understood that the specific order or hierarchical structure of the steps in the method can be rearranged while being retained within the subject matter disclosed in the present invention. The requirements advocated in the attached method are to present the elements of various steps in a sample order, and do not necessarily mean to be limited to the specific order or hierarchical structure presented. The described disclosure may be provided as a computer program product or software that may include a non-transitory machine-readable medium having instructions stored thereon, which can be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. Machine-readable media include any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., computer).

[0058] Although the present disclosure has been described with reference to various specific implementations, it should be understood that these specific implementations are illustrative, and the scope of the present disclosure is not limited to these specific implementations. Many variations, modifications, additions and improvements are possible. More generally, embodiments according to the present disclosure have been described in the context of specific specific implementations. Functions can be separated or combined together in a frame in different ways in various embodiments of the present disclosure, or described in different terms. These and other variations, modifications, additions and improvements may fall within the scope of the present disclosure as defined in the following claims.

Claims

1. A method, comprising: detecting a plurality of participants of a communication session, the plurality of participants being co-located in an area; obtaining a participant characteristic of each of the plurality of participants, the participant characteristic being captured using at least one image sensor; generating a plurality of participant tiles, each of the participant tiles corresponding to the participant characteristic of a corresponding one of the plurality of participants; generating a group interface for the communication session based on the plurality of tiles, the group interface including the participant characteristics of each of the plurality of participants; as well as The group interface is rendered for display using at least one display. The method of claim 1 , wherein the group interface removes background content from the area. 3 . The method of claim 1 , wherein the at least one image sensor comprises a plurality of image sensors having overlapping fields of view of the area.

4. The method according to claim 1, further comprising: obtaining a second participant characteristic for each second participant of one or more second participants located in a second area, the second area being outside one or more fields of view of the at least one image sensor; as well as One or more second participant tiles are generated, and the group interface for the communication session is further generated based on the one or more second participant tiles. The method of claim 1 , wherein the region is an interior of a structure.

6. The method according to claim 1, further comprising: using the participant characteristics to detect an operating mode transition trigger; as well as The electronic device is transitioned between a first operation mode and a second operation mode based on the operation mode transition trigger. 7 . The method of claim 5 , wherein the group interface includes the participant characteristics of each of the plurality of participants arranged according to a corresponding position of each of the plurality of participants within the interior of the structure.

8. The method according to claim 1, further comprising: The plurality of participant tiles are homogenized.

9. The method of claim 8, wherein homogenizing the plurality of participant tiles comprises: A tile optimization action is applied to at least one participant tile of the plurality of participant tiles, the tile optimization action modifying the at least one participant tile in two dimensions.

10. The method of claim 8, wherein homogenizing the plurality of participant tiles comprises: A feature optimization action is applied to the participant feature of at least one participant tile of the plurality of participant tiles, the feature optimization action modifying the participant feature in three dimensions.

11. The method of claim 10, wherein modifying the participant characteristics in three dimensions comprises: The perspective of one or more of the plurality of participants is modified.

12. The method according to claim 1, further comprising: detecting an obstruction of a portion of the participant feature of a particular participant of the plurality of participants; as well as An unobstructed view of the particular participant is generated using a heuristic action, the unobstructed view comprising a rendering of the portion of the participant's features.

13. The method according to claim 12, further comprising: determining an expression of the particular participant; as well as The rendering of the portion of the participant feature is matched to the expression of the particular participant.

14. One or more tangible, non-transitory computer-readable storage media storing computer-executable instructions for executing a computer process on a computing system, the computer process comprising: detecting a plurality of participants of a communication session, the plurality of participants being co-located in an area; extracting a participant feature of each of the plurality of participants from the image data of the area; generating a plurality of participant tiles, each of the participant tiles corresponding to the participant characteristic of a corresponding one of the plurality of participants; generating a group interface for the communication session based on the plurality of tiles, the group interface including the participant characteristics of each of the plurality of participants; as well as The group interface is output for display during the communication session.

15. The one or more tangible, non-transitory computer-readable storage media of claim 14, wherein the image data comprises a video.

16. The one or more tangible, non-transitory computer-readable storage media of claim 14, wherein the group interface omits background content from the area.

17. The one or more tangible, non-transitory computer-readable storage media of claim 14, further comprising: determining an expression of a particular participant among the plurality of participants; as well as A rendering of a portion of the participant feature corresponding to the particular participant is generated, the rendering being generated based on the expression.

18. A system, comprising: at least one image sensor configured to capture image data of a plurality of participants of a communication session co-located in an area; at least one processor configured to generate a group interface for the communication session based on a plurality of participant tiles, each of the participant tiles being generated based on the image data and corresponding to participant features of a corresponding one of the plurality of participants, the group interface including the participant features of each of the plurality of participants; as well as A display is provided to display the group interface during the communication session.

19. The system of claim 18, wherein the at least one image sensor comprises a depth sensor and the image data comprises depth data of the participant features, the participant features being rendered in three dimensions in the group interface based on the depth data.

20. The system of claim 18, wherein the at least one processor is configured to homogenize the plurality of participant tiles.

Images