Communication system

Abstract

To enable smooth communication between participants in a conference using a virtual reality space.SOLUTION: A VR conference system includes a projection unit and a display unit capable of communicating with the projection unit. The projection unit has VR goggles that display a three-dimensional image 21 edited in response to user operations in a virtual reality space. The display unit has a two-dimensional display that displays a field of view by the VR goggles and a user operation unit that accepts the specification of a position on the screen of the two-dimensional display. The VR goggles display a dart pointer 201d at a position in the virtual reality space that corresponds to the position.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a communication system.

Background Art

[0002] For example, the collaborative virtual reality online conference platform of Patent Document 1 shows a technology for replacing an on-site conference with a conference in a common virtual space within VR (virtual reality). This platform includes 3D (three-dimensional) point cloud data defining a virtual space, identifiers of a plurality of conference participants, and conference data including the positions of a plurality of avatars corresponding to the conference participants within the virtual space. It also includes a processor that executes an instruction to start an online conference for a plurality of conference participants. Further, the step of starting an online conference includes a step of providing the address of the 3D point cloud to each conference participant and a step of transmitting the 3D point cloud data and the conference data to each conference participant. The current location of each avatar within the virtual space is transmitted to all conference participants.

[0003] Note that in the specification of Patent Document 1, an "online conference platform" means a conference in which participants in different geographical locations talk to each other via a voice line while looking at the same computer screen. A plurality of parties can be located in their respective offices and meet at a remote location (represented by scan data) within VR.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, when companies develop various products, it is sometimes preferable for multiple personnel from different locations within the same company or its affiliates to gather in the same place simultaneously to conduct discussions about a particular product.

[0006] For example, when developing a new wire harness product for installation in a vehicle, or when modifying the specifications of an existing product, the vehicle manufacturer currently determines the wiring route and shape of the wire harness and creates the design drawings. In this process, the vehicle manufacturer needs to consider factors such as the ease of installation of the wire harness into the vehicle.

[0007] Furthermore, parts manufacturers, based on the design drawings sent from the vehicle manufacturer, consider factors such as the cost of the wire harness components and manufacturing costs, as well as the ease of manufacturing the wire harness, and then create design drawings for the parts manufacturer.

[0008] As described above, when multiple independent locations conduct their own independent studies, if a parts manufacturer discovers a problem with the wire harness design, they communicate this problem to the vehicle manufacturer, who then creates a revised design drawing to correct the issue. Upon receiving the revised design drawing from the vehicle manufacturer, the parts manufacturer creates its own design drawing and prototypes the wire harness again. Consequently, repeated specification changes and other adjustments can result in significant time loss. In particular, when representatives from each company conduct their studies while looking at two-dimensional drawings, it is difficult to grasp the actual three-dimensional shape of each part of the wire harness, making it difficult to identify problematic areas.

[0009] Therefore, it is conceivable that representatives from multiple locations, such as vehicle manufacturers and parts manufacturers, could gather in the same place simultaneously and conduct discussions while viewing actual wire harness models and jigs, for example, placed on a single work table. This could potentially reduce the number of times changes to wire harness manufacturing specifications are made, thereby improving development efficiency.

[0010] For example, if a VR system like the one in Patent Document 1 is adopted, it may be possible for personnel from various locations to simultaneously perceive the three-dimensional shape of the same product on different computer screens without having to travel and gather in one place. However, if, during a meeting, one person wants to communicate to other participants, for example, a part they want to examine in detail or a part they want to modify, it may be difficult to identify that part and communicate it verbally, which can hinder smooth communication.

[0011] This invention has been made in view of the circumstances described above, and its purpose is to enable smooth communication among participants in meetings using virtual reality space. [Means for solving the problem]

[0012] The above objective according to the present invention is achieved by the following configuration. A communication system comprising a first electronic device and a second electronic device capable of communicating with the first electronic device, The first electronic device has a first display unit that displays an object edited in response to user operation within a virtual reality space, The second electronic device is, A second display unit that displays the field of view of the first display unit of the first electronic device, The system includes a position specification unit that accepts the specification of the position of the second display unit on the screen, The first display unit displays a pointer at the position in the virtual reality space corresponding to the position, The second electronic device calculates the position information of the position received by the position designation unit in the second display unit and transmits it to the first electronic device. The first electronic device has a position based on the position information in the virtual reality space as the goal position, and the position of the first electronic device in the virtual reality space as the start position. The first electronic device displays a dart-shaped pointer in the virtual reality space, with its tip positioned at the goal position, along a vector extending from the start position to the goal position. death , The second electronic device updates the image displayed on the second display unit in accordance with the change in the field of view of the first display unit, and the display size of the image is fixed. Communication system.

Advantages of the Invention

[0013] According to the present invention, smooth communication between conference participants becomes possible.

[0014] The present invention has been briefly described above. Furthermore, the details of the present invention will be further clarified by reading through the embodiments (hereinafter referred to as "embodiments") for implementing the invention described below with reference to the attached drawings.

Brief Description of the Drawings

[0015] [Figure 1] FIG. 1 is a schematic diagram showing the situation of an online conference using the virtual reality conference system of the present invention. [Figure 2] FIG. 2 is a block diagram showing a configuration example of the VR conference system. [Figure 3] FIG. 3 is a block diagram showing a configuration example of the conference management server and the projection unit. [Figure 4] FIG. 4 is an explanatory diagram of the dart pointer display process. [Figure 5] FIG. 5 is an explanatory diagram of the dart pointer display process. [Figure 6] FIG. 6 is an explanatory diagram of the dart pointer display process.

Embodiments for Implementing the Invention

[0016] Specific embodiments of the present invention will be described below with reference to each figure.

[0017] <Situation of Online Conference> FIG. 1 is a schematic diagram showing the situation of an online conference using the virtual reality conference system of the present invention. The virtual reality conference system is an example of a communication system.

[0018] For example, when developing a wire harness product for use in automobiles, the vehicle manufacturer first creates design drawings, and component manufacturers such as wire harness manufacturers then adapt these drawings to create designs suitable for manufacturing wire harnesses. However, if a problem occurs when routing the wire harness manufactured based on the design drawings created by the component manufacturer into the vehicle, this information is transmitted to the vehicle manufacturer, who then redraws the design drawings, and the component manufacturer then creates its own design drawings based on those revised drawings. In this way, every time a problem occurs, the design has to be redone at each location, reducing the efficiency of product development.

[0019] Therefore, in order to improve the efficiency of product development, for example, personnel from design centers, domestic manufacturing bases, overseas manufacturing bases within a corporate group that provides wire harness products, as well as the design centers of automobile companies that manufacture vehicles equipped with the manufactured wire harnesses, will participate in the same meeting and discuss the matter simultaneously.

[0020] When developing a wire harness product, it is necessary to appropriately determine the three-dimensional shape of each part of the wire harness so that it follows an appropriate routing path in accordance with the structure of the vehicle body and the arrangement of various electrical components. Furthermore, it is necessary to appropriately determine the layout of the jigs used in manufacturing the wire harness in accordance with the three-dimensional shape of each part of the wire harness. In addition, it is necessary to appropriately determine the manufacturing procedures and other aspects to improve the work efficiency of the wire harness manufacturing process. Moreover, it is necessary to appropriately determine the branching points and wire lengths of the wire harness in order to reduce the cost of the individual wires and other components that make up the wire harness.

[0021] Therefore, each person in charge at each location needs to understand the three-dimensional shape of each part of the wire harness they are designing and conduct their own considerations from their own unique perspective. For this reason, it is common practice to prepare a physical model and have all the people in charge at all locations travel to the same meeting place and conduct their discussions while looking at the same physical model. This can improve the efficiency of the meeting. However, since each location is often far away, such as overseas, the time and cost of travel to hold meetings becomes a significant burden.

[0022] When using the VR conferencing system of this embodiment, for example, by using the VR meeting room 20 shown in Figure 1, personnel at each location can hold online meetings without having to travel.

[0023] The example shown in Figure 1 assumes a scenario where multiple personnel from four independent locations H1, H2, H3, and H4 simultaneously gather in a common VR discussion room 20 to hold a discussion meeting. Locations H1, H2, H3, and H4 correspond to design bases, domestic manufacturing bases, overseas manufacturing bases, and design bases of automobile companies that manufacture vehicles equipped with the wire harnesses, respectively, within a corporate group that provides wire harness products.

[0024] Within base H1, there is a specific location where a real venue V1 exists for the study group. Similarly, within bases H2, H3, and H4, there are specific locations where real venues V2, V3, and V4 exist, respectively, for the study group.

[0025] In the example shown in Figure 1, there is one or more reviewers P11 and one or more other participants P12 in venue V1. Similarly, there is one or more reviewers P21 and one or more other participants P22 in venue V2. Also, there is one or more reviewers P31 and one or more other participants P32 in venue V3. And there is one or more reviewers P41 and one or more other participants P42 in venue V4.

[0026] Furthermore, the venue V1 is equipped with system equipment 10A necessary for online meetings, including one or more display units 11, one or more projection units 12, and one or more sets of position sensors 13 and 14. The projection unit 12 is an example of a first electronic device, and the display unit 11 is an example of a second electronic device.

[0027] Furthermore, venue V2 is equipped with system equipment 10B, which includes one or more display units 11, one or more projection units 12, and one or more sets of position sensors 13 and 14. Venue V3 is equipped with system equipment 10C, which includes one or more display units 11, one or more projection units 12, and one or more sets of position sensors 13 and 14. Venue V4 is equipped with system equipment 10C, which includes one or more display units 11, one or more projection units 12, and one or more sets of position sensors 13 and 14.

[0028] Participant P11 in venue V1, while wearing the projection unit 12 included in the VR conferencing system, will be able to move around as avatar A11 in the virtual reality space of VR discussion room 20 and change the posture of avatar A11. Furthermore, while wearing the projection unit 12, participant P11 will be able to view the stereoscopic image 21 in the field of view at avatar A11's position as an image that actually appears three-dimensional. The same applies to participants P21 to P41 in the other venues V2 to V4. The stereoscopic image 21 is an example of an object.

[0029] In practice, by using projection displays such as the VR goggles 15 described later, it becomes possible to display a three-dimensional image that can be perceived in three dimensions by creating parallax between the image projected to the left eye and the image projected to the right eye of the examiner P11.

[0030] The VR study room 20 is a three-dimensional space virtually created by computer processing, and is formed as a box-shaped space similar to, for example, the interior of a typical conference room. In the example shown in Figure 1, a three-dimensional image 21 generated based on the design data of a wire harness to be developed as a product virtually exists in the space of the VR study room 20. More specifically, the three-dimensional image 21 includes a three-dimensional model of the wire harness and three-dimensional models of numerous jigs placed on a workbench used when manufacturing the wire harness.

[0031] Furthermore, a 3D model of Avatar A11, which corresponds to examiner P11 in venue V1 (a character such as a doll), is placed within the VR discussion room 20. In addition, 3D models of Avatars A21 to A41, which correspond to examiners P21 to P41 in the other venues V2 to V4, are also placed within the VR discussion room 20.

[0032] When participant P11 moves in the real space within venue V1, position sensors 13 and 14 installed in venue V1 detect this change in position. The actual three-dimensional position change of participant P11 detected by position sensors 13 and 14 is reflected in the three-dimensional position change of avatar A11 in the virtual space of VR discussion room 20. In addition, the actual posture change of participant P11 is detected by the projection unit 12 worn by participant P11, and the result is reflected in the posture change of avatar A11 in VR discussion room 20. The same applies to participants P21 to P41 and avatars A21 to A41 in the other venues V2 to V4.

[0033] On the other hand, the display unit 11 located within venue V1 is a computer equipped with a two-dimensional display, such as a laptop computer (PC), and is connected to the projection unit 12 within venue V1 so as to work in conjunction with it. Specifically, an image of the VR study area 20, covering approximately the same range as the image seen in the field of view of the examiner P11 wearing the projection unit 12 within venue V1, is displayed on the screen of the display unit 11 in synchronization with the position and orientation of the projection unit 12. Of course, since the screen of the display unit 11 is a two-dimensional display, the three-dimensional image 21 within the VR study area 20 is coordinate-transformed and displayed as a two-dimensional image on the two-dimensional screen of the display unit 11.

[0034] Similarly, the display unit 11 in venue V2 can display an image of the VR study area 20, covering approximately the same range as the image seen in the field of view of examiner P21 wearing the projection unit 12 in venue V2, on the screen of the display unit 11 in synchronization with the position and orientation of the projection unit 12. The display unit 11 in venue V3 can display an image of the VR study area 20, covering approximately the same range as the image seen in the field of view of examiner P31 wearing the projection unit 12 in venue V3, on the screen of the display unit 11 in synchronization with the position and orientation of the projection unit 12. The display unit 11 in venue V4 can display an image of the VR study area 20, covering approximately the same range as the image seen in the field of view of examiner P41 wearing the projection unit 12 in venue V4, on the screen of the display unit 11 in synchronization with the position and orientation of the projection unit 12.

[0035] The display units 11 in each venue (V1-V4) are, for example, placed on a table. Microphones for collecting sound within the same venue and speakers for amplification are also placed on the same table.

[0036] Participant P11 in venue V1 can move around in the real-world venue V1 while wearing the projection unit 12, and simultaneously move around in the virtual space of the VR discussion room 20. In other words, participant P11's actual movements are reflected in the changes in position and posture that participant P11 is looking at in the VR discussion room 20, and are also reflected in the content of participant P11's field of view projected by the projection unit 12. The same applies to participants P21 to P41 in the other venues V2 to V4.

[0037] In each venue (V1-V4), participants P11-P41 can move around, allowing them to visually grasp in detail the three-dimensional shape and other conditions of the parts of the product or jig they wish to examine within the VR review area (20).

[0038] Furthermore, since each avatar A11-A41, corresponding to the reviewers P11-P41 in each venue V1-V4, exists within the VR review room 20, each reviewer P11-P41 can immediately grasp the position and posture of each reviewer at other locations through the image projected by their own projection unit 12. Therefore, it is easy for multiple reviewers at different locations to simultaneously check and review the same part of the product within the VR review room 20.

[0039] Furthermore, participants P12 to P42, other than the reviewers P11 to P41 within each venue V1 to V4, can also understand the video of the area that reviewers P11 to P41 are examining, based on the content displayed by the display unit 11.

[0040] On the other hand, as will be described later, the projection unit 12 at each location is equipped with a user operation unit that accepts input operations from each reviewer P11 to P41. In addition, the display unit 11 at each location is also equipped with a user operation unit that accepts input operations from each participant P12 to P42.

[0041] Each reviewer P11-P41 can modify, add, or delete data from the 3D image 21 in the VR discussion room 20 by operating the user control panel of the projection unit 12 they are wearing. The modifications made by each reviewer P11-P41 are recorded as data by the VR conferencing system and reflected in the 3D image 21 in real time. Furthermore, the results of any modification made by any of the reviewers P11-P41 are reflected in real time in the projection content of all reviewers P11-P41's projection units 12, and in the display content of all participants P12-P42's display units 11.

[0042] Therefore, all reviewers P11-P41 located at different locations can simultaneously review the same 3D image 21 within nearly the same field of view using the common VR review space 20. Furthermore, all reviewers P11-P41 at different locations can make necessary modifications to the shape, structure, and layout of the product or jig projected as the 3D image 21, and confirm the results of these modifications in real time. Additionally, participants P12-P42, other than reviewers P11-P41, can simultaneously review the 2D image corresponding to the 3D image 21 by referring to the display screen of the display unit 11, viewing it within nearly the same field of view as reviewers P11-P41. This allows all participants in the online meeting to efficiently proceed with their review work. In particular, each participant P12-P42 can specify areas of interest, such as areas they want to review in detail or areas they wish to modify, on the display screen of the display unit 11, and share these specified areas with other reviewers P11-P41 and participants P12-P42. This enables smooth communication among participants.

[0043] <System Configuration> Figure 2 is a block diagram showing an example configuration of the VR conferencing system 100. Figure 3 is a block diagram showing an example configuration of the conferencing management server 30 and the projection unit 12.

[0044] In the example shown in Figure 2, system equipment 10A to 10D located in venues V1 to V4 at different locations H1 to H4 are connected to each other via a communication network 25 so that they can communicate with one another.

[0045] Furthermore, in order to enable simultaneous online meetings between multiple locations H1-H4 using a common VR discussion room 20, a meeting management server 30 is connected to the communication network 25. The communication network 25 is expected to include local networks within each location H1-H4, dedicated communication lines within companies, and public communication lines such as the internet.

[0046] Furthermore, if the conference communication is conducted via the internet, the security of the communication can be ensured by encrypting the data. In addition, the conference management server 30 may be installed at any of the multiple locations H1 to H4, or it may be installed at a data center or other location other than locations H1 to H4.

[0047] The conference management server 30 shown in Figure 3 comprises a communication device 31, a participant management unit 32, database management units 33 and 34, a VR data generation unit 35, an avatar control unit 36, and a change control unit 37.

[0048] The communication device 31 has the function of securely communicating data between system equipment 10A to 10D within each of the locations H1 to H4 via the communication network 25. The participant management unit 32 has the function of managing access for each participant who participates in a common online meeting as a reviewer P11-P41 or participant P12-P42.

[0049] The database management unit 33 holds and manages design data corresponding to wire harnesses under development. This design data includes data representing the shape, dimensions, and various components of each part of the target wire harness, as well as data representing the shape and layout of various jigs used in manufacturing the wire harness.

[0050] The database management unit 34 has the function of holding and managing update data that represents modifications to a specific version of data held by the database management unit 33. For example, data representing a change in the shape of a specific part of a wire harness, data representing the addition of a new part to a specific part of a wire harness, and data representing the deletion of a part to a specific part of a wire harness are sequentially registered and held in the database management unit 34 during an online meeting.

[0051] The VR data generation unit 35 generates data for three-dimensional images 21 to be placed in the three-dimensional virtual space of the VR study area 20. The data for three-dimensional images 21 generated by the VR data generation unit 35 includes three-dimensional images corresponding to the wire harness design data held by the database management unit 33, three-dimensional images corresponding to each avatar managed by the avatar control unit 36, and three-dimensional images corresponding to the update data managed by the database management unit 34.

[0052] The avatar control unit 36 ​​has the function of managing the characters in the VR discussion room 20 corresponding to the participants P11 to P41 at each location H1 to H4 who are participating in the online meeting of the VR conference system 100, as avatars A11 to A41. In addition, the avatar control unit 36 ​​has the function of constantly monitoring the position (three-dimensional coordinates) and posture (direction of gaze) of each participant P11 to P41 to understand their latest status.

[0053] The change control unit 37 has the function of receiving input operations performed by each of the participants P11 to P41 from each of the locations H1 to H4 participating in the online meeting of the VR conference system 100 on the user operation unit of the projection unit 12, and input operations performed by each of the participants P12 to P42 from each of the locations H1 to H4 on the user operation unit of the display unit 11, as modification instructions for the stereoscopic image 21 in the VR discussion room 20. In addition, the change control unit 37 has the function of registering the received modification instructions as update data representing changes, additions, deletions, etc., to the stereoscopic image 21 in the database management unit 34.

[0054] On the other hand, the projection unit 12 shown in Figure 3 includes a communication device 12a, a user position detection unit 12b, a user operation unit 12c, an audio transmission unit 12d, VR goggles 15, and a headset 16. The VR goggles 15 also include the functions of a VR image generation unit 15a, a left-eye display 15b, a right-eye display 15c, and a user posture detection unit 15d. The headset 16 has a built-in microphone and speaker. The VR goggles 15 is an example of a first display unit.

[0055] The communication device 12a connects to the conference management server 30 via the communication network 25 and can send and receive data with the conference management server 30. Specifically, it periodically acquires data of the 3D image 21 in the VR study area 20 from the conference management server 30. The data of the 3D image 21 acquired by the communication device 12a from the conference management server 30 includes design data such as the three-dimensional shape of the wire harness and the jig layout, as well as data on the three-dimensional shape, position, and orientation of each avatar A11 to A41.

[0056] Furthermore, for example, the communication device 12a in the projection unit 12 at base H1 can periodically transmit to the conference management server 30 the three-dimensional position coordinates of the examiner P11 detected by the position sensors 13 and 14 at base H1, and the posture (direction of gaze) of the examiner P11 detected by the VR goggles 15 worn by the examiner P11.

[0057] Furthermore, for example, the communication device 12a in the projection unit 12 of base H1 is connected to the display unit 11 in base H1 and can periodically transmit to the display unit 11 information representing the field of view of the reviewer P11 in the virtual reality space, which is determined based on the three-dimensional position coordinates of the reviewer P11 and the posture of the reviewer P11. In addition, as will be described later, for example, when a position is specified on the display screen of the display unit 11 in base H1, for example by mouse operation, the communication device 12a in the projection unit 12 of base H1 receives position information of the position specified by the click from the display unit 11.

[0058] The user position detection unit 12b can detect the three-dimensional position coordinates of each examiner P11 to P41 in real space and their changes based on the detection state of a pair of position sensors 13 and 14 located in the area facing the examiner P11 to P41 wearing VR goggles 15 within each site H1 to H4.

[0059] The user operation unit 12c is a device that can accept various button operations and coordinate input operations by the user, such as a mouse, which is a common input device. In this embodiment, the user operation unit 12c can accept input operations from each participant P11 to P41 wearing the VR goggles 15 of the projection unit 12. Specifically, the user operation unit 12c can give modification instructions such as moving, changing, adding, or deleting parts of the three-dimensional image 21 of the VR study area 20 projected by the VR goggles 15.

[0060] The voice transmission unit 12d can transmit the voice information of the reviewer, captured by the microphone of the headset 16, to other locations via the communication device 12a and the communication network 25. Furthermore, the voice transmission unit 12d can receive voice information emitted by reviewers at other locations via the communication network 25 and the communication device 12a, and output it as audio from the speaker of the headset 16.

[0061] The VR goggles 15 have the function of projecting stereoscopically recognizable images onto the left and right eyes of the user wearing them.

[0062] The VR video generation unit 15a constantly monitors the position and posture (e.g., direction of gaze) of the user (student P11 to P41) in the three-dimensional virtual space of the VR study room 20, and identifies the user's field of view. It then obtains data from the conference management server 30 that is within the user's field of view from among the data of the stereoscopic image 21 existing in the three-dimensional virtual space of the VR study room 20, and generates two types of two-dimensional image data visible from the viewpoint positions of the user's left and right eyes by transforming the coordinates of the stereoscopic image 21 data.

[0063] The left-eye display 15b receives the two-dimensional image data for the left eye generated by the VR image generation unit 15a from the VR image generation unit 15a and projects it as a two-dimensional image onto the position of the user's left eye.

[0064] The right-eye display 15c receives the two-dimensional image data for the right eye generated by the VR image generation unit 15a from the VR image generation unit 15a and projects it as a two-dimensional image onto the position of the user's right eye.

[0065] The user posture detection unit 15d detects the direction of the user's gaze by tracking the position of the user's pupils, for example, by a camera. Alternatively, the user posture detection unit 15d detects the three-axis angles (roll angle, pitch angle, yaw angle) representing the orientation of the user's head using a three-axis accelerometer or the like.

[0066] The user's posture detected by the user posture detection unit 15d and the position information detected by the user position detection unit 12b are input to the conference management server 30 via the communication device 12a and the communication network 25. The user's position and posture are then reflected in the corresponding positions and postures of each avatar A11 to A41 in the virtual reality space within the VR discussion room 20 through processing by the conference management server 30.

[0067] On the other hand, the display unit 11 shown in Figure 2 includes a communication device 11a, a two-dimensional image generation unit 11b, a two-dimensional display 11c, a user operation unit 11d, and an audio transmission unit 11e. The two-dimensional display 11c is an example of a second display unit, and the user operation unit 11d is an example of a position selection unit.

[0068] The communication device 11a connects to the conference management server 30 via the communication network 25 and can send and receive data with the conference management server 30. Specifically, it periodically acquires data of the 3D image 21 in the VR study area 20 from the conference management server 30.

[0069] Furthermore, the communication device 11a is connected to the projection unit 12 at the same location, and can obtain information from the projection unit 12 necessary to synchronize the field of view of the examiner wearing the projection unit 12 with the display range of the display unit 11.

[0070] The two-dimensional image generation unit 11b identifies the field of view of a study participant wearing the projection unit 12 within the same location based on information transmitted from the projection unit 12, and acquires data of a three-dimensional image 21 existing in the three-dimensional virtual space of the VR study area 20 from the conference management server 30 for a range equivalent to the study participant's field of view. The two-dimensional image generation unit 11b then generates two-dimensional image data of the image visible from the study participant's viewpoint by transforming the coordinates of the three-dimensional image 21 data.

[0071] The two-dimensional display 11c displays the two-dimensional image data generated by the two-dimensional image generation unit 11b as a two-dimensional image on the screen. Alternatively, the display unit 11 may acquire one of the two types of left and right two-dimensional image data generated by the VR image generation unit 15a of the VR goggles 15 from the projection unit 12 and display it on the two-dimensional display 11c.

[0072] The user operation unit 11d is a device that can accept various button operations and coordinate input operations from the user, such as a mouse or keyboard, which are common input devices. In this embodiment, the user operation unit 11d can accept input operations from each participant P12 to P42. Specifically, the user operation unit 11d can make modification instructions such as moving, changing, adding, or deleting parts of the 3D image 21 of the VR study area 20 displayed on the screen of the two-dimensional display 11c. The user operation unit 11d also accepts the specification of a position on the screen of the two-dimensional display 11c as an input operation from each participant P12 to P42. The user operation unit 11d calculates the position clicked on the screen of the two-dimensional display 11c. The calculated position information is transmitted to the conference management server 30 via the communication device 11a. Then, as will be described later, a dart pointer is generated at the specified position through processing within the conference management server 30 and reflected in the 3D image 21.

[0073] The audio transmission unit 11e can transmit participant voice information captured by the microphone of the headset 17 to other locations via the communication device 11a and the communication network 25. In addition, the audio transmission unit 11e can receive voice information emitted by researchers and participants at other locations via the communication network 25 and the communication device 11a, and output it as audio from the speaker of the headset 17.

[0074] <System Operation> An example of the operation of the VR conferencing system 100, particularly an example of the process for displaying a dart pointer, will be explained with reference to Figures 4 to 6.

[0075] The VR data generation unit 35 on the conference management server 30 generates three-dimensional data of the three-dimensional image 21 in the VR space of the VR study room 20 based on the wire harness design data held by the database management unit 33. In addition, the VR data generation unit 35 also generates three-dimensional data of the three-dimensional image 21 in the VR space of the VR study room 20 for each avatar A11 to A41 managed by the avatar control unit 36.

[0076] When starting an online meeting using the VR conferencing system 100, the projection units 12 worn by the participants P11 to P41 at each location, and the display units 11 used by each participant P12 to P42, are connected to the meeting management server 30 via the communication network 25, enabling them to communicate with each other.

[0077] Each projection unit 12 at each location acquires the data of the stereoscopic image 21 within the VR discussion room 20 generated by the VR data generation unit 35 of the conference management server 30. This data is then coordinate-transformed into a stereoscopic image that will be projected into the field of view of each participant P11 to P41's left and right eyes, and projected onto the VR goggles 15 so that each participant P11 to P41 can see it. The visible area (visible region) of the image that each participant P11 to P41 can see with the VR goggles 15 is determined for each set of VR goggles 15.

[0078] Furthermore, the display unit 11 at each location acquires the data of the 3D image 21 within the VR discussion room 20 generated by the VR data generation unit 35 of the conference management server 30, and transforms the coordinates of this data so that it closely matches the 3D image seen in the field of view of each participant P11 to P41 at the same location, and displays it on the screen of the two-dimensional display 11c.

[0079] Figure 4 shows an example of the display screen 111 of the display unit 11 located within base H1. The display screen 111 is generated by the two-dimensional image generation unit 11b and includes a VR screen 113 and a VR operation screen 115. The VR operation screen 115 is a screen for inputting operations such as tool selection for the VR screen 113. For example, if the dart pointer mode is selected on the VR operation screen 115, an icon 113d is displayed on the VR screen 113, and the dart pointer 201d (see Figure 5) becomes available. The VR screen 113 is a two-dimensional image whose coordinates have been transformed to closely match the three-dimensional image projected by the projection unit 12 within base H1 into the field of view of the examiner P11. The two-dimensional image generation unit 11b fixes the size of the VR screen 113 to the area visible through the VR goggles 15 within base H1. If the size of VR screen 113 is not fixed, when the size of VR screen 113 is changed, the image displayed in the viewer P1's field of view will differ, and part of it will be cut off.

[0080] As each reviewer P11-P41 moves in the real world or changes their posture and gaze while viewing the 3D image projected by the VR goggles 15, these changes are detected by the user position detection unit 12b and the user posture detection unit 15d. The changes in the posture and gaze of each reviewer P11-P41 in the real world are then reflected in the changes in their field of view within the VR review area 20.

[0081] The VR image generation unit 15a of the VR goggles 15 updates the stereoscopic image projected onto the left-eye display 15b and the right-eye display 15c in accordance with the change in the field of view of the study. In addition, the two-dimensional image generation unit 11b of the display unit 11 updates the image displayed on the two-dimensional display 11c to follow the change in the field of view of the study located at the same site. The display size of this image is fixed.

[0082] The reviewers P11-P41 at each location can operate the user control unit 12c to make changes to the areas of interest in the 3D image displayed in their field of view as needed. For example, they can change the shape of the area of ​​interest in the wire harness, move the position of each jig, or add or remove parts and jigs from the wire harness.

[0083] Furthermore, participants P12 to P42 at each location can communicate areas of interest to reviewers P11 to P41 and other participants P12 to P42 by operating the user control unit 11d. For example, as shown in Figure 4, if participant P12 positions the mouse pointer 113e over an area of ​​interest on the VR screen 113 and clicks the mouse, the position on the VR screen 113 is entered as the specified location.

[0084] Correction inputs made by reviewers P11-P41 at each location, and designated position inputs made by participants P12-P42 at each location, are input from the projection unit 12 and display unit 11 to the conference management server 30 via the communication network 25. The change control unit 37 of the conference management server 30 receives the correction inputs from each reviewer P11-P41 and the designated position inputs from each participant P12-P42 and records their contents as update data in the database management unit 34.

[0085] For example, as shown in Figure 4, when a specified location is entered by participant P12, the user operation unit 11d calculates the position clicked with the mouse, using the lower left corner of the fixed-size VR screen 113 as the reference point 113a. The user operation unit 11d calculates the X coordinate position 113b to the right of the reference point 113a, and the Y coordinate position 113c upwards from the reference point 113a, and obtains the location information of the specified location. The calculated location information is input from the communication device 11a to the conference management server 30 via the communication network 25 and recorded in the database management unit 34 as update data.

[0086] When the VR data generation unit 35 of the conference management server 30 detects that new update data has been added to the database management unit 34, it reflects the modifications from the update data in the 3D image 21 of the VR discussion field 20 that it generates, and generates new 3D image 21 data. If a specified position is input from the display unit 11, the VR data generation unit 35 obtains at least the position data of the VR goggles 15 worn by the participant P11 and sets it as the starting position of the dart pointer. The VR data generation unit 35 uses the position information recorded in the database management unit 34 as the goal position and generates 3D image 21 data for the dart pointer as it moves from the starting position to the goal position and reaches the goal position.

[0087] The communication device 31 of the conference management server 30 transmits the modified stereoscopic image data 21 generated by the VR data generation unit 35 to each of the system equipment 10A to 10D at each of the locations H1 to H4.

[0088] Each projection unit 12 at each location acquires the data of the modified stereoscopic image 21 transmitted from the conference management server 30, transforms the coordinates of this data into a stereoscopic image that will be projected into the field of view of each participant P11 to P41's left and right eyes, and projects it so that each participant P11 to P41 can see it using VR goggles 15. For example, the projection unit 12 at location H1 projects the stereoscopic image 201 shown in Figure 5. In the stereoscopic image 201 of Figure 5, the dart pointer 201d is displayed so that its tip is positioned at position 201e. Position 201e is the position where the X coordinate position 201b and Y coordinate position 201c correspond to the X coordinate position 113b and Y coordinate position 113c shown in Figure 4, with the lower left corner as the reference point 201a. The dart pointer 201d is drawn, for example, as shown in Figure 6, with the position of avatar A11 in the VR play area 20, i.e., the position of the VR goggles 15, as the starting position T1, and along the vector VE that points toward the goal position T2, so that its tip is positioned at the goal position T2. ​​The dart pointer 201d is an example of a pointer.

[0089] Each display unit 11 at each location acquires the data of the modified stereoscopic image 21 transmitted from the conference management server 30, transforms the coordinates of this data so that it closely matches the stereoscopic image seen in the field of view of each reviewer P11 to P41 at the same location, and displays it on the screen of the two-dimensional display 11c.

[0090] Therefore, by using the VR conferencing system 100 shown in Figures 2 and 3, as shown in Figure 1, all of the participants—scientists P11 and P12 at site H1, scholars P21 and P22 at site H2, scholars P31 and P32 at site H3, and scholars P41 and P42 at site H4—can conduct online meetings without moving, using the virtual reality space of the common VR discussion room 20.

[0091] In particular, each reviewer P11 to P41 can perceive the three-dimensional image 21 projected by the VR goggles 15 in the same way as the real object, and their movement and changes in posture are reflected in their field of view within the VR review area 20, making it easy to confirm the three-dimensional shape and structure of the parts that need to be reviewed in detail.

[0092] Furthermore, since avatars A11-A41, corresponding to the reviewers P11-P41 at each location, are also included in the 3D image 21 within the VR review space 20, each reviewer P11-P41 can recognize the parts of the wire harness and the direction of their gaze that other reviewers are examining. In other words, even though each reviewer P11-P41 is at a different location, they can easily grasp their relative positions to one another, allowing them to efficiently review common areas of interest on the 3D image 21, just as they would in a meeting held in a shared physical space.

[0093] Furthermore, the reviewers P11-P41 at each location can perform modifications to the 3D model 21, such as changes, additions, and deletions, by inputting information as needed. In addition, since the results of the modifications are reflected in the content of the projected 3D model 21, the reviewers P11-P41 at each location can easily grasp the 3D shape and structure of the modified 3D model 21.

[0094] Furthermore, participants P12-P42 at each location can view a two-dimensional image on the display unit 11 that covers approximately the same range as the three-dimensional image 21 seen in the field of view of examiners P11-P41 at the same location. Therefore, even if there are no physical models in each venue V1-V4, participants P12-P42 can easily grasp the shape and structure of the part under consideration, just like examiners P11-P41.

[0095] In particular, in the VR conferencing system 100, participants P12 to P42 can specify a position on the display unit 11 screen using a mouse or the like, and a dart pointer 201d will be displayed at the corresponding position on the 3D image 21. Therefore, participants in an online meeting can share areas of interest, enabling smooth communication.

[0096] Furthermore, the present invention is not limited to the embodiments described above, and can be modified, improved, etc., as appropriate. In addition, the material, shape, dimensions, number, placement, etc. of each component in the embodiments described above are arbitrary and not limited, as long as they can achieve the present invention.

[0097] For example, in the embodiment described above, it is assumed that an online meeting is held using the VR conferencing system 100 to discuss the design of an automotive wire harness at multiple locations. However, not only wire harnesses, but various types of products can be placed as three-dimensional images 21 in the VR study area 20 and used as subjects for study.

[0098] Furthermore, if the same functionality as the conference management server 30 is installed in any of the system equipment 10A to 10D at multiple locations, the conference management server 30 can be omitted. For example, if the same functionality as the conference management server 30 is installed in the projection unit 12 of system equipment 10A, the display unit 11 may receive a location specification from the user operation unit 11d and transmit the calculated location information to the projection unit 12 via the communication device 11a. Then, processing within the projection unit 12 generates a dart pointer 201d, which is reflected in the 3D image 21.

[0099] Furthermore, each display unit 11 located at each site H1 to H4 displays the field of view of the respective reviewers P11 to P42 at each site H1 to H4. However, the display unit 11 may be configured to allow participants P12 to P42 at each site H1 to H4 to select which reviewer's field of view to display on the screen of each display unit 11. In other words, each display unit 11 can display the field of view of a selected reviewer from among the reviewers P11 to P41 at any of the sites H1 to H4. For example, if participant P12 selects the field of view of reviewer P21, an image of the VR discussion room 20 with approximately the same range as the image in reviewer P21's field of view will be displayed on the screen of the display unit 11 at site H1 via the conference management server 30. By participant P12 specifying a position on this screen, examiner P21's projection unit 12 can display a dart pointer 201d in the VR discussion area 20, representing the pointer moving from the position of examiner P21's projection unit 12 to the specified position and reaching that position. This display of the dart pointer 201d is reflected in real time in the field of view of other examiners P11, P31, and P41, and is also reflected in the display content of display units 11 in other locations H2, H3, and H4.

[0100] Herein, the features of the virtual reality conferencing system according to the embodiments of the present invention described above are briefly listed below in [1] to [3]. [1] A communication system (VR conferencing system 100) comprising a first electronic device (projection unit 12) and a second electronic device (display unit 11) capable of communicating with the first electronic device, The first electronic device has a first display unit (VR goggles 15) that displays an object (3D image 21) edited in response to user operation within a virtual reality space, The second electronic device is, A second display unit (two-dimensional display 11c) that displays the field of view (VR screen 113) of the first display unit of the first electronic device, It includes a position specification unit (user operation unit 11d) that accepts the specification of the position of the second display unit on the screen, The first display unit displays a pointer (dart pointer 201d) at a position in the virtual reality space corresponding to the position. Communication system.

[0101] According to the communication system with the configuration described in [1] above, when a location is specified on the second display unit of the second electronic device, a pointer is displayed on the first display unit of the first electronic device. Therefore, a person looking at the second display unit of the second electronic device and a person looking at the first display unit of the first electronic device can share the area of ​​interest, enabling smooth communication.

[0102] [2] The size of the field of view displayed on the second display unit by the first display unit is fixed. The communication system described in [1] above.

[0103] According to the communication system with the configuration described in [2] above, the size of the field of view displayed on the second display unit by the first display unit is fixed, so the second display unit can display an image equivalent to the field of view of the first display unit without any part of the field of view of the first display unit being cut off.

[0104] [3] The second electronic device calculates the position information of the position received by the position designation unit in the second display unit and transmits it to the first electronic device. The first electronic device uses the position based on the position information as the goal position of the pointer, and the position of the first electronic device in the virtual reality space as the starting position, and displays the pointer in the virtual reality space in a state in which it has moved from the starting position to the goal position and reached the goal position. The communication system described in [2] above.

[0105] According to the communication system with the configuration described in [3] above, in the virtual reality space, a pointer is displayed in the line of sight of the person looking at the first display unit, and their gaze is guided to the goal position, so that the person looking at the first display unit can easily grasp the area of ​​interest. [Explanation of Symbols]

[0106] 10A, 10B, 10C, 10D System Equipment 11 Display Unit 11a Communication equipment 11b Two-dimensional image generation unit 11c 2D display 11d User Operation Unit 11e Voice transmission unit 12 Projection Units 12a Communication equipment 12b User position detection unit 12c User Operation Unit 12d Audio transmission unit 13,14 Position Sensor 15 VR goggles 15a VR video generation section 15b Left eye display 15c Right Eye Display 15d User posture detection unit 16,17 Headset 20 VR Study Group 21 3D image 25 Communication Networks 30 Meeting Management Server 31 Communication equipment 32 Participant Management Department 33,34 Database Management Department 35 VR Data Generation Unit 36 Avatar Control Unit 37 Change Control Unit 100 VR conferencing systems 201d Dart Pointer A11, A21, A31, A41 Avatars H1, H2, H3, H4 locations Reviewers on pages P11, P21, P31, and P41. P12,P22,P32,P42 participants V1, V2, V3, V4 venues

Claims

1. A communication system comprising a first electronic device and a second electronic device capable of communicating with the first electronic device, The first electronic device has a first display unit that displays an object edited in response to user operation within a virtual reality space, The aforementioned second electronic device is A second display unit that displays the field of view of the first display unit of the first electronic device, The system includes a position designation unit that accepts the designation of the position of the second display unit on the screen, The first display unit displays a pointer at the position in the virtual reality space corresponding to the position, The second electronic device calculates the position information of the position received by the position designation unit in the second display unit and transmits it to the first electronic device. The first electronic device has a position based on the position information in the virtual reality space as the goal position, and the position of the first electronic device in the virtual reality space as the start position. It displays a dart-shaped pointer in the virtual reality space, with its tip positioned at the goal position, along a vector from the start position to the goal position. The second electronic device updates the image displayed on the second display unit in accordance with the change in the field of view of the first display unit, and the display size of the image is fixed. Communication system.

2. The image of the virtual reality space displayed on the first display unit includes a three-dimensional image of an avatar corresponding to a person wearing the first electronic device in real space. The change in the position of the person being studied in the real space is detected by a position sensor placed in the real space and reflected in the change in the position of the avatar in the virtual reality space. Changes in the posture of the examiner in the real space are detected by the first electronic device worn by the examiner and reflected in changes in the posture of the avatar in the virtual reality space. The communication system according to claim 1.

Images