Design supporting system
The design support system uses MR technology to create a realistic virtual environment for product design evaluation, addressing inefficiencies and costs associated with traditional methods by allowing accurate and geographically dispersed evaluations.
Patent Information
- Application Number
- JP2023070976
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-24
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2043-04-24
AI Technical Summary
The challenge in product design is the inefficiency and high cost of evaluating large-scale designs, particularly for products like vehicles, due to the need for accurate evaluations by multiple parties in various environments, geographical dispersion, and the requirement for confidentiality, which complicates the workflow and increases the risk of missed market opportunities.
A design support system utilizing mixed reality (MR) technology to create a high sense of reality by fusing real and virtual spaces, allowing designers to evaluate designs in a virtual environment that reflects the surrounding light environment using a headset with an external camera, attitude sensors, and image processing units to generate a highly realistic presentation image.
The system enhances design evaluation efficiency by providing a high sense of reality and accuracy without the need for physical models, reducing costs and geographical barriers, and enabling simultaneous evaluation by multiple parties.
Smart Images

Figure 2024156478000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a design support system. More specifically, the present disclosure relates to a system for supporting the design activity of an article. [Background technology]
[0002] The design of the appearance of various items such as industrial products and the internal structure that realizes it affect each stage of the design, production, and sales of the item from the planning stage of the item, and are usually decided with the involvement of many stakeholders. For example, in the automobile industry, the specifications of the vehicle are decided according to various conditions, and the design of the exterior (exterior) and interior (interior) are carefully decided through a complex process according to the destination and expected customers. In this process, the parties involved in the planning and manufacturing of the item repeatedly review (design review) and decide the design of the product to be developed. In some cases, evaluations (design clinics) are conducted in which designs under development or before announcement are presented to third parties such as expected users. In order to accurately evaluate the design, not only sketches and design drawings but also prototypes and models that realize the final product with a good reproducibility have been adopted. Even for large items such as vehicles, the final product has been reproduced as accurately as possible by producing large prototypes such as full-scale prototypes and life-size clay models.
[0003] Recently, advances in computer technology have been incorporated into design activities, for example, 3D prototyping is being used in computer-aided design (CAD) systems, and virtual reality (VR) systems and mixed reality (MR) systems are being used in industrial design. Patent Document 1 (JP Patent Publication 2017-59213 A) discloses a method and device for overcoming technical challenges in managing design changes to object models. Patent Document 2 (Registration No. 7150354) discloses a system for supporting vehicle interior design work. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2017-59213 A [Patent Document 2] Registration No. 7150354 Summary of the Invention [Problem to be solved by the invention]
[0005] In the design of an item, for example, a vehicle, the greater the investment required, the more involved the number of people from various positions are. In addition, the larger the size of the item, the more useful it is to use prototypes or models (referred to as "models, etc.") made for evaluation to improve the accuracy of the design evaluation. However, due to their size, the time required to create the models, etc. is great, and the workload is heavy. Large models, etc. are limited in the environment in which they can be placed, and their transportation costs are high, so it is not easy to have many evaluators from various positions evaluate them. In this way, there is a contradiction in that the more you want to improve the accuracy of your judgment in design evaluation, the more difficult it is to implement an ideal workflow, and the design evaluation itself requires a great deal of cost. In addition, for items whose design itself has a large impact, confidentiality is often required until the item is made public. For such items, it is also a heavy burden to carry out design evaluations in various environments and for evaluators who are geographically dispersed to evaluate the design. The perspective of time is also important, and if it takes a long time to reflect the design of an item in the final product, the risk of missing the opportunity to sell the item due to changes in design trends increases. In the design of products that take into account a wide range of factors and are designed to be used in a variety of environments, there is a need for methods to evaluate and verify products efficiently and with little burden, as well as to overcome geographical distances.
[0006] The present disclosure aims to solve at least some of the above problems, and by providing an efficient system and computer program that supports product design work, it is possible to increase the efficiency of product design work and contribute to the sophistication of the products. [Means for solving the problem]
[0007] The inventor conceived the idea that the design evaluation activity of an item can be made more efficient by utilizing the progress of mixed reality (MR) technology, which can provide a higher level of reality than virtual reality (VR) systems. In MR technology, when the real space, which is one's own environment, and the virtual space in which the item is placed are merged, the user no longer distinguishes between the real space and the virtual space, and can see the design of the item with a high level of reality. When the design of an item is reproduced using advanced MR technology, even if the item requires high reproducibility for realism, such as an item that is familiar to the user (e.g., a vehicle), it can be made to feel as if a model is placed in front of the user, and the user can immerse himself / herself in the design work. Moreover, the design stage item reproduced in the mixed reality space can be easily modified, such as changing the structure, or changing attributes such as color, shape, and texture. If the design of the item can be perceived by the user with a high level of reality using MR technology, a design evaluation can be performed with high accuracy without the need to create a model, and the design evaluation work can be made more efficient.
[0008] In order to utilize MR technology, the inventors of the present application focused on factors that may affect the reproducibility of the design of an item to provide a high sense of reality and may affect the sense of immersion. The inventors focused on the fact that if the surrounding light environment cannot be sufficiently reflected in the appearance of the item using MR technology, it is difficult to obtain a sufficient sense of reality even if the appearance of the item is observed. However, the surrounding light environment is infinitely diverse, and measuring the brightness alone is insufficient. In addition, it is not easy to measure the surrounding light environment using a special measuring device. Therefore, the inventors of the present application found that using a headset such as an HMD equipped with an external camera can lead to a highly practical method in order to reflect the surrounding light environment in the appearance of the item. In this disclosure, a design support system is provided that can support design work by providing a high sense of reality to the design of an item using a virtual space.
[0009] That is, in one aspect of the present disclosure, there is provided an item data recording unit that stores structure data for determining a structure of at least one item in a virtual space and attribute candidate data for selecting or adjusting attributes of the item; at least one headset to be worn on the head of an operator, the headset including a posture sensor that can detect the posture of the headset in real space, at least one display that can display an image to be presented to the operator, and at least one external camera that can capture at least a portion of the external world corresponding to the field of view of the operator; an ambient light recording unit that records brightness distribution data indicating the brightness in each direction around the operator captured by the external camera in a pre-capture in association with the output of the posture sensor; and a background image rendering unit that generates a background image corresponding to the output of the operation input unit; at least one operation input unit that receives from the operator an operation input for selecting or adjusting the attribute of the item; an image adjustment unit that generates attribute data for the attribute selected or adjusted from the attribute candidate data in accordance with instructions of the operation input received by the operation input unit; an item image rendering unit that generates an item image under illumination corresponding to the brightness distribution data, which corresponds to the instructions of the operation input and the output of the posture sensor at each time point, from the structure data called up from the item data recording unit, the attribute data from the image adjustment unit, and the brightness distribution data called up from the ambient light recording unit; and an image integration unit that synthesizes the background image and the item image and outputs the result as an image signal for the presented image.
[0010] The present disclosure further provides a computer program for the above-described design support system.
[0011] In this application, technical terms used in the field of computer graphics may be used unless otherwise specified. In this application, the object to be designed is called an "article." In this application, the object is not particularly limited, and includes any object whose shape can be recognized by the end user through vision, such as movable property, real estate, tangible objects, and intangible objects in real space. Therefore, the article in this application includes any object with a real shape, and may also include an object that is intended to be reproduced by simulating real space in a virtual space or a composite space (a space that combines virtual space and real space). Note that many explanations in this application are given with vehicles (automobiles) as the object, and technical terms for vehicles also appear, but these are merely examples. In this application, design includes not only designs as aesthetic appearances, but also those that target shapes, such as structural design of articles. For this reason, design work is not necessarily limited to activities that focus only on aesthetic points of view, but also includes a wide range of creative activities, including structural design and mechanism design. Design work is typically an activity in which a design is determined by evaluating or modifying the design in the above activities including structural design and exterior design. However, the design support system of the present disclosure can also be used to exhibit or experience a design, and in that sense, activities that do not necessarily bring about active changes to an item can also be design work in the present disclosure. Pose is a combination of orientation and position, and a sensor that detects posture is intended to detect at least one or more components of orientation and position. A virtual space is a three-dimensional space defined on a computer, and is described as being distinguished from a real space, but this distinction is for the sake of explanation. In this application, an operator generally refers to a person who experiences or uses the design support system or a user. In the context of evaluating the design of an item, the evaluator can be the operator. A real space is a space in which the operator's body actually exists. A headset is a device that can present an image to at least the operator's visual sense by being fixed to the operator's head in mixed reality technology.Head-mounted displays (HMDs), VR glasses, XR glasses, and the like are typical examples of such headsets. The headset of the present application is equipped with an external camera. This external camera is typically a pair of cameras corresponding to both eyes of the operator, and there may be multiple pairs. In this disclosure, images include still images and videos. Furthermore, videos include videos, but may also be called videos even if they are partially or entirely still images. Effect of the Invention
[0012] According to one aspect of the present disclosure, there is provided a design support system or computer program for streamlining product design work. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is an explanatory diagram outlining a state in which a design support system according to an embodiment of the present disclosure is used; [Diagram 2] FIG. 1 is a block diagram showing functions of a design support system according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is an explanatory diagram illustrating a configuration of a headset employed in the design support system according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating an overview of a method of using the design support system according to an embodiment of the present disclosure. [Figure 5A-B] 5A is an explanatory diagram for explaining the operation of an external camera in the design support system according to the embodiment of the present disclosure, and FIG. 5B is a display showing an example of the captured image. [Figure 6A-B] 6A is an explanatory diagram for explaining the operation of an external camera in the design support system according to the embodiment of the present disclosure, and FIG. 6B is a display showing an example of the captured image. [Figure 7A-C] FIG. 7A is an explanatory diagram showing the hardware configuration of a design support system according to an embodiment of the present disclosure, and FIG. 7B and FIG. 7C are explanatory diagrams showing how a background image, an item image, and a foreground image are synthesized in the operation of generating presentation images for both the left and right eyes. [Figure 8] FIG. 1 is a block diagram showing functions of a design support system according to an embodiment of the present disclosure. [Figure 9] FIG. 1 is an explanatory diagram illustrating a hardware configuration of a design support system according to an embodiment of the present disclosure. [Figure 10A-C] 1 is a captured image of a display in an example of a design support system according to an embodiment of the present disclosure. [Figure 11A-B] 11A and 11B are capture images of a display in an example of the design support system according to an embodiment of the present disclosure, showing a vehicle displayed as an item before (FIG. 11A) and after (FIG. 11B) changing the body color. [Figure 12A-B] 12A and 12B are captured images of the display of a headset in an example of a design support system according to an embodiment of the present disclosure, showing specular reflection (FIG. 12A) and diffuse reflection (FIG. 12B) when a sphere is displayed as an object. [Figure 13A-C] 13A and 13B are explanatory diagrams showing the operation of a transformation function in the design support system according to the embodiment of the present disclosure, illustrating the state before transformation (FIG. 13A), the bounding box (FIG. 13B), and the state after transformation (FIG. 13C). [Figure 14A-B] These are captured images of the display of a headset in a design support system according to an embodiment of the present disclosure, showing the state when using the distance measuring function (FIG. 14A) and the state when using the cross-sectional display function (FIG. 14B). [Figure 15A-B] 15A and 15B are capture images of the display of a headset in a design support system according to an embodiment of the present disclosure, showing an example of displaying an image of real space from an external camera for the background image, and an example of displaying an image of virtual space (FIG. 15B). [Figure 16] FIG. 1 is an explanatory diagram showing an example configuration in a situation where multiple operators work together via a network according to an embodiment of the present disclosure. [Figure 17] FIG. 1 is an explanatory diagram of a design support system including an eye tracking detection unit employed to detect and record a gaze point in an embodiment of the present disclosure. [Figure 18]1 is a block diagram of a design support system employing a headset with an eye tracking detector in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, an embodiment of an article design support system according to the present disclosure will be described with reference to the drawings. Throughout the drawings, common parts or elements are given common reference symbols unless otherwise specified in the description. In addition, in the drawings, each of the elements of each embodiment is not necessarily shown to scale.
[0015] 1. System Configuration The design support system provided in the present disclosure will be described.
[0016] 1-1.Configuration (overview) FIG. 1 is an explanatory diagram outlining the use of a design support system 1 according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing the functions of the design support system 1 provided in the present disclosure. In a real space (real space) in which the design support system 1 is used, an operator 2 wears a headset 10. The real space 9 is, for example, a room in which the operator 2 actually exists, and the space includes a light 92 and the like. The real space 9 may also include light from a window or various other lights emitted by some light source. A presented image 4 that appears as if an object 3 exists in the real space 9 is provided to the operator 2 through a display 102 of the headset 10 via an appropriate eyepiece optical system (not shown). The presented image 4 includes, for example, a background image 42 and an object image 44. The operator uses such a design support system 1 to evaluate the design of the object 3. When the design support system 1 of the present disclosure is adopted, the object 3 in the presented image 4 has a high sense of reality for the operator 2. In the design support system 1 of the present disclosure, the brightness distribution data 522 captured by the external camera 108 of the headset 10 is captured for the background image 42. The captured data for the background image 42 is captured by the external camera 108 by the operator 2 wearing the headset 10 on his head and looking around, etc., in a stage before the design of the article 3 is evaluated, which is called pre-capture, and is recorded in the ambient light recording unit 520. The headset 10 is equipped with a posture sensor (posture sensor mover 104). While the posture sensor detects the posture of the headset 10 at each time, a background image 42 and an article image 44 suitable for that posture are generated by the computer 50. The background image 42 is synthesized with the article image 44 to become the presented image 4. Here, when the article image 44 is generated according to the posture of the headset 10, it is generated so as to reflect the structure and attributes of the article 3 in the virtual space. The brightness distribution of the background in the photographed data of the external camera 108 is reflected in the appearance of the article 3 in the article image 44. In other words, the article image 4 realizes a highly realistic display that reflects how the external appearance of the article 3 looks under lighting based on the brightness distribution acquired by the external camera 108.This appearance reflects differences in brightness of various parts of the surface of the article 3 that reflect the surrounding light environment, such as brightness according to the illuminance of the lighting 92, and furthermore, if the surface of the article 3 exhibits, for example, specular reflectivity, even reflections are reproduced. Therefore, the article 3 perceived by the operator 2 through his / her eyes has a high sense of reality, as if it were an actual object or a model existing in the real space 9.
[0017] The design support system 1 can be realized as hardware by a headset 10 and associated peripheral devices and a computer 50. The headset 10 is equipped with an external camera 108, and typically does not have an optically transparent part. The computer 50 can be a personal computer or a workstation with high graphic performance, but as a result of recent advances, even an easily available computer can achieve satisfactory performance. The computer 50 can be one having general computing resources such as a recording device (main storage device, auxiliary storage device), a graphic device, an arithmetic device, an input / output device, a network device, and an input device. The headset 10 and a controller (operation input unit) 120 are connected through an appropriate wired or wireless connection interface. It is important to note that the minimum hardware configuration for realizing the design support system 1 does not include additional hardware (e.g., a celestial camera) for the purpose of acquiring luminosity distribution data for the real space 9, and the adoption of such an additional hardware configuration is optional. For this reason, the design support system 1 requires only a relatively easy-to-obtain and inexpensive hardware configuration.
[0018] The design support system 1 includes an item data recording unit 510, at least one headset 10, an ambient light recording unit 520, a background image rendering unit 530, at least one operation input unit 120, an image adjustment unit 540, an item image rendering unit 550, and an image integration unit 580. The item data recording unit 510 stores structure data 512 for determining the structure of at least one item 3 in the virtual space and attribute candidate data 514 for selecting or adjusting attributes of the item 3. Here, an attribute is a property that is subject to selection or adjustment in design work. Therefore, when the purpose of the design work is a design change involving a shape change, a structure for specifying a shape can be a typical example of an attribute. The headset 10 will be described in detail in the next section. The ambient light recording unit 520 records brightness distribution data 522 indicating the brightness in each direction around the operator 2 captured by the external camera 108 in the pre-shooting in association with the output of the posture sensors 104 and 106. The background image rendering unit 530 generates a background image 42 corresponding to the output of the orientation sensors 104, 106 at each time point, i.e., the latest output, from the brightness distribution data 522 called up from the ambient light recording unit 520. At least one operation input unit 120 receives an operation input from the operator 2 for selecting or adjusting an attribute of the article 3. The image adjustment unit 540 generates attribute data 542 for an attribute selected or adjusted from the attribute candidate data according to the instruction content of the operation input received by the operation input unit 120. The article image rendering unit 550 generates an article image 44 under illumination corresponding to the brightness distribution data 522, which corresponds to the instruction content of the operation input and the output of the orientation sensors 104, 106 at each time point, from the structure data 512 called up from the article data recording unit 510, the attribute data 542 from the image adjustment unit, and the brightness distribution data 522 called up from the ambient light recording unit 520. The image integrating unit 580 combines the background image 42 and the item image 44 and outputs the combined image as an image signal for the presentation image 4.
[0019] In the design support system 1, it is preferable that the article image rendering unit 550 reflects a surrounding reflected image corresponding to the brightness distribution data in the surface reflection of the article image. Also, it is preferable that the background image rendering unit 530 in the design support system 1 retrieves structural data from the article data recording unit, and reduces the brightness of the background image in the direction of the shadow that the article should cast in real space, corresponding to the brightness distribution data.
[0020] The brightness distribution data 522 may also acquire brightness by color. In this case, the ambient light recording unit 520 records color brightness data (not shown) indicating the brightness of each color photographed by the external camera 108 in the pre-photographing in association with the output of the orientation sensors 104, 106. The background image rendering unit 530 generates a color background image corresponding to the output of the orientation sensors 104, 106 at each time point from the color brightness data called up from the ambient light recording unit 520, and the article image rendering unit 550 generates an article image under illumination corresponding to the color brightness data corresponding to the instruction content of the operation input and the output of the orientation sensors 104, 106 at each time point from the structure data 512 called up from the article data recording unit 510, the attribute data 542 from the image adjustment unit 540, and the color brightness data called up from the ambient light recording unit 520, so that the article image has a higher sense of reality in which the colors in the real space 9 are reflected.
[0021] At least one operation input unit 120 may be any device capable of inputting instructions to the computer 50. Even when the operator 2 wearing the headset 10 visually checks the photographed data at the time of the preliminary operation in the real space 9 only through the display 102, the operator 2 can perform the operation in real time. As the operation input unit 120 suitable for the design support system 1 that can handle such a situation, in addition to any controller that can be held in the hand to scan the direction or control the input button, a sensor (hand tracking sensor) that can capture the movement of the operator's hand as shown in the hand tracking camera 108A (FIG. 3) can be adopted. If a laser pointer or menu display in the virtual space is provided to the operator 2 through the presented image 4 in response to the operation of the operation input unit 120, or a display suitable for input in the virtual space such as a virtual keyboard is adopted, it becomes easy to give an execution instruction as a configurator, such as a color change, a model change, a texture change, or a shape change, in order to determine the attributes of the article 3.
[0022] 1-2.Headset 3 is an explanatory diagram illustrating a configuration of a headset 10 employed in the design support system of the embodiment of the present disclosure. A typical example of the headset 10 is, for example, the Varjo XR-3 manufactured by Varjo Technologies Oy.
[0023] The headset 10 is to be worn by the operator 2 on the head, and includes a posture sensor mover 104 capable of detecting the posture of the headset 10 in the real space 9, a posture sensor stator 106, at least one display 102 capable of displaying a presentation image 4 to the operator 2, and at least one outside camera 108 capable of capturing at least a part of the outside world corresponding to the field of view of the operator 2. In a typical configuration, only the presentation image 4, i.e., an image generated through the design support system 1, exists as a visual stimulus in the field of view that the operator 2 can perceive through his / her eyes. For this reason, the headset 10 does not have an optically transparent part such as HoloLens (registered trademark) that allows the operator 2 to directly view the outside world and provides an optical pass-through image. In the presentation image 4 presented to the operator 2, both the background image 42 and the item image 44 are provided through the display 102 of the headset 10.
[0024] The display 102 and the external camera 108 are paired to correspond to both eyes in order to enhance the sense of reality in terms of depth perception by so-called stereoscopic vision that reflects binocular parallax. Accordingly, it is also preferable that each headset is provided with two systems of the article image rendering unit 550, the background image rendering unit 530, and the image integration unit 580.
[0025] 1-3. Acquiring brightness distribution data using an external camera As shown in FIG. 3, the external camera 108 can capture images in a field of view fixed to the headset 10. The external camera 108 is preferably one that has a wide angle of view close to the field of view of the human eyeball (approximately 120°). In FIG. 3, the range of the angle of view that the external camera 108 that is paired with both eyes can capture is represented by a cone mark. In the design support system 1 of the present disclosure, the external camera 108 acquires brightness distribution data 522. If the orientation of the headset 10 is fixed, the external camera 108 may not be able to capture the celestial sphere (the entire celestial sphere or a hemisphere). In this case, the operator 2 scans the direction of the camera 108 to acquire the brightness data 522. In other words, it is possible to acquire the brightness for each direction by capturing images using the external camera while changing the orientation of the headset in the pre-capture. In this pre-shooting, for example, the operator 2 wearing the headset 10 tucks his chin in, maintains an upright position, scans all directions in the horizontal plane, then lifts his chin and scans all directions in the horizontal plane again, and so on. In this way, shooting in the direction required for the brightness distribution data 522 can be performed. At this time, it is preferable that posture data is associated with the shooting data and stored in the ambient light recording unit 520. From this posture data, the shooting direction of the external camera 108 can be determined, and in order to construct brightness distribution data over a sufficient range of directions such as the celestial sphere, this can be useful for image stitching, for example. In addition, this posture data can also be used to associate the brightness distribution data with directions in real space.
[0026] It is useful for the lightness distribution data to be stored with sufficient reproducibility of the lightness of surrounding objects in a luminance range so as to be useful for reproducing lighting and reflections. For this purpose, for example, any format for a wide latitude HDR (high dynamic range) image is suitable. However, the design support system 1 does not necessarily require photometric or colorimetric accuracy or traceability of the lightness distribution data. The degree of accuracy required for the operator 2 to have a sense of reality can be determined according to the specific use and purpose of the design evaluation.
[0027] FIG. 4 is a flowchart illustrating an outline of a method of using the design support system according to the embodiment of the present disclosure. In the above-mentioned pre-photographing S02, luminance distribution data 522 is acquired. This luminance distribution data 522 is stored in the ambient light recording unit 520 in association with the posture. After that, the attributes of the article 3 are set (S04), and the design is displayed or evaluated (S06). In this display or evaluation S06, the luminance distribution data 522 captured by the operator 2 of the article 3 in his / her own environment through the external camera 108 is used for two purposes. That is, the luminance distribution data is used for the background image 42 and for illumination of the article 3 in the article image 44. After the design of the article 3 is evaluated in the attribute setting S04, the conditions are changed as necessary (S08), and the next conditions are set (S10). The conditions changed here are typically attributes for the design.
[0028] In the above operation, the brightness distribution data 522 at the time of the pre-photographing S02 is used as it is for the background image 42 and the article image 44 observed by the operator 2. In the design support system 1, in order to enhance the sense of reality, it is also useful to replace the data of the brightness distribution data 522 corresponding to at least a part of the field of view of the operator 2 wearing the headset 10 with brightness data at each time point captured by the external camera 108. Such processing is performed by the background update processing unit 532 (FIG. 2). In the flowchart of FIG. 4, since the display or evaluation S06 is an evaluation of the article 3 by the operator 2, it is performed over a certain amount of time. During this time, the article 3 is observed from various directions, and other considerations are performed. If the brightness distribution data 522 reflecting the content captured by the external camera 108 is used instead of the fixed brightness distribution data 522 whose content does not change during the process, the change in lighting in the real space 9 and, for example, the presence and movement of another operator on the opposite side are also reflected in the evaluation of the article 3. Since the lighting conditions for the article 3 and the reflection of its surface are also updated in the article image 44, updating at least a portion of the brightness distribution data 522 based on the contents of the image captured by the external camera 108 helps to enhance the realism of the article 3 and immerse the operator 2 in the evaluation task.
[0029] 5A-B and 6A-B are explanatory diagrams (FIGS. 5A, 6A) for explaining the operation of the external camera in the design support system according to the embodiment of the present disclosure, and displays (FIGS. 5B, 6B) showing examples of the images captured. In the pre-capture, the brightness distribution data is acquired, for example, from the celestial sphere, which is the range shown by the wire grid in FIG. 5A and FIG. 6A. FIG. 5B shows the interior state acquired in the pre-capture in the form of a panoramic development. In this way, the operator 2 captures the image using the external camera 108, and the brightness distribution data of the celestial sphere shown in FIG. 5A can be acquired, and the details are acquired as shown in FIG. 5B. By adopting image-based lighting (IBL) using the brightness distribution data exemplified in FIG. 5B as lighting and adopting a drawing method that accurately reflects the reflectivity of the article 3, the appearance of the article 3 in the virtual space can be reproduced with high realism by matching it as much as possible to the ambient light environment of the real space 9. Although some of the drawings in this application are displayed in gray in accordance with the specifications of the patent application format, they can be reproduced or photographed including colors according to human color vision in the design support system 1. Also, some of the drawings in this application are shown in a normal brightness range even when there is data that can cover a wide brightness range (latitude, dynamic range), such as an HDR image.
[0030] In FIG. 6A, the range captured by the external camera 108 is clearly indicated in a part of the wire grid. The external camera 108 is fixed to the headset 10, and although its field of view has an angle of view that covers the human field of view to a certain extent, the angle of view is only a part of the celestial sphere. In FIG. 6A and FIG. 6B, the contour of the field of view that can be captured by the external camera 108 (camera angle of view 48) is displayed by a solid line. As is clear from a comparison between FIG. 5B and FIG. 6B, the range that can be captured by the external camera 108 is only a part of the celestial sphere, but a change within the camera angle of view 48 (presence or absence of a person in FIG. 5B and FIG. 6B) can be significant for the operator 2. This is because, in a scene where the design of the article 3 is evaluated using the headset 10, the camera angle of view 48 is a position that comes into view through the article 3 being observed. It is very meaningful to display a position that may be imaged near the fovea centralis of the retina in human vision as a pass-through by the external camera 108. In this way, it is extremely useful to appropriately reflect the situation of the real space 9 at any given time, which is photographed using the external camera 108.
[0031] 1-4. Ranging, hand, foreground When the design support system 1 realizes a highly realistic reproduction of the object 3, the operator 2 tends to feel uncomfortable when his / her body (e.g., hands) are not included in the presented image. For example, when operating a controller, which is common in VR technology and MR technology, with a hand, it is difficult to operate if the controller or hand cannot be seen. In particular, when a headset 10 that does not have an optically transparent part is adopted in the design support system 1 of the present disclosure, it is desirable to add visual information about the outside world. Therefore, in a preferred configuration of the design support system 1, a depth ranging unit 130 and a foreground image rendering unit 570 are further included. The depth ranging unit 130 can measure the distance to an object in each direction in the real space. This can be realized by installing a LiDAR (Light Detection And Ranging) optical element (not shown) at the position of the outside world camera 108 or by a hand tracking camera. For a direction in which the distance to the object indicated by the depth measuring unit 130 is smaller than the distance to the article in the virtual space, the foreground image rendering unit 570 generates a foreground image 46 based on the output of the external camera 108 or the hand tracking camera 108A. In this case, the image integration unit 580 additionally combines the foreground image 46 with the background image 42 and the article image 44 and outputs an image signal for the presented image, so that the presented image can include a captured image of an object located in front of the article, such as a controller or a hand.
[0032] 7A-C are explanatory diagrams (FIG. 7A) showing a hardware configuration of a design support system according to an embodiment of the present disclosure, and (FIGS. 7B and 7C) showing a state in which a background image, an article image, and a foreground image are synthesized in an operation of generating a presentation image for both the left and right eyes. As shown in FIG. 7A, one computer 50 is employed for an operator (not shown in FIG. 7A), and the computer 50 generates presentation images 4L and 4R through a display 102 for both the left and right eyes as shown in FIG. 7C. The operator can also perceive depth by the presentation images 4L and 4R that correctly reflect the binocular parallax. The presentation images 4L and 4R for the left and right eyes in FIG. 7C are transmitted to the headset 10 through transceivers 52L and 52R in the computer 50 in FIG. 7A using independent channels for the left and right. The transceivers 52L and 52R relay transmission and reception between an appropriate input / output interface (e.g., USB-C3.0 / 3.1, Display Port) of the computer 50 and the headset 10. At this time, the image signals or data from the external camera 108 equipped on the headset 10 are also sent to the computer 50 separately for the left and right. The attitude (direction, position) of the headset 10 in the real space 9 is detected by the attitude sensors 104 and 106. The attitude sensors 104 and 106 can be a combination of a direction sensor such as a gyroscope or a magnetic sensor and a position sensor that detects the position in Cartesian coordinates using infrared rays or a laser. If necessary, the attitude sensors 104 and 106 are provided with a ground element (attitude sensor ground element 106) to ensure numerical accuracy. The background image 42 is the brightness distribution data acquired in the pre-shooting or an updated version of the same, and is photographed by the external camera 108. The item image 44 includes an image of the item 3 calculated using the brightness distribution data 522 as lighting for image-based lighting. The foreground image 46 is an image of an object that should be in front of the item 3. An explanatory diagram (FIGS. 7B and 7C) showing how the background image, the item image, and the foreground image are synthesized. This compositing can be performed by known image compositing methods, such as alpha blending, Z-buffering, etc.Parts of the article image other than the article 3 and parts of the foreground image 46 other than objects that should be in front of the article 3 are composited through transparency. By performing such composition independently for each of the left and right, it is possible to provide a highly realistic presented image in which the article 3 can be perceived as if it were a model placed in real space.
[0033] 10C actually shows the arm and hand of the operator 2 reaching out to the object 3 as a foreground image 46. Although it is impossible for the operator 2 to actually touch the object 3, when the operator 2's arm and hand are included in the presented image 4 on the display 102, the operator 2 feels a high sense of reality toward the object 3.
[0034] As shown in Figs. 6A and 6B, due to limitations in the range of the human visual field, the range that should be presented to the operator 2 as a presentation image through the display 102 at any given time is a part of the range of the celestial sphere. For this reason, in order to enhance the sense of reality as much as possible, it is also preferable to display a real-time image from the external camera 108 for the background image 42. On the other hand, as for the brightness distribution data for the lighting process of the article 3 included in the article image 44, the range of the substantially celestial sphere contributes greatly to enhancing the sense of reality due to reflection from the surface of the article 3. Therefore, it may be essential to use the brightness distribution data acquired in the pre-shooting S02 (Fig. 4) and to keep the update of the data to a minimum so that data over a wide range of directions can be used.
[0035] 1-5. Multiple Operators This disclosure also provides a design support system that is used simultaneously by multiple operators. FIG. 8 is an explanatory diagram showing the configuration of the design support system 1M. The design support system 1M is equipped with a set of a headset 10, a display 102, an external camera 108, and a computer 50 corresponding to each operator, and the sets are connected as many as the number of operators required. Each set can also operate as a standalone. For this reason, the ambient light recording unit 520, the background image rendering unit 530, the item image rendering unit 550, and the image integration unit 580 are implemented in each computer 50 (see FIG. 2; not shown in FIG. 8). For this reason, the ambient light recording unit 520 for each operator records brightness distribution data indicating the brightness in each direction around each operator 2 in association with the posture of the headset. Each background image rendering unit 530 generates a background image 42 corresponding to the output of the posture sensor in the real space of each operator at each time point. Each article image rendering unit 550 generates an article image 44 under illumination corresponding to the brightness distribution data of each operator, which corresponds to the output of the posture sensors 104, 106 of each operator at each time point. Each image integration unit synthesizes the background image 42 and the article image 44 to output an image signal for the presented image 4 for each operator. In addition, as shown in FIG. 4, it is preferable that brightness distribution data 522 indicating the brightness in each direction around each operator photographed by the external camera 108 of each operator 2 while changing the orientation of the headset 10 in the pre-photographing S02 is stored in each environmental light recording unit 520. In this way, the presented image 4 according to the real environment of each operator is displayed on the display 102 of the headset 10 worn by each operator. In this way, in this embodiment, even if multiple operators are present in different real spaces, each operator can perform design evaluation of the article 3 reflecting his or her own environment.
[0036] In addition, in a design support system used simultaneously by multiple operators, it is also preferable to configure the system so that one operator generates a background image and an article image based on the brightness distribution data of the other operators. Specifically, an ambient light recording unit for one of the multiple operators records brightness distribution data for the other operators. The background image rendering unit and the article image rendering unit for the one operator use the brightness distribution data for the other operators stored in the ambient light recording unit to generate a background image and an article image. With this configuration, the other operator can use the brightness distribution data acquired by the pre-shooting to generate the background image and the article image of the one operator. In other words, since the background image and the article image observed by the other operator are also observed by the one operator, it is possible to match the ambient light environment that affects the article evaluated by both operators with the appearance of the article. Even in this case, each operator can feel a high sense of reality for the article he or she is observing. In this configuration, since the brightness distribution data is not from the actual environment for the one operator, the situation is substantially closer to VR (virtual reality) than MR. In collaborative design evaluation work, if operators can share the appearance of the items being evaluated with each other, it is useful in deepening mutual understanding of each other's evaluations.
[0037] In addition, when such a configuration is adopted to make the appearance of the article consistent, the update process may also be performed across operators. Specifically, it is preferable that a background update processing unit is provided for one operator, and the operation of the background update processing unit targets the brightness distribution data for the other operator stored in the ambient light recording unit for the one operator. In other words, data from the brightness distribution data that corresponds to at least a part of the field of view of the one operator wearing the headset is replaced with brightness data captured by an external camera of the other operator. Since multiple operators do not necessarily face the corresponding directions at the same time, the update is not necessarily performed with the latest brightness data. However, if the update of the brightness distribution data by the external camera of the other operator is reflected in the evaluation of the design of the article by the one operator, the collaborative work of design evaluation can be smoothly performed.
[0038] On the other hand, in the case of supporting a collaborative work by a plurality of operators, the design support system 1M can be implemented, for example, by securing the above-mentioned item data recording unit 510 and the ambient light recording unit 520 in a data server 50S capable of communicating with each computer 50, and distributing the background image rendering unit 530, the background update processing unit 532, the image adjustment unit 540, the item image rendering unit 550, and the foreground image rendering unit 570 (none of which are shown in FIG. 8) to each computer 50. In this way, even when the heads of the operators are in different positions, it is possible to perform design consideration for a common model while showing appropriate presented images to each operator. Furthermore, each operator may perform operation input by himself / herself through the operation input unit 120, and also possible to present a presented image according to the operation input by any operator to the other operators. In particular, when a plurality of operators perform collaborative work, an operator may identify and show or highlight a part that should be noted in order to explain to the other operators. For such purposes, it is also preferable that the operator who gives the instruction can operate local lighting corresponding to a pointer or a flashlight in the lighting in the virtual space. For example, the operation input unit 120 held by the operator 2 can be equipped with a function for detecting a direction like a pointer. In this way, it is preferable that the design support system 1, 1M is equipped with a function for pointing to an arbitrary position and a function for operating local lighting.
[0039] The design support system 1M preferably includes a plurality of background update processors 532 for each of the plurality of operators. Each of the background update processors 532 replaces data corresponding to at least a part of the field of view of each operator with brightness data at each time point acquired by photographing with an external camera. In this way, each operator can observe the background image 42 and the item image 44 based on the brightness distribution data updated corresponding to his / her own real space 9, and a high sense of reality is realized for the plurality of operators. The design support system 1M also employs the depth distance measuring unit 130, so that the foreground image 46 can be displayed superimposed on the background image 42 and the item image 44.
[0040] FIG. 9 is an explanatory diagram showing the hardware configuration of the design support system 1M according to the embodiment of the present disclosure, in which a headset 10 using a computer 50 is used for each of a plurality of operators 2. As shown in FIG. 8, typically, one computer 50 is used for each operator 2, and the computer 50 generates a presentation image through a display 102 for both the left and right eyes. In FIG. 9, for two operators (Operator A, B), one set of a headset 10 and a computer 50 constituting the design support system 1M according to the embodiment of the present disclosure is used in the real space 9A, 9B in which each of the operators is currently present. The devices shown in FIG. 9 are connected by any network such as a LAN (local area network) or a WAN (wide area network). However, such a correspondence between the computer device and the operator can be changed depending on the processing performance of the device to be used and the required quality. If it is possible to support the design evaluation of an article by a plurality of operators, evaluation based on various viewpoints can be easily performed, and the efficiency of the design evaluation can be improved.
[0041] From a practical standpoint, it is also useful to simply record the image presented to one of the operators 2 (presented to both eyes or to one eye) as a video image or to display it on an external display device. The operator of the design support system 1M is immersed in the evaluation of the item 3, and having this recorded or monitored by a monitor device has the advantage of providing an objective record and making it possible to check whether the intended evaluation is being carried out. At this time, it is also useful to record points of interest, which will be described later.
[0042] 2. Working Example Here, an embodiment of the design support system 1 of the present disclosure, which implements the above-mentioned features and other features, will be described. The embodiment of the design support system 1 of the present disclosure is realized by the following hardware configuration. Headset, transceiver, and attitude sensor: Varjo XR-3 or Varjo XR-3 Focal Edition (Varjo Technologies Oy, Helsinki, Finland) PC: HP Inc., Windows 11 CPU: Core i7-13700K (Intel) Memory: 32GB Graphics card: GeForce RTX 4070 (NVIDIA) Software: Unreal Engine (Epic Games), OpenXR (Khronos Group) (Company names and products may include trademarks of their respective companies.) ·Item: Four-wheeled vehicle design example 10A-C and 11 are capture images of the display of the headset in the example of the design support system of the embodiment of the present disclosure, and are snapshots of images recorded for both the left and right eyes in the example system in which the design support system 1 shown in FIG. 2 is actually constructed. These figures are images presented to the operator 2 by the display 102, and since the headset 10 does not have an optically transparent part, the operator 2 sees only such electronically generated images through his eyes. In this example, a car is designed as the item 3. The real space 9 is the interior of the room as shown in FIG. 5B. The update process for the background image 42 uses the shooting signal from the external camera 108 in a pass-through manner. The brightness distribution data is the omnidirectional image captured in advance by the operator 2 who actually wears the external camera 108 and recorded in the ambient light recording unit 520, and the item 3 in the item image 44 is realized in the virtual space by using image-based lighting that utilizes the brightness distribution data. The virtual space and real space for the article 3 were matched by using a calibration marker (two-dimensional code) placed on the floor of the real space to match the coordinates. The processing for the image integration unit 580 employs alpha blending so that the background of the article image 44 is appropriately blended with the light transmitting parts (such as car windows) of the article 3 in the article image 44.
[0043] In Figs. 10A-C, the interior of a room is the real space 9, and a relatively large automobile is placed there as the object 3. The operator 2 perceives the object 3 as if it were placed so close that he or she could reach out and touch it. This is because, in addition to the three-dimensional effect due to stereoscopic vision reflecting binocular parallax, the exterior of the automobile generated by image-based lighting in the virtual space is reproduced with a high sense of reality. For example, in Fig. 10A, the interior lighting placed on the ceiling in the real space 9 is reflected on the surface of the windshield of the automobile as a reflection of the specular reflectivity of the glass surface while reflecting its curvature. In addition, when looking at the surface of the hood of the automobile, the texture of the metallic paint, which is a combination of surface reflection and diffuse reflection, is expressed with a high sense of reality, including the reflection due to surface reflection and the change in the brightness of the surface reflecting the direction of the light source. When viewed stereoscopically with both the left and right eyes, the reflected light source can be perceived to be in a different position in the depth direction from the surface of the windshield or the hood. As a result, the operator 2 can evaluate the design as if the car or its model were placed in the real space 9. Similar items 3 are shown in Figs. 10B and 10C. The operator 2 can adjust the viewpoint within the range of the real space 9. For example, the operator 2 can walk and move by himself, or change the height of his head from the floor. He can also tilt his head to look in, look up, or face in various directions. These orientations and positions (postures) can be taken. In addition to the high realism of the appearance of the item 3, the operator 2 can perform the design evaluation of the item 3 while adjusting the viewpoint with extremely natural movements. However, there is room for further improvement in the realism. The shadow that the car should cast in the real space, which is not represented in Figs. 10B and 10C, is reproduced in this embodiment system by adding and implementing a function to reduce the brightness of the background image 42 in the direction of the shadow that the item 3 should cast in the real space 9 in response to the brightness distribution data. Other features shown in Figs. 10B and 10C will be described later.
[0044] 11A and 11B show the same four-wheeled automobile model as shown in FIGS. 10A-C, before and after the body color is changed, observed from almost the same posture (position, orientation). Although it is displayed in gray due to stylistic constraints, in the embodiment system, the display 102 shows the before and after of changing a high-brightness silver metallic paint (FIG. 11A) to a low-brightness red metallic paint (FIG. 11B). As shown in FIGS. 11A and 11B, the article 3 not only has a high sense of reality, but also allows the attribute (body color) to be easily changed in the virtual space. From the comparison between FIGS. 11A and 11B, not only the difference in hue and saturation that is not expressed in the drawing, but also the difference in the relative strength of surface reflection due to the difference in brightness and the expression of the seal impression have changed, making it possible to evaluate the design with an extremely high sense of reality. Although the color change by metallic paint is exemplified here, if various types of paint such as matte, solid color, candy paint, and other types of paint, as well as patterns and color divisions other than paint, can be selected, it is possible to reproduce the display of any surface that can be reproduced in the virtual space. These adjusted and selected attribute candidate data are stored in the item data recording unit 510, and therefore can be implemented simply by changing the software and data.
[0045] The principle of the lighting was confirmed by using an object with a simpler shape. Figures 12A-B are capture images of the display of a headset in an example of the design support system 1 according to the embodiment of the present disclosure, showing the specular reflection (Figure 12A) and diffuse reflection (Figure 12B) of an object 3S displayed as a sphere. After the preliminary photographing in step S02 of Figure 4, only the reflection characteristics of the object 3S are changed from specular reflection to diffuse reflection, so the real space 9 is the same and the brightness distribution data is also the same. In the specular reflection of Figure 12A, the brightness of the real space 9 reflected on the surface of the object 3S is clearly observed. This specular reflection is an example in which a surface that has both high reflectivity and smoothness, such as a metal surface, is set. In contrast, in the diffuse reflection of Figure 12B, each part of the surface of the object 3S is displayed only as a surface having a brightness according to the illuminance with which the lighting according to the brightness distribution data illuminates each part. This is an example in which a bright diffuse reflection surface such as white or gray is set on the surface. 12A and 12B, the background image 42 is a video pass-through image of the external camera 108, but the brightness distribution data serving as the illumination for rendering the item 3S is captured in advance.
[0046] Through the above examples, it has been confirmed that the design support system 1 provides a high level of realism that can assist the operator 2 in evaluating designs by fusing the real space 9 with the virtual space, and that the expressions that can be shown by the objects reproduced in the virtual space are sufficiently diverse.
[0047] 3. Function The following will further explain in more detail the functions of the design support system 1 of the present disclosure. 3-1. Transformation and grouping of parts In the design support system 1, the advantage of using a virtual space for designing the article 3 is utilized. One of these is a transformation function that makes it easier for the operator 2 to intuitively transform the article 3. FIGS. 13A to 13C are explanatory diagrams showing the operation of the transformation function in the design support system of the embodiment of the present disclosure. When using this function, a bounding volume or a bounding box 7 that contains the article 3 is set by the bounding box setting unit 560 (FIG. 2). The bounding box 7 is divided into a lattice shape, and has a plurality of grip points 72 at lattice points on the surface that the operator 2 can grip by operating the operation input unit 120. As shown in FIG. 13B, the bounding box 7 is, for example, a rectangular prism that contains the article 3 before transformation. The grip points 72 are set, for example, at internal division points that divide each side of the rectangular prism into three equal parts, and at the intersection points of a plurality of straight lines that connect each internal division point to each other on the surface of the rectangular prism. In the bounding box 7, when the operator 2 performs an operation such as dragging to displace any of the grip points 72 in a gripped state using the operation input unit 120, the grip points 72 are displaced accordingly, and, for example, the grip points 72 that are not gripped are not displaced. At this time, the spatial coordinates between the multiple grip points 72 can be enlarged or reduced while maintaining the internal division ratio. As a result, the coordinates inside the bounding box 7 are also displaced (FIG. 13C). Such displacement of the coordinates is calculated by the coordinate displacement unit 562 (FIG. 2). The coordinates referred to by the structure data 512 are displaced in response to the displacement by the coordinate displacement unit 562. In this way, the structure of the article 3 that refers to at least some of the coordinates inside the bounding box 7 in the virtual space is deformed. This deformation is executed by the structure deformation unit 564 (FIG. 2). By having the item image rendering unit 550 display the structural data called from the item data recording unit 510 in correspondence with the coordinates displaced by the structural deformation unit 564, an item image having a deformed structure like item 3M in Figure 13C is generated.
[0048] With this function, the operator 2 can transform the object 3 in the virtual space with a sensation as if he or she were touching a clay model directly. In this case, the appearance of the object 3 shown in the real space 9 can be reproduced with a high degree of reality, and the shape can be changed while perceiving the final appearance, which makes it possible to support advanced design work by taking advantage of the advantages of virtual space.
[0049] In addition, the design support system 1 is also designed to realize the above-mentioned transformation more easily. The transformation function is particularly useful when the object 3 has a structure in which multiple parts are assembled. When the object 3 includes multiple parts (not shown), the object data recording unit 510 stores the part structure data 516 (FIG. 2) for determining the structure of the parts. Then, the structure transformation unit 564 displaces the coordinates referred to by the part structure data 516 for the multiple parts according to the displacement by the coordinate displacement unit 562, thereby grouping and transforming the multiple parts. When the object 3 is an assembled or combined object of multiple parts, not only the transformation of one part but also the transformation of other parts combined with it at the same time not only omits the transformation work for the multiple parts, but also allows the transformation to be performed while appropriately maintaining the relationship between the parts. As a result, the amount of accompanying work required by the transformation, such as the need to reconsider the possibility of assembly according to the transformation, is reduced. In this way, the design support system 1 can support the correction of the object shape affected by the structure in design work.
[0050] 3-2. Measuring distance to objects and displaying cross-sections of objects In the design support system 1, it is also preferable to provide a more detailed support for design evaluation of the article. That is, in the design support system 1, it is also preferable to provide a function for measuring the physical distance on the surface of the article 3 and a function for checking the cross section of each part of the article to support the design evaluation. FIGS. 14A-B are capture images of the display of the headset in the design support system according to the embodiment of the present disclosure, showing a state in which the distance measuring function for measuring the distance is used (FIG. 14A) and a state in which the cross section display function is used (FIG. 14B). The design support system 1 further includes a controller attitude sensor 124 (FIG. 2), a point position determination unit 572, and a distance calculation unit 574. The controller attitude sensor 124 detects the attitude of the operation input unit 120 itself. The point position determination unit 572 determines the focus position based on the intersection between the virtual line in the virtual space determined according to the output of the controller attitude sensor 124 and the surface of the structure. When there are multiple attention positions determined by the point position determination unit 572, the distance in real space corresponding to two of the attention positions, for example the distance between the pointer displays P1 and P2 shown in Fig. 14A, is calculated by the distance calculation unit 574. In order to show the operator 2 which attention positions the distance between which is being measured, the image integration unit 580 includes in the presentation image, when generating an image signal, a pointer display indicating the attention position updated in the virtual space according to the output of the controller attitude sensor 124 and the distance value calculated by the distance calculation unit 574. The pointer displays P1 and P2 and the distance value are shown in Fig. 14A.
[0051] In order to display the cross section of an article, it is preferable that the design support system 1 includes the controller attitude sensor 124 and an intersection line determination unit 576 (FIG. 2). The intersection line determination unit 576 determines an intersection line between a virtual plane in the virtual space determined according to the output of the controller attitude sensor 124 and a surface of the structure. In order to show the operator 2 on which virtual plane the cross section is cut, the image integration unit 580 includes in the presented image, when generating an image signal, a surface display indicating a virtual plane updated in the virtual space according to the output of the controller attitude sensor 124 and a display of a cut surface with the intersection line determined by the intersection line determination unit 576 as a contour or cut line. FIG. 14 shows a plane having a grid, which is the surface display, and the contour.
[0052] 3-4. Background other than the real world Although design evaluation such as that in the design support system 1 has the advantage of being able to perform design evaluation with a high sense of reality, there is also a need to use a virtual space rather than the real space 9. In order to meet such needs, in the design support system 1 of the present disclosure, it is preferable that the brightness distribution data that gives the article image 44 can be alternatively displayed by something other than the real space where the operator 2 is located, such as brightness distribution data by direction constructed for a virtual space, a surrounding environment image that has been photographed in advance, or a surrounding environment image that has been photographed at a facility or location and transmitted through a network. In this case, the brightness distribution data is rewritten by such an alternative. This allows the operator 2 to evaluate the exterior design of the article 3, for example, in VR (virtual space) or a surrounding environment image used as a reference, thereby suppressing the variation in evaluation and reconfirming his / her own evaluation scale. FIGS. 15A-B are capture images of the display of a headset in the design support system of the embodiment of the present disclosure, and show an example of displaying an image of the real space 9 from an external camera for the background image and an example of displaying an image of the background virtual space 900. Lighting is also arranged in the background virtual space 900, and its reflection is reproduced on the glass and painted surfaces of the automobile, which is the object 3.
[0053] 3-5.Network It is also preferable that the design support system 1 of the present disclosure is configured to utilize a computer network. As described in relation to FIG. 8, design evaluation by multiple operators is itself preferable. Furthermore, it is also preferable to take advantage of the advantages of a computer network to support multiple operators 2, who are geographically distant from each other, working together via the network. FIG. 16 is an explanatory diagram showing an example of the configuration of an embodiment of the present disclosure in a situation where multiple operators work together via a network.
[0054] In the present disclosure, a design support system 1N for use via a network is provided. The design support system 1N is an improved version of the system shown in the design support system 1M (FIG. 8). The system further includes a plurality of image adjustment units for each of a plurality of operators, and the image adjustment unit for one of the plurality of operators generates attribute data for the selected or adjusted attribute from the attribute candidate data in response to the instruction content of the operation input for selecting or adjusting the attribute of the article received by the operation input unit for the other of the plurality of operators. In a typical configuration, a computer device used by an administrator who is one of the plurality of operators includes an account management unit and a session management unit, and the account management unit includes a permission table that determines which functions and data files of the design support system each user can access, and the session management unit holds a list of users who can participate in the session as a session table, and determines whether each of the plurality of operators is a legitimate user by referring to the session table, and establishes a connection for a session for the legitimate user of the plurality of operators. In addition, in this typical configuration, it is preferable that the computer equipment used by the user acts as a client terminal when the computer equipment used by an operator acts as a server device, and is capable of communicating over the network, and that a user ID for identifying the user is used to establish a session over the network.
[0055] In order to implement and function the design support system 1N as a program in a computer device such as a personal computer (PC), each operator operates an execution program to realize the system in his / her own PC. Thus, each PC 50A-50C becomes a part of the design support system 1N through a network 68. Specifically, operator A operates a server execution unit 60, which may be an execution program in a computing device, in the PC 50A that he / she operates. Similarly, operators B and C operate client execution units 66, which may be an execution program in a computing device, in the PCs 50B and 50C that he / she operates. Each operator's PC 50A-50C can communicate with each other using an appropriate protocol (e.g., TCP-IP) via a public line such as the Internet or a dedicated line.
[0056] The client execution unit 66 operates the PCs 50A-50C used by the operators A-C as client terminals in a specific design review session (hereinafter referred to as a "session") by the design review network community. In contrast, the server execution unit 60 operates the PC 50A used by the operator A as a server device in the session. The function as a server device is realized by implementing an account management unit 602 and a session management unit 604. The server execution unit 60 is typically implemented as a function of a server execution program, and the account management unit 602 and the session management unit 604 are implemented as its functions. Operator A executes the server execution unit 60 in order to set the functions of the design support system 1N as an administrator.
[0057] The account management unit 602 implements a function of managing the accounts of users who can access the necessary resources of the design support system 1N. For example, the resources of the design support system 1N are typically functions and data files used in the design support system 1N, and the function of managing the accounts can include setting the use authority and access rights (permissions) for these resources. A permission table (not shown) records the permission level of each resource for each user account, and the account management unit 602 refers to the permission table to determine, for example, whether an operator who logs in is a user who has an account in the design support system 1N. Specifically, operators B and C who operate their own PCs 50B and C functioning as client terminals can log in with their own accounts by means of password authentication or the like, and can further access resources within the range permitted by the permissions of functions and files set in advance. The files are stored in, for example, the data server 50S. The account management unit 602 refers to the permission table to perform such login management and permission management for functions and data files. For example, operators A to C are given unique user IDs, and permissions for functions and data files are managed for each user ID in a permission table.
[0058] The session management unit 604 controls whether or not a user will participate in a session. By referencing a session table (not shown), which is a list that records which users will participate in each session, the session management unit 604 determines whether a connection request from the client execution unit 66 and subsequent communications are valid, and controls participation in the session. In this way, when operators A to C are valid users in a session, they can participate in a session in a design review network community whose members have been determined in advance.
[0059] The user identification performed by the account management unit 602 and the session management unit 604 is typically performed by a user ID corresponding to the account, and for security purposes, security measures used by those skilled in the art, such as a password, are also adopted.
[0060] The computers 50A-50C and the data server 50S can be equipped with a function that allows the operators A-C to manage settings for the functions and files of the design support system 1N. The settings for these functions and files include, for example, a color and an avatar for distinguishing oneself from others in collaborative work. The color and avatar are associated with a user ID as necessary and used for display in an appropriate situation. This allows the operators A-C to wear the VR headset 10 and view the activities of others in the virtual space, for example by their user ID.
[0061] In the design support system 1N using the computer network described above, operators 2 in remote locations can simultaneously perform evaluation of a common item 3 from the viewpoint of each operator 2. When the item 3 is a product such as an automobile that is mass-produced on a large scale and can recoup investment by being shipped to various parts of the world, a process of carefully determining the design from various viewpoints is essential. By adopting the design support system 1N in such an application, the efficiency of design evaluation can be improved. In addition, in FIG. 16, the participants are operators 2, and all of them wear headsets 10 to participate in the design evaluation. Such operators 2 can perform design reviews and design clinic activities without being conscious of geographical distances. In addition, as shown in FIG. 9, it is not necessarily required that all of the operators wear headsets 10. It is also useful in the embodiment using a computer network to simply record the presented image (presented to both the left and right eyes or to one eye) presented to any of the operators 2 as a video image or present it on an external display device, so that, for example, observers (Observers C, D) present at the observer can observe the progress of the design work while showing the presented image from the viewpoint of one of the operators (Operator A).
[0062] 3-7.Gaze point In the design support system 1 of the present disclosure, it is preferable to detect and record the gaze point. The line of sight of both eyes of the operator 2 is defined as a straight line starting from the position (viewpoint) of the eyeball in real space and ending at the gaze point. This gaze point indicates the position at which the operator 2 focuses at each moment, since it is the position at which an image is formed at the position with the highest resolution of the eyeball. FIG. 17 is an explanatory diagram of a headset 10A equipped with an eye tracking detection unit 110 employed for detecting and recording the gaze point, and FIG. 18 is a block diagram of a design support system 1A employing the headset 10A. For the headset 10A and the design support system 1A, elements indicated by the same reference numerals are the same as those described for the headset 10 and the design support system 1. The headset 10A is equipped with an eye tracking detection unit 110 for identifying the gaze of at least one of the eyes of the operator 2. The eye tracking detection unit 110 is realized by, for example, a combination of an infrared camera and an infrared light source, and identifies the line of sight of the eyeball of the operator 2, that is, the straight line connecting the gaze point and the eyeball, by a method such as the corneal reflex method, the bright pupil method, or the dark pupil method. Such an eye tracking detection unit 110 can also detect blinking and opening and closing of the eyelids. The eye tracking signal 112 output from the eye tracking detection unit 110 is received by the computer 50 via an appropriate interface and input to the gaze point determination unit 582. The gaze point determination unit 582 determines the gaze of the operator 2 in the virtual space using the eye tracking signal 112. The gaze point determination unit 582 uses the gaze to determine the gaze point on the three-dimensional object surface of the article 3, the gaze point in the article image 44, and the gaze point in the background image 42, based on the structure data 512 and the brightness distribution data 522. This gaze point can be determined, for example, by a so-called collision determination algorithm. Another method for determining the gaze point that can be adopted for processing by the gaze point determination unit 582 is a gaze intersection method for determining an intersection point where the left and right gaze directions converge. This gaze point can also be used to update the background image 42. The gaze point position recording unit 584 records data 586 of the gaze point position.The gaze point position data 586 is used for processing by a gaze point counting unit 588 .
[0063] 4. Variations: It is also useful to make various modifications to the above-mentioned design support systems 1, 1N, 1M, and 1A. 4-1. Supplementary images It is useful that the above-mentioned design support system 1, 1N, 1M, 1A further includes an auxiliary image rendering unit 534 (FIG. 2) that generates an auxiliary image. In this case, the image integration unit 580 synthesizes the auxiliary image with the background image 42 and the article image 44 and outputs the result as an image signal for the presented image. The auxiliary image may be various information presentation images or 3D objects. For example, if something like a menu display (illustrated in FIG. T0B, FIG. 11A, B, FIG. 15A, B) can be displayed when the operator 2 performs an operation using the operation input unit 120, the operator 2 can easily select and adjust attribute candidates. In addition, a marker that can be left as a sticky note on the surface of the article 3 in the design evaluation work can be one of these auxiliary images. Furthermore, guide information for guiding the operator 2 in design evaluation, such as informing the viewpoints and points of interest for evaluating the design, is also useful as an auxiliary image. In addition, a plane (FIG. 14B) having a measure display for length measurement and a grid serving as a scale, which is used together with the above-mentioned distance calculation unit 574 and intersection line determination unit 576, can also be an auxiliary image. In addition, when investigating the relationship between the product 3 being designed and the body of the user (driver, passenger), such as an automobile, auxiliary images such as 3D objects of personas (mannequins) are also useful. From a different perspective, when an operator 2 is performing a design evaluation, the display of images or objects for creating a sense of realism, for example, recreating rain or snow conditions or showing falling leaves, can also serve as auxiliary images. In this way, auxiliary images can be of great use in design evaluation activities.
[0064] The auxiliary image may be generated by an operation using the operation input unit 120, or may be displayed as a prompt, or may be displayed when a predetermined condition is met. For example, when the gaze point matches a feature of an article to which an intentional commentary is to be added by eye tracking, it is useful to automatically display an explanatory image or text based on the match. Furthermore, it is also useful to configure the display of an auxiliary image when the attitude sensor mover 104 and the attitude sensor stator 106 detect the motion of the operator 2 tilting the head 22.
[0065] One of the viewpoints for design evaluation that may be specific to the case where the article 3 is an automobile is the beauty angle. The beauty angle is a viewpoint (observation position) that matches the design intention, specified as a relative angle with respect to the article 3. In other words, when a designer has already completed the design of the article 3, the designer often envisions a viewpoint for evaluation as a relative angle with respect to the article 3, and may specify that angle as the beauty angle. In such a case, it is also useful to present information as a guide to make it easier for the operator 2 to specify the viewpoint position as an auxiliary image.
[0066] Furthermore, it is also useful to display the trajectory of the movement of the operator 2 using, for example, a spline curve and record it. The viewpoint from which the operator 2 actually observed the item 3 for the design evaluation can be an indicator of whether the design evaluation by the operator 2 is reliable or can be a clue for verifying the evaluation results after the fact, and therefore can be reference information for giving a post-facto interpretation to the evaluation results.
[0067] Furthermore, it is also useful to generate a person (persona) object by computer graphics in a virtual space for the operator 2 and guide the design work as some kind of character. For example, by using a technology such as lip sync, a person who speaks to the operator 2 performing the design work from outside the design support system 1 to assist the design work can be presented, so that the design work can be carried out smoothly. In addition, when multiple operators 2 are evaluating the item 3 as in the design support systems 1M and 1N, it is also useful to assist the operators 2 in interacting with each other by voice or the like. In that case, too, by showing the personas to each other, even if only one operator 2 is present in the real space, the other operator 2 participating in the evaluation of the item 3 can be recognized in a very natural manner. In design evaluation, if multiple evaluators can proceed with collaborative work in a natural way, the design evaluation work of the operator 2 can be greatly supported. In addition, when multiple operators 2 participate, a simpler symbol display may be sufficient to recognize the other operators 2. For example, even if only the headset (such as an HMD) is generated using computer graphics, if its posture (direction, position) is synchronized with that of the headset 10 currently worn by another operator 2, the operator who can see it can easily recognize the other operator.
[0068] 4-2. Dialogue support In the design support system 1, 1M, 1N, 1A, a plurality of operators 2 participate, or a participant other than the operator 2 communicates with the operator 2 through oral communication or text, or an artificial intelligence technology is incorporated to communicate with a computer, which is an auxiliary function that is compatible with the immersion of the operator 2 in the design work. Such an auxiliary function for dialogue is typically a voice chat function or a translation function for assisting the dialogue. The voice chat function can be implemented by equipping the headset 10 with a speaker or earphone for emitting the other person's voice and a microphone for picking up one's own speech, and implementing voice communication through the computer 50 or a computer network using any protocol. In addition, the translation function can be implemented by adopting a similar headset 10 and further combining a voice recognition engine, a translation engine using any method such as deep learning, and a voice engine.
[0069] 4-3. Physical condition measurement In order to detect and record non-verbal psychological changes of the operator 2, it is effective to obtain body / brain waves, pulse, skin potential, electrocardiogram, etc. from the operator 2 who is performing the design evaluation. For this purpose, the design support system 1, 1M, 1N, 1A can add a biological condition sensor 140 as shown in FIG. 2. A signal (biovital signal) from the biological condition sensor 140 directly reflects the physical condition of the operator 2 and is transmitted to the computer 50 via a signal processing unit 142. The computer 50 stores biological information data 592 corresponding to the signal from the biological condition sensor 140 in the form of time-series data associated with, for example, an appropriate time stamp in the subject information recording unit 590. This biological information data 592 can include brain waves, electrocardiogram, myoelectric potential, etc. By analyzing these signals in association with, for example, gaze point position data 586 recorded in gaze point position recording unit 584, it is possible to investigate retrospectively what psychological state the operator 2 was in with respect to which part of the item 3, which is useful for collecting non-verbal information in design evaluation and judging the reliability of the evaluation.
[0070] 4-4. Program The hardware configuration of the above-mentioned design support systems 1, 1M, 1N, 1A, excluding the headset 10 and its associated sensors and controllers (operation input unit 120, attitude sensors 104, 106) (hereinafter referred to as headset-related devices), is a computer 50, that is, a personal computer or workstation of a general configuration with a certain level of graphic performance. Therefore, once the headset-related devices and a suitable computer device are prepared and network devices are configured as necessary, the rest can be constructed by using a computer program for configuring the design support systems 1, 1M, 1N, 1A. Therefore, the above-mentioned design support system 1 can also be realized by running a computer program for this purpose.
[0071] The embodiments of the present disclosure have been specifically described above. The above-mentioned embodiments, modifications, and examples are described to explain the invention disclosed in this application, and the scope of the invention of this application should be determined based on the description of the claims. Modifications that exist within the scope of the present disclosure, including other combinations of the embodiments, are also included in the scope of the claims. [Explanation of symbols]
[0072] 1, 1M, 1N Design Support System 10, 10A Headset 102 Display 104 Attitude Sensor Movement 106 Attitude sensor stator 108 External Camera 108A Hand Tracking Camera 110 Eye tracking detector 112 Eye Tracking Signal 120 Controller (operation input section) 124 Controller Attitude Sensor 130 Depth measurement unit 140 Biological condition sensor (biological condition detection device) 142 Signal Processing Section 2 Operator 3, 3M, 3S goods 4, 4L, 4R Presentation images 42 Background Images 44 Product Images 46 Foreground Images 48 Camera angle of view 50 Computers 50A~C Personal Computer 52L, 52R transceiver 510 Item Data Recording Department 512 Structural Data 514 Attribute candidate data 516 Parts Structure Data 520 Ambient Light Recording Unit 522 Lightness distribution data 530 Background Image Rendering Section 532 Background update processing section 534 Auxiliary Image Rendering Unit 540 Image adjustment section 542 Attribute Data 550 Product Image Rendering Department 560 Bounding box setting section 562 Coordinate Displacement Section 564 Structural Deformation Section 570 Foreground Image Rendering Unit 572 Point Position Determination Unit 574 Distance Calculation Unit 576 Intersection line determination section 580 Image Integration Department 582 Gaze point determination unit 584 Point of interest position recording unit 586 gaze point position data 588 Eye gaze counting section 590 Subject Information Record Section 592 Biometric Data 60 Server execution unit 66 Client execution unit 602 Account Management Department 604 Session Management Unit 68 Network 7 Bounding Box 72 Grip Points 9, 9A, 9B Real Space 92 Lighting 900 Virtual space for background
Claims
1. an item data recording unit for storing structure data for determining a structure of at least one item in a virtual space and attribute candidate data for selecting or adjusting attributes of the item; At least one headset to be worn on the head of an operator, the headset including: a posture sensor capable of detecting a posture of the headset in a real space; at least one display capable of displaying an image to be presented to the operator; and at least one external camera capable of capturing an image of at least a part of the external world corresponding to a visual field of the operator; an environmental light recording unit that records brightness distribution data indicating brightness in each direction around the operator captured by the external camera in a pre-capture in association with an output of the posture sensor; a background image rendering unit that generates a background image corresponding to an output of the posture sensor at each time point from the brightness distribution data retrieved from the ambient light recording unit; at least one operation input unit that receives an operation input from the operator for selecting or adjusting the attribute of the article; an image adjustment unit that generates attribute data for the attribute selected or adjusted from the attribute candidate data in response to the instruction content of the operation input received by the operation input unit; an article image rendering unit that generates an article image under illumination corresponding to the brightness distribution data, which corresponds to the instruction content of the operation input and the output of the attitude sensor at each time point, based on the structure data retrieved from the article data recording unit, the attribute data from the image adjustment unit, and the brightness distribution data retrieved from the ambient light recording unit; an image integration unit that combines the background image and the item image and outputs the combined image as an image signal for the presentation image; A design support system equipped with:
2. The brightness distribution data indicates brightness in each direction obtained by photographing with the external camera while changing the orientation of the headset in the pre-photographing. The design support system according to claim 1 .
3. The headset further includes a background update processing unit that replaces data corresponding to at least a part of the field of view of the operator wearing the headset, among the luminance distribution data, with luminance data at each point in time captured by the external camera. The design support system according to claim 1 .
4. The article image rendering unit reflects a surrounding reflection image corresponding to the brightness distribution data in a surface reflection of the article image. The design support system according to claim 1 .
5. The background image rendering unit retrieves the structure data from the item data recording unit, and reduces the brightness of the background image in a direction of a shadow that the item should cast in the real space in accordance with the brightness distribution data. The design support system according to claim 1 .
6. the environmental light recording unit records color lightness data indicating lightness of each color captured by the external camera in the pre-capture in association with the output of the attitude sensor; the background image rendering unit generates a color background image corresponding to the output of the posture sensor at each time point from the color brightness data retrieved from the ambient light recording unit; The article image rendering unit generates an article image under illumination corresponding to the color brightness data, which corresponds to the instruction content of the operation input and the output of the attitude sensor at each time point, from the structure data called from the article data recording unit, the attribute data from the image adjustment unit, and the color brightness data called from the environmental light recording unit. The design support system according to claim 1 .
7. a depth measuring unit capable of measuring a distance to an object in each direction in the real space; a foreground image rendering unit that generates a foreground image based on an output of the external camera in a direction in which the distance to the object indicated by the depth measuring unit is smaller than the distance to the item in the virtual space; It also has The image integration unit additionally integrates the foreground image with the background image and the item image, and outputs an image signal for the presentation image. The article design support system according to claim 1 .
8. the item data recording unit stores the structure data and the attribute candidate data for each of a plurality of items, the operation input unit receives, from the operator, an operation input for selecting or adjusting the attributes of the plurality of articles and a display instruction input regarding whether or not each article needs to be displayed or a display mode of each article; The image adjustment unit generates attribute data for the attribute selected or adjusted from the attribute candidate data for an item to be displayed among the plurality of items in accordance with the operation input and the display instruction input received by the operation input unit. The article image rendering unit generates, for the plurality of articles, article images under illumination corresponding to the brightness distribution data, which corresponds to the instruction content of the operation input and the output of the attitude sensor at each time point. The article design support system according to claim 1 .
9. the at least one display in the headset is a pair of displays corresponding to each of the operator's eyes; In the headset, the at least one external camera is an external camera paired with each of the operator's eyes, The item image rendering unit, the background image rendering unit, and the image integration unit are provided in two systems per headset. The design support system according to claim 1 .
10. the at least one display in the headset is a pair of displays corresponding to each of the operator's eyes; In the headset, the at least one external camera is an external camera paired with each of the operator's eyes, The item image rendering unit, the background image rendering unit, the foreground image rendering unit, and the image integrating unit are provided in two systems per headset. The design support system according to claim 7.
11. a controller attitude sensor that detects the attitude of the operation input unit itself; a point position determination unit that determines a focus position based on an intersection between a virtual line in the virtual space determined in response to an output of the controller attitude sensor and a surface of the structure; a distance calculation unit that calculates a distance in a real space corresponding to two of the plurality of focus positions determined by the point position determination unit; It also has The image integration unit generates the image signal by including in the presented image a pointer display indicating the focus position updated in the virtual space according to the output of the controller attitude sensor and a distance value calculated by the distance calculation unit. The design support system according to claim 1 .
12. a controller attitude sensor that detects the attitude of the operation input unit itself; an intersection line determination unit that determines an intersection line between a virtual plane in the virtual space determined in accordance with an output of the controller attitude sensor and a surface of the structure; It also has The image integration unit generates the image signal by including in the presented image a surface display indicating the virtual surface updated in the virtual space according to the output of the controller attitude sensor and a display of a cut surface with the intersection line as a contour or a cut line by the intersection line determination unit. The design support system according to claim 1 .
13. a bounding box setting unit that sets a bounding box that is defined to include the object in the virtual space and has a surface with a plurality of grip points that can be gripped by the operator through an operation using the operation input unit; a coordinate displacement unit that displaces at least a portion of coordinates within the bounding box in accordance with an operation of the operator in a grip state with respect to any one of the grip points; a structure deformation unit that deforms the structure that references at least some coordinates within the bounding box in the virtual space by displacing coordinates referenced by the structure data in accordance with the displacement by the coordinate displacement unit; Further equipped with The article image rendering unit generates the article image having the structure after transforming the structure data called from the article data recording unit so that the structure data corresponds to the coordinates displaced by the structure transformation unit. The design support system according to claim 1 .
14. the item data recording unit stores part structure data for determining a structure of a plurality of parts constituting the item, The structural deformation unit displaces coordinates referred to by the part structural data for the plurality of parts in accordance with the displacement by the coordinate displacement unit, thereby grouping and deforming the plurality of parts. The design support system according to claim 13.
15. Further comprising an auxiliary image rendering unit for generating an auxiliary image. The image integration unit integrates the background image and the item image with the auxiliary image and outputs the integrated image as an image signal for the presentation image. The design support system according to claim 1 .
16. a plurality of said headsets, each of which is worn by a respective one of a plurality of operators; a plurality of said ambient light recorders for each of said plurality of operators; a plurality of the background image renderers for each of the plurality of operators; a plurality of said article image rendering units for each of said plurality of operators; a plurality of said image integrators for each of said plurality of operators; Equipped with Each of the environmental light recording units records brightness distribution data indicating brightness in each direction around each operator captured by the external camera in the pre-capture in association with the posture of the headset, each background image rendering unit generates a background image corresponding to an output of the attitude sensor in the real space of each operator at each time point; Each article image rendering unit generates an article image under illumination corresponding to the brightness distribution data of each operator, which corresponds to the output of the posture sensor of each operator at each time point; each image integrating unit integrates the background image and the item image to output an image signal for the presentation image for each operator; The presentation image for each operator is displayed on the display of the headset worn by each operator. The design support system according to any one of claims 1 to 15.
17. Each of the environmental light recording units stores brightness distribution data indicating the brightness in each direction around each operator captured by the external camera of each operator while changing the orientation of the headset in the pre-capture. The design support system according to claim 16.
18. a plurality of background update processors for each of the plurality of operators; Each background update processing unit replaces data corresponding to at least a part of the field of view for each operator with brightness data at each point in time captured by the external camera. The design support system according to claim 16.
19. a plurality of depth measuring units for each of the plurality of operators capable of measuring a distance to an object in each direction in real space; a foreground image renderer for each of the plurality of operators; It also has the foreground image rendering unit generates a foreground image based on an output of the external camera in a direction in which the distance to the object indicated by the depth measuring unit is shorter than the distance to the item in the virtual space; The image integration unit additionally integrates the foreground image with the background image and the item image, and outputs an image signal for the presentation image of each operator. The design support system according to claim 16.
20. a plurality of said image adjustment units for each of said plurality of operators; The image adjustment unit for one of the plurality of operators generates attribute data for the attribute selected or adjusted from the attribute candidate data in response to an instruction content of an operation input for selecting or adjusting the attribute of the article received by the operation input unit for another of the plurality of operators. The design support system according to claim 16.
21. the environmental light recording unit for one operator among the plurality of operators records the luminosity distribution data for another operator among the plurality of operators, The background image rendering unit for the one operator and the article image rendering unit generate the background image and the article image for the one operator based on the brightness distribution data for the other operator in the ambient light recording unit for the one operator. The design support system according to claim 16.
22. a background update processor for the one operator, The background update processing unit replaces data corresponding to at least a part of the field of view for the one operator wearing the headset, among the luminance distribution data for the other operator stored in the environmental light recording unit for the one operator, with luminance data acquired by photographing the outside world camera of the other operator. The design support system according to claim 21.
23. a computer device used by an administrator who is one of the plurality of operators includes an account management unit and a session management unit; the account management unit includes a permission table that determines which functions and data files each user may access in the design support system; The session management unit holds a list of users who can participate in a session as a session table, and determines whether each of the plurality of operators is a legitimate user by referring to the session table, and establishes a connection for the session for the legitimate user among the plurality of operators. The design support system according to claim 16.
24. The computer device used by the user is capable of communicating through a network as a client terminal when the computer device used by the one operator is a server device; A user ID for identifying the user is used to establish the session over the network. The design support system according to claim 23.
25. a computer having a computing device, a recording device and a graphics device; At least one headset connected to the computer, the headset being worn on the head of an operator, the headset comprising: a posture sensor capable of detecting a posture of the headset in a real space; at least one display capable of displaying an image to be presented to the operator; and at least one external camera capable of capturing an image of at least a part of the external world corresponding to the field of view of the operator; at least one operation input unit connected to the computer and configured to receive an operation input from the operator; A computer program for implementing the design support system according to claim 1 by operating a computer system comprising: The computing device, the recording device, and the graphic device of the computer function as the item data recording unit, the ambient light recording unit, the background image rendering unit, the image adjustment unit, the item image rendering unit, and the image integration unit. Computer program.
Citation Information
Patent Citations
Vehicle design support system
JP2021111374A
Image processing device
WO2017217296A1
Image processing device, image processing method and image processing program
WO2019088273A1
Immersive design management system
JP2017059213A
JP7150354
Cited By
A system for viewing a virtual space, a method for viewing a virtual space, a program, and an information processing device.
JP2026155412A
