Design Support System

Abstract

To support and streamline a design job for articles.SOLUTION: A design supporting system 1 according to the present disclosure comprises an article data recording part 510, a headset 10, an ambient light recording part 520, a background image rendering part 530, an operation input part 120, an image adjustment part 540, an article image rendering part 550, and an image integration part 580. The headset comprises an environment camera 108. In advance photography, the brightness distribution data indicating the brightness of an area around an operator 2 photographed by the environment camera is recorded in the ambient light recording part. A background image is generated from the brightness distribution data, and an image of an article 3 under the lighting corresponding to the brightness distribution data is generated from the structural data, attribute data, and brightness distribution data. The background image 42 and an article image 44 are combined to output an image signal for a presentation image 4.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a design support system. More specifically, the present disclosure relates to a system for supporting the design activities of articles.

Background Art

[0002] The design of the appearance of various articles such as industrial products and the internal structure for realizing the same affect each stage from the planning stage to the design, production, and sales of the article, and are usually determined under the involvement of many related parties. For example, in the automotive industry, the specifications of vehicles are determined according to various conditions, and the designs of the exterior and interior are carefully determined through a complicated process according to the destination and the assumed customers. In that process, the related parties involved in the planning and manufacturing of the article repeat reviews (design reviews) to determine the design of the product to be developed. An evaluation (design clinic) may be carried out by presenting the design during development or before release to a third party such as an assumed user. In order to accurately evaluate the design, not only sketches and design drawings but also prototypes and models that reproduce the final product with good reproducibility have been adopted. Even for large articles such as vehicles, by producing large-scale ones such as trial-produced actual products and full-scale clay models, the final product has been reproduced as accurately as possible.

[0003] Recently, the progress of computer technology has also been incorporated into design activities. For example, 3D prototyping is used in a computer-aided design (CAD) system, and virtual reality (VR) systems and mixed reality (MR) systems are also being utilized in design. Patent Document 1 (Japanese Patent Application Laid-Open No. 2017-59213) discloses a method and apparatus for overcoming technical problems in managing design changes of an object model. Patent Document 2 (Registration No. 7150354) discloses a system for supporting the design work of vehicle interiors.

Prior Art Documents

Patent Documents

[0004] Patent Document 1 Japanese Unexamined Patent Application Publication No. 2017-59213 Patent Document 2 Registration No. 7150354 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] In the design of an article, for example, as represented by a vehicle, it is common that a large number of stakeholders with various positions are involved in an article that involves a large investment. Also, the larger the size of the article, the more useful prototypes or models (referred to as "models, etc.") produced for evaluation are for improving the accuracy of design evaluation. However, due to their size, the production of models, etc. itself requires time and the work burden is large. Since large-sized models, etc. have a limited placement environment and their transportation costs are also high, it is not easy to conduct evaluations by a large number of evaluators with various positions. Thus, there is a contradiction that it is more difficult to implement an ideal workflow for an article for which it is desired to improve the judgment accuracy in design evaluation, and the design evaluation itself requires a large cost. Also, for an article that has a great impact on the design itself, it is often necessary to keep it confidential until it is publicly disclosed. For such an article, it is also a heavy burden to conduct design evaluations in various environments or for evaluators who are geographically dispersed to evaluate the design. The aspect of time is also important. If it takes a long time to reflect the design of an article in the final product, the risk of missing the sales opportunity due to a change in design trends increases. In the design activity of an article that assumes various environments and considers a wide range of elements, a method is required to efficiently conduct evaluations and verifications with a small burden and to overcome geographical distances.

[0006] This disclosure aims to solve at least some of the above problems, and by providing an efficient system and computer program for supporting the design work of an article, it improves the efficiency of the design work of the article and contributes to the sophistication of the article. MEANS FOR SOLVING THE PROBLEM

[0007] The inventor of the present invention conceived that the design evaluation activity of an article can be made more efficient by enjoying the progress of mixed reality (MR) technology that can provide a higher sense of reality than a virtual reality (VR) system. In MR technology, when the real space, which is one's own environment, and the virtual space in which an article is placed are fused, the user can no longer distinguish between the real space and the virtual space, and can view the design of the article with a high sense of reality. When reproducing the design of an article with advanced MR technology, even for an article that requires high reproducibility for a sense of reality, such as an article that one is familiar with in the real world (e.g., a vehicle), it is possible to bring about a feeling as if a model or the like is placed in front of one's eyes, and the user can immerse themselves in the design work. Moreover, for an article at the design stage reproduced in a mixed reality space, changes such as changing the structure, or changing attributes such as color, shape, and texture can be easily made. If the design of an article can be perceived by the user with a high sense of reality by utilizing MR technology, design evaluation with high accuracy can be performed without producing a model or the like, and the design evaluation work can be made more efficient.

[0008] What the inventor of the present application focused on when trying to utilize MR technology was what factors affect the reproducibility of the design of an article for providing a high sense of reality and may influence the sense of immersion. What was noted was that when the ambient light environment cannot be sufficiently reflected in the appearance of an article by MR technology, it is difficult to obtain a sufficient sense of reality even when observing the appearance of the article. However, the ambient light environment varies widely, and simply measuring brightness is insufficient. Also, it is not easy to measure the ambient light environment with a special measuring device. Therefore, the inventor of the present application found that using a headset such as an HMD equipped with an external camera can lead to a highly practical method for reflecting the ambient light environment in the appearance of an article. In the present disclosure, a design support system that can support design work is provided by giving a high sense of reality to the design of an article using a virtual space.

[0009] That is, in one aspect of the present disclosure, there is provided an article data recording unit that stores structure data for determining the structure of at least one article in a virtual space and attribute candidate data for selecting or adjusting attributes of the article, a head-mounted display device for an operator, the head-mounted display device including an attitude sensor capable of detecting the attitude of the head-mounted display device in the real space, at least one display capable of displaying a presentation image to the operator, and at least one external camera capable of photographing at least a part of the external world corresponding to the field of view of the operator; an ambient light recording unit that records, in association with the output of the attitude sensor, brightness distribution data indicating brightness in different directions around the operator photographed by the external camera in a pre-shooting process; a background image rendering unit that generates a background image corresponding to the output of the attitude sensor at each time point from the brightness distribution data called from the ambient light recording unit; at least one operation input unit that receives an operation input from the operator for selecting or adjusting the attributes of the article; an image adjustment unit that generates attribute data for the attributes selected or adjusted from the attribute candidate data according to the instruction content of the operation input received by the operation input unit; an article image rendering unit that generates an article image under illumination corresponding to the brightness distribution data, corresponding to the instruction content of the operation input and the output of the attitude sensor at each time point, from the structure data called from the article data recording unit, the attribute data from the image adjustment unit, and the brightness distribution data called from the ambient light recording unit; and an image integration unit that synthesizes the background image and the article image and outputs the result as an image signal for the presentation image. A design support system is provided.

[0010] Furthermore, the present disclosure also provides a computer program for the above-described design support system.

[0011] In this application, unless otherwise specified, technical terms used in the field of computer graphics may be employed. In this application, the object to be designed is referred to as an "article". In this application, this article is not particularly restricted and includes any object that can be recognized in terms of shape through vision by the end user, such as movable property, immovable property, tangible objects, and intangible objects in the physical space. Therefore, the articles in this application include any real-shaped object, and may also include articles that are only premised on being reproduced by simulating the physical space in a virtual space or a composite space (a space combining a virtual space and a physical space). Note that in this application, many explanations are given with respect to vehicles (automobiles), and technical terms for vehicles also appear, but these are for illustrative purposes. In this application, the design includes not only the design as an aesthetic appearance, but also those targeting shapes such as the structural design of an article. Therefore, design work is not necessarily limited to activities focusing only on the aesthetic point of view, but includes a wide range of creative activities including structural design, mechanism design, etc. Design work typically refers to activities of determining the design by evaluating or changing the design in the above activities including structural design and appearance design. However, it is also possible to use the design support system of the present disclosure to display or experience the design, and in that sense, activities that do not necessarily bring about an active change to the article can also be design work in the present disclosure. A pose is a combination of orientation and position, and a sensor for detecting a pose is intended to detect at least one or more components of orientation and position. A virtual space is a three-dimensional space defined on a computer and is described separately from the physical space, but this distinction is for the purpose of explanation. An operator generally refers to an experiencer or user who experiences or uses the design support system in this application. In the context of evaluating the design of an article, the evaluator can be an operator. The physical space is the space where the operator's body actually exists. A headset is a device that can present an image to at least the vision of an operator by being fixed to the head of the operator in augmented reality technology.What are called head-mounted displays (HMDs), VR glasses, XR glasses, etc. are typical examples of such headsets. The headset of the present application is equipped with an external camera. This external camera typically consists of a pair of cameras corresponding to both eyes of the operator, and there may be multiple pairs. In the present disclosure, images include still images and moving images. Also, video includes moving images, but in some cases, even if part or all of it is a still image, it may still be called a video.

Advantages of the Invention

[0012] In any aspect of the present disclosure, a design support system or a computer program for streamlining the design business of an article is provided.

Brief Description of the Drawings

[0013] [Figure 1] It is an explanatory diagram outlining the use of the design support system in an embodiment of the present disclosure. [Diagram 2] It is a block diagram showing the functions of the design support system in an embodiment of the present disclosure. [Diagram 3] It is an explanatory diagram exemplifying the configuration of the headset adopted in the design support system of an embodiment of the present disclosure. [Figure 4] It is a flowchart exemplifying the outline of the usage method of the design support system of an embodiment of the present disclosure. [Figure 5A-B] It is an explanatory diagram (Figure 5A) explaining the operation of the external camera in the design support system of an embodiment of the present disclosure, and a display (Figure 5B) showing an example of its photograph. [Figure 6A-B] It is an explanatory diagram (Figure 6A) explaining the operation of the external camera in the design support system of an embodiment of the present disclosure, and a display (Figure 6B) showing an example of its photograph. [Figure 7A-C] It is an explanatory diagram (Figure 7A) showing the hardware configuration of the design support system of an embodiment of the present disclosure, and explanatory diagrams (Figures 7B, C) showing how a background image, an article image, and a foreground image are combined in the operation of generating a presentation image for both the left and right eyes. [Figure 8] It is a block diagram showing the functions of the design support system in the embodiments of the present disclosure. [Figure 9] It is an explanatory diagram showing the hardware configuration of the design support system in the embodiments of the present disclosure. [Figure 10A-C] It is a captured image of the display in the design support system in the embodiments of the present disclosure. [Figure 11A-B] It is a captured image of the display in the design support system in the embodiments of the present disclosure, showing before (Fig. 11A) and after (Fig. 11B) the change of the body color when a vehicle is displayed as an article. [Figure 12A-B] It is a captured image of the display of the headset in the design support system in the embodiments of the present disclosure, showing the specular reflection (Fig. 12A) and the diffuse reflection (Fig. 12B) when a sphere is displayed as an article. [Figure 13A-C] It is an explanatory diagram showing the operation of the deformation function in the design support system in the embodiments of the present disclosure, showing before deformation (Fig. 13A), the bounding box (Fig. 13B), and after deformation (Fig. 13C). [Figure 14A-B] It is a captured image of the display of the headset in the design support system in the embodiments of the present disclosure, showing the state of using the distance measurement function to measure the distance (Fig. 14A) and the state of using the cross-section display function (Fig. 14B). [Figure 15A-B] It is a captured image of the display of the headset in the design support system in the embodiments of the present disclosure, showing an example of displaying an image of the real space from an external camera (Fig. 15A) and an example of displaying an image of the virtual space (Fig. 15B) for the background image. [Figure 16] It is an explanatory diagram showing an example of the configuration in the situation where multiple operators in the embodiments of the present disclosure cooperate via a network. [Figure 17] It is an explanatory diagram of the design support system including an eye tracking detection unit employed for the detection and recording of the fixation point in the embodiments of the present disclosure. [Figure 18]A block diagram of a design support system that employs a headset having an eye tracking detection unit in an embodiment of the present disclosure.

Embodiments for Carrying Out the Invention

[0014] Embodiments of an article design support system according to the present disclosure will be described with reference to the following drawings. Throughout the description, common reference numerals are assigned to common parts or elements unless otherwise particularly noted. Also, in the drawings, the elements of each embodiment are not necessarily shown while maintaining their scale ratios to each other.

[0015] 1. System Configuration The design support system provided in the present disclosure will be described.

[0016] 1-1. Configuration (Overview) FIG. 1 is an explanatory diagram schematically showing a state in which the design support system 1 according to an embodiment of the present disclosure is being used. FIG. 2 is a block diagram showing the functions of the design support system 1 provided in the present disclosure. In the real space (physical space) where the design support system 1 is used, an operator 2 is wearing a headset 10. The physical space 9 is, for example, the room where the operator 2 is actually present, and there is lighting 92 and the like in that space. There may also be various lights emitted from the light from the window or some light source in the physical space 9. To the operator 2, a presented image 4 that appears as if an article 3 exists in the physical space 9 is provided via an appropriate ocular optical system (not shown) through the display 102 of the headset 10. This presented image 4 includes, for example, a background image 42 and an article image 44. The operator uses such a design support system 1 to evaluate the design of the article 3. When the design support system 1 of the present disclosure is adopted, the article 3 in the presented image 4 has a high sense of reality for the operator 2. In the design support system 1 of the present disclosure, brightness distribution data 522 captured by the external camera 108 provided in the headset 10 is captured for the background image 42. The shooting data for this background image 42 is captured from the external camera 108 by an operation such as the operator 2 wearing the headset 10 on the head and looking around in the stage before evaluating the design of the article 3 called pre-shooting, and is recorded in the ambient light recording unit 520. The headset 10 is provided with an attitude sensor (attitude sensor mover 104). While the attitude of the headset 10 at each moment is detected by the attitude sensor, a background image 42 and an article image 44 suitable for that attitude are generated by a computer 50. The background image 42 is combined with the article image 44 to become the presented image 4. Here, when the article image 44 is generated according to the attitude of the headset 10, it is generated so as to reflect the structure and attributes of the article 3 in the virtual space. The brightness distribution of the background in the shooting data of the external camera 108 is reflected in the appearance of the article 3 in the article image 44. That is, the article image 4 realizes a highly realistic display that reflects how the appearance of the article 3 looks under the illumination based on the brightness distribution acquired by the external camera 108.This view reflects the difference in brightness of each part of the surface of the article 3 that reflects the surrounding light environment, such as the brightness according to the illuminance by the illumination 92. Further, when the surface of the article 3 exhibits specular reflectivity, for example, the reflection is reproduced. Therefore, the article 3 perceived by the operator 2 through vision has a high sense of reality as if it were an actual object or model existing in the real space 9.

[0017] The design support system 1 can be realized as hardware by the headset 10, related peripheral devices, and the computer 50. The headset 10 is equipped with an external camera 108 and typically does not have an optical transmission part. The computer 50 can be a personal computer or a workstation with high graphic performance. However, due to recent advancements, even an easily accessible one can achieve satisfactory performance. The computer 50 can adopt those having general computer resources such as a recording device (main memory device, auxiliary memory device), a graphic device, an arithmetic device, an input / output device, a network device, and an input device. The headset 10 and the controller (operation input part) 120 are connected through an appropriate wired or wireless connection interface. Importantly, the minimum hardware configuration for realizing the design support system 1 does not include additional hardware (such as a celestial sphere camera) for the purpose of acquiring brightness distribution data for the real space 9, and adopting such an additional hardware configuration is optional. Therefore, the design support system 1 only requires a relatively easily accessible and inexpensive hardware configuration.

[0018] The design support system 1 includes an article data recording unit 510, at least one headset 10, an ambient light recording unit 520, a background image rendering unit 530, at least one operation input unit 120, an image adjustment unit 540, an article image rendering unit 550, and an image integration unit 580. The article data recording unit 510 stores structure data 512 for determining the structure of at least one article 3 in the virtual space and attribute candidate data 514 for selecting or adjusting attributes of the article 3. Here, an attribute is a property that is subject to selection or adjustment in the design work. Therefore, when a design change involving a shape change is the purpose of the design work, the structure for specifying the shape can also be a typical example of an attribute. The headset 10 will be described in detail in the next section. The ambient light recording unit 520 records, in association with the outputs of the attitude sensors 104 and 106, brightness distribution data 522 indicating the brightness in different directions around the operator 2 photographed by the external camera 108 in the pre-shooting. The background image rendering unit 530 generates a background image 42 corresponding to the latest outputs of the attitude sensors 104 and 106, that is, the outputs at each time point, from the brightness distribution data 522 called from the ambient light recording unit 520. The at least one operation input unit 120 receives operation inputs from the operator 2 for selecting or adjusting attributes of the article 3. The image adjustment unit 540 generates attribute data 542 for the attributes selected or adjusted from the attribute candidate data according to the instruction content of the operation input received by the operation input unit 120. The article image rendering unit 550 generates an article image 44 under illumination corresponding to the brightness distribution data 522, corresponding to the instruction content of the operation input and the outputs of the attitude sensors 104 and 106 at each time point, from the structure data 512 called from the article data recording unit 510, the attribute data 542 from the image adjustment unit, and the brightness distribution data 522 called from the ambient light recording unit 520. The image integration unit 580 synthesizes the background image 42 and the article image 44 and outputs them as an image signal for the presentation image 4.

[0019] In the design support system 1, it is preferable that the article image rendering unit 550 reflects the surrounding reflected image corresponding to the brightness distribution data on the surface reflection of the article image. Further, in the design support system 1, it is preferable that the background image rendering unit 530 calls the structure data from the article data recording unit and reduces the brightness of the background image in the direction of the shadow where the article should fall in the real space corresponding to the brightness distribution data.

[0020] The brightness distribution data 522 can also acquire brightness by color. In this case, the ambient light recording unit 520 records the color brightness data (not shown) indicating the brightness by color photographed by the external camera 108 in the pre-photographing in association with the outputs of the attitude sensors 104 and 106. The background image rendering unit 530 generates a color background image corresponding to the outputs of the attitude sensors 104 and 106 at each time point from the color brightness data called from the ambient light recording unit 520, and the article image rendering unit 550 generates a color brightness data corresponding to the operation input instruction content and the outputs of the attitude sensors 104 and 106 at each time point from the structure data 512 called from the article data recording unit 510, the attribute data 542 from the image adjustment unit 540, and the color brightness data called from the ambient light recording unit 520. When an article image under illumination is generated, a higher sense of reality with the colors in the real space 9 reflected in the article image is brought about.

[0021] At least one operation input unit 120 can be any device capable of inputting an instruction to the computer 50. Even when the operator 2 wearing the headset 10 views the shooting data during pre-operation in the real space 9 only through the display 102, real-time operations can be performed. A device suitable for the design support system 1 as the operation input unit 120 capable of coping with such a situation, in addition to any controller that can be held by hand and scanned in a direction or control input buttons, can adopt a sensor (hand tracking sensor) capable of capturing the movement of the operator's hand as shown in the hand tracking camera 108A (FIG. 3). When a laser pointer or menu display in the virtual space is provided according to the operation of the operation input unit 120 through the presented image 4 to the operator 2, or a display suitable for input in the virtual space such as a virtual keyboard is adopted, in order to determine the attributes of the article 3, for example, color change, model change, texture change, shape deformation, etc., execution instructions as a configurator become easy.

[0022] 1-2. Headset FIG. 3 is an explanatory diagram illustrating the configuration of the headset 10 adopted in the design support system of the embodiment of the present disclosure. A typical example of the headset 10 is, for example, Varjo XR-3 manufactured by Varjo Technologies Oy.

[0023] The headset 10 is for the operator 2 to wear on the head, and includes a posture sensor mover 104 and a posture sensor stator 106 that can detect the posture of the headset 10 in the real space 9, at least one display 102 that can display the presented image 4 to the operator 2, and at least one external camera 108 that can capture at least a part of the external world corresponding to the field of view for the operator 2. In a typical configuration, in the field of view that the operator 2 can perceive through vision, only the presented image 4, that is, the image generated through the design support system 1, exists as a visual stimulus. Therefore, the headset 10 does not have an optical transmission part such as HoloLens (registered trademark) that directly allows the operator to visually recognize the outside world and provides an optical pass-through image. In the presented image 4 presented to the operator 2, both the background image 42 and the article image 44 are provided through the display 102 of the headset 10.

[0024] The display 102 and the external camera 108 are paired corresponding to both eyes in order to enhance the sense of reality also from the perspective of depth perception by so-called stereoscopic vision that reflects binocular parallax. Accordingly, it is also preferable that the article image rendering unit 550, the background image rendering unit 530, and the image integration unit 580 are provided in two systems per headset.

[0025] 1-3. Acquisition of brightness distribution data by an external camera As shown in FIG. 3, the external camera 108 can take pictures within a field of view fixed to the headset 10. The external camera 108 preferably has a wide angle of view close to the field of view by a human eyeball (about 120°). In FIG. 3, the range of the angle of view that can be photographed by the pair of external cameras 108 for both eyes is represented by a cone mark. In the design support system 1 of the present disclosure, the brightness distribution data 522 is acquired by the external camera 108. When the orientation of the headset 10 is fixed, the external camera 108 may not be able to photograph the celestial sphere (the entire celestial sphere or the half celestial sphere). In this case, in order to acquire the brightness data 522, the operator 2 scans the direction of 108. That is, in the pre-photographing, it is possible to acquire the brightness for each direction obtained by photographing with the external camera while changing the orientation of the headset. This pre-photographing is scanned, for example, by the operator 2 wearing the headset 10 pulling the jaw and maintaining the standing position, scanning all directions in the horizontal plane, raising the jaw, and scanning all directions in the horizontal plane again. In this way, photographing in the necessary directions for the brightness distribution data 522 can be performed. At this time, it is preferable that the photographing data is associated with the attitude data and stored in the ambient light recording unit 520. From this attitude data, the photographing direction of the external camera 108 can be determined, and it can be used for, for example, image stitching in order to construct the brightness distribution data over a sufficient range of directions such as the entire celestial sphere. Also, this attitude data can be used to associate the brightness distribution data with the direction in the real space.

[0026] It is useful that the brightness distribution data is stored with the reproducibility of the brightness of surrounding objects in a sufficient luminance range so as to be useful for the reproduction of illumination and the reproduction of reflection. For this purpose, for example, any format for an HDR (High Dynamic Range) image with a wide latitude is suitable. However, in the design support system 1, the photometric or colorimetric accuracy and traceability of the brightness distribution data are not necessarily required. The degree of accuracy required for the operator 2 to have a sense of reality can be determined according to the specific use and purpose of the design evaluation.

[0027] FIG. 4 is a flowchart exemplifying an outline of a method of using a design support system according to an embodiment of the present disclosure. In the pre-shooting S02 described above, brightness distribution data 522 is acquired. This brightness distribution data 522 is stored in the ambient light recording unit 520 in association with the posture. Thereafter, the attributes of the article 3 are set (S04), and the design is displayed or evaluated (S06). In this display or evaluation S06, the brightness distribution data 522 obtained by the operator 2 photographing the article 3 through the external camera 108 in his or her environment is used for two purposes. That is, the brightness distribution data is used for the background image 42 and for the illumination of the article 3 in the article image 44. After the design of the article 3 is evaluated in the attribute setting S04, the conditions are changed as necessary (S08), and the next conditions are set (S10). The conditions changed here are typically the attributes for the design that are changed.

[0028] In the above operation, the brightness distribution data 522 at the time of pre-shooting S02 is directly adopted for the background image 42 and the article image 44 observed by the operator 2. In the design support system 1, in order to enhance the sense of reality, it is also useful to replace, with the brightness data at each time point captured by the external camera 108, the data corresponding to at least a part of the visual field for the operator 2 wearing the headset 10 among the brightness distribution data 522. Such processing is performed by the background update processing unit 532 (Fig. 2). In the flowchart of Fig. 4, since the display or evaluation S06 is the evaluation of the article 3 by the operator 2, it is carried out over a certain period of time. During this time, considerations such as observing the article 3 from various directions are made. In the middle of this, if the brightness distribution data 522 reflecting the content captured by the external camera 108 is adopted instead of the fixed brightness distribution data 522 with unchanged content, the change in lighting in the real space 9 and, for example, the presence of another operator on the opposite side moving around are also reflected in the evaluation of the article 3. In the article image 44, since the lighting conditions for the article 3 and the reflection on its surface are also updated, updating at least a part of the brightness distribution data 522 according to the captured content of the external camera 108 helps to enhance the sense of reality of the article 3 and immerse the operator 2 in the evaluation work.

[0029] Figures 5A - B and 6A - B are explanatory diagrams (Figures 5A, 6A) for explaining the operation of the external camera in the design support system of the present disclosure, and displays (Figures 5B, 6B) showing examples of the captured images. In pre - shooting, the brightness distribution data is acquired, for example, for the entire sky sphere, which is the range indicated by the wire grid in Figures 5A and 6A. Figure 5B shows the state of the interior obtained in pre - shooting in the form of a panoramic unfolded view. In this way, by using the external camera 108 for the operator 2 to take pictures, the brightness distribution data of the entire sky sphere shown in Figure 5A can be acquired, and details can be obtained as shown in Figure 5B. By adopting image - based lighting (IBL) that uses the brightness distribution data exemplified in Figure 5B as lighting and adopting a drawing method that accurately reflects the reflectivity of the article 3, the appearance of the article 3 in the virtual space can be reproduced with a high sense of reality by matching it as much as possible to the ambient light environment of the real space 9. Note that although some of the drawings in this application are shown in gray in accordance with the specifications of the patent application format, in the design support system 1, colors can be reproduced or photographed according to human color vision. Also, in some of the drawings of this application, even when there is data that can cover a wide brightness range (latitude, dynamic range), such as an HDR image, it is shown within the normal brightness range.

[0030] In FIG. 6A, the range photographed by the external camera 108 is shown explicitly in a partial area of the wire grid. The external camera 108 is fixed to the headset 10 and has an angle of view that covers a certain extent of the human field of view, but when viewed from the entire sphere, the angle of view remains only partial. In FIGS. 6A and 6B, the contour of the field of view (camera angle of view 48) that can be photographed by the external camera 108 is shown by a solid line. As is clear from the comparison between FIG. 5B and FIG. 6B, the range that can be photographed by the external camera 108 remains only a part of the entire sphere, but the changes within its camera angle of view 48 (the presence or absence of a person in FIGS. 5B and 6B) can be significant for the operator 2. This is because in the scene where the design of the article 3 is being evaluated using the headset 10, the inside of the camera angle of view 48 is the position where the view enters through the article 3 being observed. The position where an image may be formed near the fovea of the retina in human vision is of great significance for the pass-through display by the external camera 108. Thus, it is extremely useful to appropriately reflect the situation of the real space 9 at that time photographed using the external camera 108.

[0031] 1-4. Distance measurement, hand, foreground When the design support system 1 realizes a highly realistic reproduction of the article 3, the operator 2 tends to feel discomfort that his / her body (e.g., hand) is not included in the presented image. For example, when operating a controller that is common in VR technology or MR technology by hand, it becomes difficult to operate if the controller or hand cannot be seen. In particular, when a headset 10 without an optically transmissive part is adopted in the design support system 1 of the present disclosure, it is desirable to add visual information about the outside world. Therefore, in a preferred configuration of the design support system 1, it further includes a depth ranging unit 130 and a foreground image rendering unit 570. The depth ranging unit 130 can measure the distance to an object in each direction of the real space. This can be achieved by equipping a LiDAR (Light Detection And Ranging) optical element (not shown) at the position of the external camera 108 or by means of a hand tracking camera. For a direction in which the distance to the object indicated by the depth ranging unit 130 is smaller than the distance to the article in the virtual space, the foreground image rendering unit 570 generates a foreground image 46 based on the output of the external camera 108 or the hand tracking camera 108A. In this case, the image integration unit 580 outputs an image signal for the presented image by additionally synthesizing the foreground image 46 with the background image 42 and the article image 44, so that the presented image can include a captured image of an object located in front of the article, such as a controller or a hand.

[0032] Figs. 7A to 7C are explanatory diagrams showing the hardware configuration of the design support system according to an embodiment of the present disclosure (Fig. 7A), and explanatory diagrams showing how a background image, an object image, and a foreground image are combined in the operation of generating presentation images for both the left and right eyes (Figs. 7B and 7C). As shown in Fig. 7A, one computer 50 is employed for an operator (not shown in Fig. 7A). In this computer 50, as shown in Fig. 7C, presentation images 4L and 4R for both the left and right eyes are generated through displays 102 for both the left and right eyes. The operator can perceive depth by the presentation images 4L and 4R that correctly reflect binocular disparity. The presentation images 4L and 4R for each of the left and right eyes in Fig. 7C are transmitted to the headset 10 through transceivers 52L and 52R in independent channels for the left and right in the computer 50 of Fig. 7A. The transceivers 52L and 52R relay the transmission and reception between an appropriate input / output interface (e.g., USB-C3.0 / 3.1, Display Port) of the computer 50 and the headset 10. At this time, a video signal or data from the external camera 108 equipped in the headset 10 is also transmitted to the computer 50 separately for the left and right. The posture (orientation, position) of the headset 10 in the real space 9 is detected by posture sensors 104 and 106. The posture sensors 104 and 106 can be a combination of a direction sensor such as a gyroscope or a magnetic sensor and a position sensor that detects a position in Cartesian coordinates by infrared rays or lasers. If necessary, the posture sensors 104 and 106 are provided with a ground unit (posture sensor ground unit 106) to ensure numerical accuracy. The background image 42 is the brightness distribution data acquired in advance shooting or the updated data thereof, and is photographed by the external camera 108. The object image 44 includes an image of the object 3 calculated by using the brightness distribution data 522 as illumination for image-based lighting. The foreground image 46 is an image of an object that should exist in front of the object 3. They are explanatory diagrams (Figs. 7B and 7C) showing how the background image, the object image, and the foreground image are combined. This combination can be executed by a known image combination method such as alpha blending or Z-buffering.In the object image, parts other than the object 3, and in the foreground image 46, parts other than the objects that should be in front of the object 3 are transparently synthesized. By performing such synthesis independently for each of the left and right sides, a highly realistic presentation image can be provided in which the object 3 can be perceived as if it were a model placed in the real space.

[0033] Note that in FIG. 10C, the state where the operator 2's own arm and hand extend to the object 3 is actually shown as the foreground image 46. Although it is impossible to actually touch the object 3, when the operator 2's own arm and hand are included in the presentation image 4 of the display 102, the operator 2 will have a high sense of reality with respect to the object 3.

[0034] As shown in FIGS. 6A and 6B, due to the limitation of the human visual field range, the range that should be presented to the operator 2 as a presentation image through the display 102 is a part of the entire spherical range. Therefore, for the purpose of enhancing the sense of reality as much as possible, it is also preferable to display the real-time image from the external camera 108 for the background image 42. On the other hand, for the brightness distribution data for the lighting process of the object 3 included in the object image 44, those in the range of substantially the entire spherical range due to the reflection by the surface of the object 3 greatly contribute to enhancing the sense of reality. Therefore, it may be important to be able to use data in a wide range in terms of direction by using the brightness distribution data acquired in the pre-shooting S02 (FIG. 4) or keeping its update to the minimum necessary.

[0035] 1-5. Multiple operators The present disclosure also provides a design support system that can be used simultaneously by a plurality of operators. FIG. 8 is an explanatory diagram showing the configuration of the design support system 1M. The design support system 1M is provided with a set of a headset 10, a display 102, an external camera 108, and a computer 50 corresponding to each operator, and the sets are connected in the required number. Each set can also operate stand-alone. For this reason, the ambient light recording unit 520, the background image rendering unit 530, the article image rendering unit 550, and the image integration unit 580 are implemented in each computer 50 (see FIG. 2. Not shown in FIG. 8). For this reason, the ambient light recording unit 520 for each operator records the brightness distribution data indicating the brightness in each direction around each operator 2 in association with the posture of the headset. Each background image rendering unit 530 generates a background image 42 corresponding to the output of the posture sensor at each point in time in the real space of each operator. Each article image rendering unit 550 generates an article image 44 under illumination corresponding to the brightness distribution data of each operator, corresponding to the output of the posture sensors 104 and 106 of each operator at each point in time. Each image integration unit synthesizes the background image 42 and the article image 44 and outputs an image signal for the presentation image 4 for each operator. Further, as shown in FIG. 4, it is preferable that the brightness distribution data 522 indicating the brightness in each direction around each operator photographed by the external camera 108 of each operator 2 while changing the direction of the headset 10 in the pre-shooting S02 is stored in each ambient light recording unit 520. In this way, the presentation image 4 corresponding to the real environment of each operator is displayed on the display 102 of the headset 10 worn by each operator. In this way, in the present embodiment, even when a plurality of operators exist in different real spaces, each operator can execute the design evaluation of the article 3 reflecting his or her own environment.

[0036] In addition, in a design support system that can be used simultaneously by a plurality of operators, it is also preferable to configure the system such that a background image and an article image are generated by one operator based on the brightness distribution data of another operator. Specifically, an ambient light recording unit for one of the plurality of operators records the brightness distribution data for another operator. The background image rendering unit and the article image rendering unit for that one operator utilize the brightness distribution data for another operator in the ambient light recording unit to generate the background image and the article image. With such a configuration, it becomes possible to use the brightness distribution data acquired by another operator in a pre-shooting operation to generate the background image and the article image for one operator. That is, since the background image and the article image observed by another operator are also observed by one operator, it is possible to match the ambient light environment affecting the article being evaluated by both operators and the appearance of the article. Even in this case, each operator can feel a high sense of reality with respect to the article being observed by himself / herself. In this configuration, for one operator, since the brightness distribution data is not from his / her actual environment, the situation is substantially closer to VR (virtual reality) than MR. In collaborative work for design evaluation, it is useful if operators can share the appearance of the article being evaluated with each other, as it deepens their understanding of each other's evaluations.

[0037] In the case of adopting such a configuration to match the appearance of an article, processing across operators can also be performed in the update process. Specifically, a background update processing unit is provided for one operator, and it is preferable that the operation of the background update processing unit targets the brightness distribution data of another operator in the ambient light recording unit for one operator. That is, among the brightness distribution data, the data corresponding to at least a part of the field of view for one operator wearing a headset is replaced with the brightness data obtained by shooting with the external camera of another operator. Since multiple operators do not necessarily face the corresponding directions simultaneously, the update is not necessarily made with the latest brightness data. However, if the update of the brightness distribution data by the external camera of another operator is reflected in the evaluation of the design of an article by one operator, the collaborative work of the design evaluation can be smoothly carried out.

[0038] When assisting collaborative work by multiple operators, in the design support system 1M, for example, the above-described article data recording unit 510 and ambient light recording unit 520 are secured in a data server 50S that can communicate with each computer 50, and the background image rendering unit 530, background update processing unit 532, image adjustment unit 540, article image rendering unit 550, and foreground image rendering unit 570 (none of which are shown in FIG. 8) are distributed to each computer 50. Such an implementation can be performed. In this way, while showing an appropriate presentation image to each operator even when the heads of the respective operators are in different postures, design considerations can be performed for a common model. Further, the operation input by the operation input unit 120 may be performed independently by each operator, or a presentation image according to the operation input by one of the operators can be presented to other operators. Particularly when performing collaborative work by multiple operators, an operator may perform operations such as specifying and highlighting a portion to be noted for explaining to other operators. For such purposes, in lighting within the virtual space, it is also preferable to enable an operator who gives an instruction to operate local lighting corresponding to a pointer or a flashlight (torch). For example, the operation input unit 120 held by operator 2 can be provided with a function of detecting a direction like a pointer. In this way, in the design support systems 1 and 1M, it is preferable to implement a function of indicating an arbitrary position and an operation function of local lighting.

[0039] The design support system 1M preferably includes a plurality of background update processing units 532 for each of a plurality of operators. Each of the background update processing units 532 replaces data corresponding to at least a part of the field of view for each operator with the brightness data at each time point acquired by photographing with an external camera. In this way, each operator can observe the background image 42 and the article image 44 based on the brightness distribution data updated corresponding to his or her real space 9, and a high sense of reality is realized for the plurality of operators. Also in the design support system 1M, by adopting the depth ranging unit 130, the foreground image 46 can be superimposed and displayed on the background image 42 and the article image 44.

[0040] FIG. 9 is an explanatory diagram showing the hardware configuration of the design support system 1M according to the embodiment of the present disclosure, and has a configuration in which a headset 10 using a computer 50 is adopted for each of a plurality of operators 2. As shown in FIG. 8, typically one computer 50 is adopted per operator 2, and a presentation image is generated on the computer 50 through displays 102 for both the left and right eyes. In FIG. 9, for two operators (Operator A, B), a headset 10 and a computer 50 that form the design support system 1M according to the embodiment of the present disclosure are adopted one set each in the real spaces 9A and 9B that are the spaces where they are actually located. The devices shown in FIG. 9 are connected by an arbitrary network such as a LAN (Local Area Network) or a WAN (Wide Area Network). However, such a correspondence relationship between the computer device and the operator can be changed in consideration of the processing performance of the devices to be adopted and the required quality. If it is possible to support the design evaluation of articles by a plurality of operators, evaluation based on various viewpoints can be easily carried out, and the efficiency of the design evaluation can be improved.

[0041] From a practical perspective, it is also useful to record the presented image (presented to either or both of the left and right eyes) presented to any one of the operators 2 simply as a video image or present it on an external display device. The operator of the design support system 1M is immersed in the evaluation work of the article 3, but the fact that the state is recorded or monitored by a monitor device has advantages such as being an objective record and being able to confirm whether the intended evaluation is being carried out. At this time, it is also beneficial to record the points of attention described later.

[0042] 2. Embodiment Here, an embodiment of the design support system 1 of the present disclosure that implements the above-described features and other features will be described. An embodiment of the design support system 1 of the present disclosure was realized with the following hardware configuration. · Headset, transceiver, attitude sensor: Varjo XR-3 or Varjo XR-3 Focal Edition (Varjo Technologies Oy, Helsinki, Finland) · PC: Manufactured by HP Inc., Windwos11 · CPU: Core i7-13700K (manufactured by Intel) · Memory: 32GB · Graphics board: GeForce RTX 4070 (manufactured by NVIDIA) · Software: Unreal Engine (manufactured by Epic Games), OpenXR (manufactured by Khronos Group) (Each company name and product may include registered trademarks of each company) · Article: Four-wheel automobile design example Figures 10A - C and 11 are captured images of the display of the headset in an example of the design support system according to an embodiment of the present disclosure. In an example system where the design support system 1 shown in FIG. 2 was actually constructed, snapshots of videos recording the images of both the left and right eyes were captured. These figures are images presented to the operator 2 by the display 102. Since the headset 10 does not have an optically transmissive part, the operator 2 will only see such electronically generated Picture images through vision. Here, an automobile is being designed as the article 3. Also, the real space 9 is the interior as shown in FIG. 5B. For the update process of the background image 42, the shooting signal from the external camera 108 was used as a pass-through. The brightness distribution data is what was recorded by the environmental light recording unit 520 from the all-sky image taken by the operator 2 who actually wore the external camera 108 during pre-shooting. The article 3 in the article image 44 was realized in the virtual space using image-based lighting that utilized the brightness distribution data. The correspondence between the virtual space and the real space for the article 3 was made to match the coordinates by a calibration marker (2D code) placed on the floor surface of the real space. For the processing of the image integration unit 580, alpha blending was adopted so that the background of the article image 44 was appropriately synthesized at the light transmission part (such as the window of the automobile) of the article 3 in the article image 44.

[0043] In FIGS. 10A to 10C, the interior is set as the real space 9, and a relatively large automobile is arranged as the article 3 therein. To the operator 2, the article 3 is perceived as being placed at a very close distance so that it can be touched by reaching out the hand. This is because, in addition to the three-dimensional feeling by stereoscopic vision reflecting binocular parallax, the appearance of the automobile generated by image-based lighting in the virtual space is reproduced with a high sense of reality. For example, in FIG. 10A, the interior lighting arranged on the ceiling in the real space 9 is reflected as a specular reflection on the glass surface while reflecting its curvature on the windshield surface of the automobile. Also, looking at the hood surface of the automobile, the texture of the metallic paint combining surface reflection and diffuse reflection is expressed with a high sense of reality, including the reflection by surface reflection and the change in the brightness of the surface reflecting the direction of the light source. The reflected light sources can also be perceived to be at different positions in the depth direction from the windshield surface and the hood surface when viewed stereoscopically with both the left and right eyes. As a result of these, the operator 2 can evaluate the design as if an automobile or its model is placed in the real space 9. Similar articles 3 are shown in FIGS. 10B and 10C. The operator 2 can adjust the viewpoint within the range of the extent of the real space 9. For example, the operator can move by walking himself / herself or change the height of the head from the floor surface. Also, the operator can peek in by tilting the head, look up, or turn in various directions. The operator can also take these orientations and positions (postures). In addition to the high sense of reality of the appearance of the article 3, the operator 2 can perform the evaluation while adjusting the viewpoint by extremely natural movements Other items 3 The and can execute the evaluation. However, there is still room for further improvement in the sense of reality. The shadow that the automobile not shown in FIGS. 10B and 10C should cast in the real space is reproduced in this example system by adding a function to reduce the brightness of the background image 42 in the direction of the shadow that the article 3 should cast in the real space 9 corresponding to the brightness distribution data. Other features shown in FIGS. 10B and 10C will be described later.

[0044] Figures 11A and 11B show the state before and after changing the body color for the same four-wheel vehicle model shown in FIGS. 10A to 10C, observed from substantially the same posture (position and orientation). Although it is shown in gray due to format constraints, in the example system, before and after changing from a high-brightness silver metallic paint (FIG. 11A) to a low-brightness red metallic paint (FIG. 11B), it is displayed on the display 102. As shown in FIGS. 11A and 11B, the article 3 not only has a high sense of reality but also can easily change its attributes (body color) in the virtual space. Before and after that, from the comparison between FIG. 11A and FIG. 11B, not only the differences in hue and saturation not shown in the drawing but also the differences in the relative strength of surface reflection due to the difference in brightness and the expression of shadows change in details, making it possible to evaluate the design with a very high sense of reality. Here, the color change with metallic paint is exemplified, but in addition to metallic paint, any surface display reproducible in the virtual space can be reproduced as long as various paints such as matte, solid color, and candy paint, as well as patterns and painting other than paint, can be selected. Since these adjusted or selected attribute candidate data are stored in the article data recording unit 510, it can be implemented only by changing software and data.

[0045] The writing's principle of operation was confirmed using an article with a simpler shape. FIGS. 12A - B are captured images of the display of a headset in an example of the design support system 1 of an embodiment of the present disclosure, showing specular reflection (FIG. 12A) and diffuse reflection (FIG. 12B) when a sphere is displayed as the article 3S. After the pre - shooting in step S02 of FIG. 4, only the reflection characteristics of the article 3S are changed from specular reflection to diffuse reflection, so the real space 9 is the same and the brightness distribution data is also the same. In the specular reflection of FIG. 12A, the brightness of the real space 9 reflected on the surface of the article 3S is clearly observed. This specular reflection is a case where a surface having high reflectivity and smoothness like a metal surface is set. In contrast, in the diffuse reflection of FIG. 12B, each part of the surface of the article 3S is displayed only as a surface having a luminance corresponding to the illuminance with which the illumination by the brightness distribution data illuminates each part. This is a case where a bright diffuse reflection surface such as white or gray is set on the surface. Also, in FIGS. 12A and B, although the background image 42 is the video pass - through video of the external camera 108, the brightness distribution data for illuminating the article 3S is that of the pre - shooting.

[0046] Through the above - mentioned examples, it was confirmed that the design support system 1 can bring a high sense of reality so as to support the design evaluation of the operator 2 by fusing the real space 9 and the virtual space, and that the expressions that the articles reproduced in the virtual space can show are sufficiently diverse.

[0047] 3. Function The more detailed functions of the design support system 1 of the present disclosure will be further described additionally. 3 - 1. Deformation, grouping of parts In the design support system 1, the advantage of using a virtual space for the design of the article 3 is utilized. One of them is a deformation function that makes it easier for the operator 2 to perform an intuitive deformation on the article 3. FIGS. 13A to 13C are explanatory diagrams showing the operation of the deformation function in the design support system according to the embodiment of the present disclosure. When using this function, a bounding volume or a bounding box 7 that encloses the article 3 is set by a bounding box setting unit 560 (FIG. 2). The bounding box 7 is divided into a grid (lattice) shape and has a plurality of grip points 72 on the lattice points on the surface that can be gripped by the operator 2 by operating the operation input unit 120. As shown in FIG. 13B, the bounding box 7 is, for example, a prism that encloses the article 3 before deformation. The grip points 72 are set, for example, at the internal division points that divide each side forming the prism into three equal parts and at the intersections of a plurality of straight lines that connect the internal division points to each other on the surface of the prism. In the bounding box 7, when the operator 2 operates the operation input unit 120 to displace any one of the grip points 72 to a gripped state, for example, by dragging, the grip point 72 is displaced accordingly. For example, those of the grip points 72 that are not gripped do not move. At this time, the spatial coordinates between the plurality of grip points 72 can be enlarged or reduced while maintaining the internal division ratio. As a result, the coordinates inside the bounding box 7 also move (FIG. 13C). Such coordinate displacement is calculated by a coordinate displacement unit 562 (FIG. 2). The coordinates referred to by the structural data 512 are displaced according to the displacement by the coordinate displacement unit 562. In this way, the structure of the article 3 that refers to at least a part of the coordinates within the bounding box 7 in the virtual space is deformed. This deformation is executed by a structure deformation unit 564 (FIG. 2). By causing the article image rendering unit 550 to display the structural data called from the article data recording unit 510 corresponding to the coordinates displaced by the structure deformation unit 564, an article image having a deformed structure like the article 3M in FIG. 13C is generated.

[0048] With such a function, the operator 2 can deform the article 3 in the virtual space as if directly touching clay work. The fact that the appearance shown by the article 3 in the real space 9 can be reproduced with a high sense of reality means that the shape can be changed while perceiving the final appearance, thus realizing support that takes advantage of the virtual space for advanced design work.

[0049] In addition, in the design support system 1, measures are also taken to more easily realize the above-described deformation. In particular, the above deformation function is useful when the article 3 has a structure assembled from a plurality of parts. When the article 3 includes a plurality of parts (not shown), the article data recording unit 510 stores part structure data 516 (FIG. 2) for determining the structure of the parts. Then, the structure deformation unit 564 displaces the coordinates referred to by the part structure data 516 for the plurality of parts in accordance with the displacement by the coordinate displacement unit 562, thereby grouping and deforming the plurality of parts. When the article 3 is assembled or combined from a plurality of parts, not only the deformation of one part but also the other parts combined with it are deformed at the same time. This not only omits the deformation work for the plurality of parts, but also enables the appropriate maintenance and deformation of the relationship between the parts. As a result, the accompanying work required due to the deformation, such as the need to reconsider the assemblability, for example, in response to the deformation, is reduced. In this way, the design support system 1 can support the correction of the shape of the article affected by the structure in the design work.

[0050] 3-2. Distance measurement of article, cross-sectional display of article In the design support system 1, it is also preferable to devise a way to support the design evaluation of an article in more detail. That is, in the design support system 1, it is also preferable to support the design evaluation by realizing a function of measuring a physical distance on the surface of the article 3 and a function of checking the cross section of each part of the article. FIGS. 14A to 14B are captured images of the display of the headphone display in the design support system according to the embodiment of the present disclosure, showing a state of using the distance measuring function for measuring the distance (FIG. 14A) and a state of using the cross section display function (FIG. 14B). The design support system 1 further includes a controller attitude sensor 124 (FIG. 2), a point position determination unit 572, and a distance calculation unit 574. The controller attitude sensor 124 detects the attitude of the operation input unit 120 itself. The point position determination unit 572 determines the attention position based on the intersection of the virtual line and the surface of the structure in the virtual space determined according to the output of the controller attitude sensor 124. When there are a plurality of attention positions determined by the point position determination unit 572, the distance in the real space corresponding to two of the attention positions, for example, the distance between the pointer displays P1 and P2 shown in FIG. 14A, is calculated by the distance calculation unit 574. The image integration unit 580 includes, in the presentation image, a pointer display indicating the attention position updated in the virtual space according to the output of the controller attitude sensor 124 and the numerical value of the distance by the distance calculation unit 574 in order to show the operator 2 which distance between the attention positions is being measured. In FIG. 14A, the pointer displays P1 and P2 and the numerical value of the distance are shown.

[0051] For the cross-sectional display of an object, it is preferable that the design support system 1 includes the controller attitude sensor 124 and the intersection line determination unit 576 (Fig. 2). The intersection line determination unit 576 determines the intersection line between the virtual plane in the virtual space determined according to the output of the controller attitude sensor 124 and the surface of the structure. When generating an image signal to indicate to the operator 2 the cross-section cut on which virtual plane, the image integration unit 580 includes, in the presentation image, a plane display indicating the virtual plane updated in the virtual space according to the output of the controller attitude sensor 124 and a display of the cross-section with the intersection line by the intersection line determination unit 576 as a contour or a cutting line. Fig. 14 shows a plane with a grid that is the plane display and the contour.

[0052] 3-4. Background other than the real space In a design evaluation such as the design support system 1, although there is an advantage that the design evaluation can be performed with a high sense of reality, there is also a need to use a virtual space rather than the real space 9. In the design support system 1 of the present disclosure, in order to meet such needs, the brightness distribution data providing the object image 44 can be alternatively displayed with something other than the real space where the operator 2 is located, such as the brightness distribution data in the direction constructed for the virtual space, the ambient environment image taken in advance, or the ambient environment image taken at any facility or position and transmitted through the network. In this case, the brightness distribution data is rewritten with such an alternative. Thereby, the operator 2 can evaluate the appearance design of the object 3, for example, in VR (virtual space) or in the ambient environment image used as a reference, and can suppress the variation in evaluation and reconfirm his / her own evaluation criteria. Figs. 15A - B are captured images of the display of the headset in the design support system of the embodiment of the present disclosure, showing an example of displaying an image of the real space 9 from the external camera for the background image and an example of displaying an image of the virtual space 900 for the background. Also in the virtual space 900 for the background, lighting is arranged, and its reflection is reproduced on the glass surface and the painted surface of the automobile which is the object 3.

[0053] 3-5. Network It is also preferable that the design support system 1 of the present disclosure is configured to utilize a computer network. As described in relation to FIG. 8, design evaluation by a plurality of operators is itself preferable. Furthermore, taking advantage of the benefits of a computer network, it is also preferable to support a plurality of operators 2 who are geographically separated from each other to collaborate via the network. FIG. 16 is an explanatory diagram showing an example configuration in a situation where a plurality of operators according to an embodiment of the present disclosure collaborate via a network.

[0054] In the present disclosure, a design support system 1N for network use is provided. The design support system 1N is a further improvement of the system shown in the design support system 1M (FIG. 8). The design support system 1N further includes a plurality of image adjustment units for each of a plurality of operators. An image adjustment unit for one of the plurality of operators generates attribute data for an attribute selected or adjusted from attribute candidate data according to the instruction content of an operation input for selecting or adjusting an attribute of an article received by an operation input unit for another one of the plurality of operators. In a typical configuration, a computer device used by an administrator who is one of the plurality of operators includes an account management unit and a session management unit. The account management unit includes a permission table that determines what functions and data files of the design support system each user can access. The session management unit holds a list of users who can participate in a session as a session table, determines whether each of the plurality of operators is a legitimate user by referring to the session table, and establishes a connection for a session for the legitimate users among the plurality of operators. Also, in this typical configuration, when the computer device used by a user is a client terminal when the computer device used by one operator is a server device, it can communicate through a network, and it is preferable that a user ID for identifying the user is used through the network for establishing a session.

[0055] To implement and operate the design support system 1N as a program on computer devices such as personal computers (PCs), each operator activates an execution program for realization on their own PC. Therefore, each of the PCs 50A to 50C becomes a part of the design support system 1N through the network 68. Specifically, operator A activates the server execution unit 60, which may be an execution program in the arithmetic unit, on the PC 50A that they operate. Similarly, operators B and C activate the client execution unit 66, which may be an execution program in the arithmetic unit, on the PCs 50B and 50C that they operate. The PCs 50A to 50C of each operator can communicate with each other via an appropriate protocol (e.g., TCP-IP) through a public line such as the Internet or a dedicated line.

[0056] The client execution unit 66 operates the PCs 50A to 50C used by operators A to C as client terminals in a specific design review session (hereinafter referred to as "session") by a design review network community. In contrast, the server execution unit 60 operates the PC 50A used by operator A as a server device in the session. The function as a server device is realized by implementing the account management unit 602 and the session management unit 604. The server execution unit 60 is typically implemented as a function of a server execution program, and the account management unit 602 and the session management unit 604 are implemented as its functions. Operator A executes the server execution unit 60 to set the functions of the design support system 1N as an administrator.

[0057] The account management unit 602 implements a function to manage the accounts of users who can access the necessary resources of the design support system 1N. The account management unit 602, for example, the resources of the design support system 1N are typically functions and data files used in the design support system 1N, and the function of managing accounts may include setting usage rights and access rights (permissions) for these resources. The permission table (not shown) records the permission levels of each resource for each user account, and the account management unit 602 refers to the permission table to determine, for example, whether the operator who logs in is a user who has an account in the design support system 1N. Specifically, operators B and C who operate their own PCs 50B and C functioning as client terminals log in with their own accounts by means such as password authentication, and can further access resources within the range permitted by the pre-set permissions of functions and files. The files are stored, for example, in the data server 50S. The account management unit 602 refers to the permission table and executes such login management and permission management for functions and data files. For example, operators A to C are given unique user IDs, and for each user ID, the permissions for functions and data files are managed in the permission table.

[0058] The session management unit 604 controls whether a user participates in a session. By referring to a session table (not shown) which is a list recording which users participate in each session, the session management unit 604 determines whether a connection request from the client execution unit 66 and subsequent communications are legitimate connections and controls participation in the session. In this way, when operators A to C are legitimate users in the session, they can participate in the session by the design review network community where the members have been determined in advance.

[0059] The identification of users executed by the account management unit 602 and the session management unit 604 is typically performed by a user ID corresponding to the account, and security means such as passwords, which are used by those skilled in the art for security purposes, are also adopted together.

[0060] Computers 50A to 50C and data server 50S can implement functions for operators A to C to manage the functions and settings of the design support system 1N and files. The settings for these functions and files include, for example, colors and avatars for distinguishing oneself from others in collaborative work. Colors and avatars are used for display in appropriate situations in association with the user ID as needed. Thereby, operators A to C can wear the VR headset 10 and visually recognize the activities of others in the virtual space, for example, by the user ID.

[0061] In the design support system 1N that uses the above computer network, operators 2 who are located remotely from each other can simultaneously execute evaluations from the perspectives of each operator 2 regarding a common article 3. When the article 3 is a product such as an automobile that can recover investments by mass-producing on a large scale and shipping to destinations around the world, a process of carefully determining the design from various perspectives is essential. By adopting the design support system 1N for such applications, the efficiency of design evaluation can be enhanced. In FIG. 16, the participants are operators 2, and all are participating in the design evaluation while wearing headsets 10. Such operators 2 can conduct design reviews and design clinic activities without being aware of geographical distances. Also, as shown in FIG. 9, it is not always necessary for all to be operators wearing headsets 10. The presentation image (presented to both or either of the left and right eyes) presented to any one of the operators 2 can simply be recorded as a video image or presented to an external display device, and for example, while showing the presentation image from the perspective of one of the operators (Operator A) to the observing visitors (Observer C, D) present, allowing them to observe the progress of the design work can also be useful in the mode of using the computer network.

[0062] 3- 6 .Point of fixation In the design support system 1 of the present disclosure, it is preferable to detect and record the fixation point. The line of sight of both eyes of the operator 2 is defined as a straight line starting from the position of the eyeball (viewpoint) in the real space and ending at the fixation point. Since this fixation point is the position where the image is formed at the position with the highest resolution of the eyeball, it indicates the position on which the operator 2 focuses at each moment. FIG. 17 is an explanatory diagram of the headset 10A including the eye tracking detection unit 110 employed for detecting and recording the fixation point, and FIG. 18 is a block diagram of the design support system 1A employing the headset 10A. For the headset 10A and the design support system 1A, the elements denoted by the same reference numerals are the same as the explanations for the headset 10 and the design support system 1. The headset 10A includes an eye tracking detection unit 110 for specifying the line of sight of at least one of the eyeballs of the operator 2. The eye tracking detection unit 110 is realized, for example, by a combination of an infrared camera and an infrared light source, and specifies the straight line connecting the line of sight of the operator 2's eyeball, that is, the fixation point being focused on and the eyeball, by a method such as the corneal reflection method, the bright pupil method, or the dark pupil method. Such an eye tracking detection unit 110 can further detect the blinking and opening / closing of the eyelids. The eye tracking signal 112, which is the output from the eye tracking detection unit 110, is received by the computer 50 via an appropriate interface and input to the fixation point determination unit 582. The fixation point determination unit 582 determines the line of sight of the operator 2 in the virtual space using the eye tracking signal 112. In the fixation point determination unit 582, using the line of sight, the fixation point on the three-dimensional object surface of the article 3, the fixation point in the article image 44, and the fixation point in the background image 42 are determined based on the structure data 512 and the brightness distribution data 522. This fixation point can be determined, for example, by an algorithm such as a so-called collision determination algorithm. One of the other fixation point determination methods that can be employed for the processing in the fixation point determination unit 582 is the line of sight intersection method for determining the intersection point where the left and right line of sight directions converge. Note that this fixation point can also be used to update the background image 42. Data 586 of the fixation point position is recorded in the fixation point position recording unit 584.The data 586 of the fixation point position is used for the processing by the fixation point aggregation unit 588.

[0063] 4. Variation: It is also useful to make various changes to the above-described design support systems 1, 1N, 1M, and 1A. 4-1. Auxiliary Image It is useful that the above-described design support systems 1, 1N, 1M, and 1A further include an auxiliary image rendering unit 534 (FIG. 2) that generates an auxiliary image. At this time, the image integration unit 580 synthesizes the auxiliary image with the background image 42 and the article image 44 and outputs it as an image signal for the presentation image. The auxiliary image can be various information presentation images or 3D objects. For example, if a menu display (illustrated in FIGS. T0B, 11A, B, 15A, B) can be displayed when the operator 2 performs an operation by the operation input unit 120, the operator 2 can easily select and adjust the attribute candidates. Also, in the design evaluation work, a marker that can be left as a sticker on the surface of the article 3 can also be one of these auxiliary images. Furthermore, guidance information for guiding the design evaluation of the operator 2, such as informing the viewpoints and attention points for evaluating the design, is also useful as an auxiliary image. Also, a plane (FIG. 14B) having a measure display and scales for length measurement, which is used together with the above-described distance calculation unit 574 and intersection determination unit 576, can also be an auxiliary image. In addition, if the article 3 to be designed is for investigating the relationship with the body of a user (driver, passenger) represented by an automobile, an auxiliary image such as a 3D object of a persona (mannequin) is also useful. From a different perspective, when the operator 2 performs a design evaluation, display of these images and objects can also be an auxiliary image when creating a sense of presence, for example, reproducing a rain or snow situation or showing the state of falling dead leaves. Thus, the auxiliary image can be very useful in the activity of design evaluation.

[0064] The auxiliary image may be generated or prompted by an operation of the operation input unit 120, or may be configured to be displayed when meeting predetermined conditions. For example, when the gaze point coincides with a feature part to which an intentional explanation is to be added among articles by eye tracking, it is also useful to automatically display an explanatory image or characters with this coincidence as a condition for establishment. Further, when the posture sensor mover 104 and the posture sensor stator 106 detect an operation in which the operator 2 tilts the head 22, a configuration in which an auxiliary image is displayed based on this is also useful.

[0065] One of the viewpoints of design evaluation that may be specific when the article 3 is an automobile is the beauty angle. The beauty angle is a viewpoint (observation position) that matches the design intention specified by the relative angle with the article 3. That is, when the designer has already completed the design of the article 3 once, the designer often assumes a viewpoint for evaluation as the relative angle with the article 3, and may specify that angle as the beauty angle. In such a case, it is also useful to present information serving as a guide for the operator 2 to easily identify the viewpoint position as an auxiliary image.

[0066] Furthermore, it is also useful to display the trajectory of the movement of the operator 2 by, for example, a spline curve or the like, or record it. How the operator 2 actually observes the article 3 from what viewpoints for design evaluation serves as an indicator of whether the design evaluation by the operator 2 is reliable, or a clue when verifying the evaluation result afterwards, and thus becomes reference information when giving a post - facto interpretation to the evaluation result.

[0067] Furthermore, it is also useful to generate an object of a person (persona) by computer graphics in the virtual space for Operator 2 and guide the design work as some kind of character. For example, by using technologies such as lip sync, a person who talks to assist the design work can be presented to Operator 2 who is performing the design work from outside the design support system 1 or the like, enabling a smooth design work to be carried out. Also, when multiple Operators 2 are evaluating the article 3 as in the design support systems 1M and 1N, it is also useful to assist the interaction between the Operators 2 with each other by voice or the like. Also in that case, by showing the personas to each other, even if only one Operator 2 is present on the spot in the real space, the other Operator 2 participating in the evaluation of the article 3 can be recognized in a very natural manner. In design evaluation, if the collaborative work evaluated by multiple evaluators can be advanced in a natural feeling, the design evaluation work of Operator 2 is greatly supported. Incidentally, in the case where multiple Operators 2 participate, a simpler symbol display may be sufficient to recognize the other Operators 2. For example, even if only a headset (such as an HMD) is generated by computer graphics, if its posture (orientation, position) is synchronized with that of the headset 10 actually worn by the other Operator 2, the Operator who can see it can easily recognize the other Operator.

[0068] 4-2. Dialogue Assistance In design support systems 1, 1M, 1N, and 1A, it is also an auxiliary function with good immersion and compatibility with the operator 2's design work for multiple operators 2 to participate, for another participant different from the operator 2 to conduct oral communication or text-based dialogue with the operator 2, or to incorporate artificial intelligence technology to interact with the computer. Such dialogue assistance functions typically include a voice chat function and a translation function for assisting the dialogue. The voice chat function can be implemented by equipping the headset 10 with a speaker or earphone for emitting the voice of the other party and a microphone for picking up one's own speech, and conducting voice communication through the computer 50 or computer network using any protocol. Also, the translation function can be implemented by adopting a similar headset 10 and further combining a voice recognition engine, a translation engine using any method such as deep learning, and a voice synthesis engine.

[0069] 4-3. Measurement of Physical Condition It is effective for the purpose of detecting or recording the non-verbal psychological changes of the operator 2 to obtain physical / brain waves, pulse, skin potential, electrocardiogram, etc. from the operator 2 who is conducting a design evaluation. For this purpose, in the design support systems 1, 1M, 1N, and 1A, as shown in FIG. 2, a biological state sensor 140 can be added. The signal (biological vital signal) from the biological state sensor 140 directly reflects the physical state of the operator 2 and is transmitted to the computer 50 through the signal processing unit 142. The computer 50 stores the biological information data 592 corresponding to the signal from the biological state sensor 140 in the subject information recording unit 590 in the form of time-series data, for example, by associating it with an appropriate time stamp. This biological information data 592 can include brain waves, electrocardiograms, myoelectric potentials, etc. By analyzing these signals in association with the data 586 of the gaze point position recorded in the gaze point position recording unit 584, for example, it is also useful to collect non-verbal information in the design evaluation and to judge the reliability of the evaluation by investigating afterwards what psychological state the operator 2 had with respect to which part of the article 3.

[0070] 4-4. Program Excluding the headset 10, related sensors, and the controller (operation input unit 120, attitude sensors 104, 106) (hereinafter referred to as headset-related devices), the hardware configurations of the above-described design support systems 1, 1M, 1N, and 1A are a general configuration personal computer or workstation with a computer 50 having a certain degree of graphic performance. Therefore, by preparing a headset-related device and a corresponding computer device and setting up network equipment as necessary, the design support systems 1, 1M, 1N, and 1A can be constructed by using a computer program for configuring them. For this reason, the above-described design support system 1 can also be realized by operating such a computer program for the purpose.

[0071] The embodiments of the present disclosure have been specifically described above. The above-described embodiments, variations, and examples are described for explaining the invention disclosed in this application, and the scope of the invention of this application should be determined based on the description of the claims. Variations within the scope of the present disclosure including other combinations of embodiments are also included in the claims.

Description of Reference Numerals

[0072] 1, 1M, 1N Design Support Systems 10, 10A Headsets 102 Display 104 Attitude Sensor Mover 106 Attitude Sensor Fixer 108 External Camera 108A Hand Tracking Camera 110 Eye Tracking Detection Unit 112 Eye Tracking Signal 120 Controller (Operation Input Unit) 124 Controller Attitude Sensor 130 Depth Measurement Unit ​140 Biosensor (biological state detection device) 142 Signal processing unit 2 Operator 3, 3M, 3S Items 4, 4L, 4R Prompt images 42 Background image 44 Item image 46 Foreground image 48 Camera viewing angle 50 Computer 50A - C Personal computer 52L, 52R Transceiver 510 Item data recording unit 512 Structure data 514 Attribute candidate data 516 Component structure data 520 Ambient light recording unit 522 Brightness distribution data 530 Background image rendering unit 532 Background update processing unit 534 Auxiliary image rendering unit 540 Image adjustment unit 542 Attribute data 550 Item image rendering unit 560 Bounding box setting unit 562 Coordinate displacement unit 564 Structure deformation unit 570 Foreground image rendering unit 572 Point position determination unit 574 Distance calculation unit 576 Intersection line determination unit 580 Image integration unit 582 Fixation point determination unit 584 Fixation point position recording unit 586 Fixation point position data 588 Fixation point aggregation unit 590 Subject information recording unit 592 Biological information data 60 Server execution unit 66 Client execution unit 602 Account management unit 604 Session Management Unit 68 Network 7 Bounding Box 72 Grip Point 9, 9A, 9B Real Space 92 Lighting 900 Virtual Space for Background

Claims

1. An article data recording unit that stores structure data for determining the structure of at least one article in a virtual space and attribute candidate data for selecting or adjusting attributes for the article, At least one headset for the operator to wear on the head, comprising a posture sensor capable of detecting the posture of the headset in the real space, at least one display capable of displaying a presented image to the operator, and at least one external camera capable of photographing at least a part of the external world corresponding to the operator's field of view, An ambient light recording unit that records, in association with the output of the posture sensor, brightness distribution data indicating brightness by direction around the operator photographed by the external camera in the pre-photographing, A background image rendering unit that generates a background image corresponding to the output of the posture sensor at each time point from the brightness distribution data called from the ambient light recording unit, At least one operation input unit that receives an operation input from the operator for selecting or adjusting the attribute of the article, An image adjustment unit that generates attribute data for the attribute selected or adjusted from the attribute candidate data according to the instruction content of the operation input received by the operation input unit, An article image rendering unit that generates an article image under illumination corresponding to the brightness distribution data, corresponding to the instruction content of the operation input and the output of the posture sensor at each time point, from the structure data called from the article data recording unit, the attribute data from the image adjustment unit, and the brightness distribution data called from the ambient light recording unit, An image integration unit that synthesizes the background image and the article image and outputs it as an image signal for the presented image A design support system comprising.

2. The brightness distribution data indicates the brightness by direction obtained by photographing with the external camera while changing the orientation of the headset in the pre-photographing. The design support system according to Claim 1.

3. The design support system according to Claim 1, further comprising a background update processing unit that replaces data corresponding to at least a part of the field of view for the operator wearing the headset with brightness data at each time point obtained by photographing with the external camera in the brightness distribution data. The design support system according to Claim 1.

4. The object image rendering unit reflects the surrounding reflected image corresponding to the brightness distribution data on the surface reflection of the object image. The design support system according to claim 1.

5. The background image rendering unit calls the structure data from the object data recording unit and reduces the brightness of the background image in the direction of the shadow that the object should cast in the real space corresponding to the brightness distribution data. The design support system according to claim 1.

6. The ambient light recording unit records, in association with the output of the attitude sensor, color brightness data indicating the brightness by color photographed by the external camera in pre-shooting. The background image rendering unit generates a color background image corresponding to the output of each time point of the attitude sensor from the color brightness data called from the ambient light recording unit. The object image rendering unit generates an object image under illumination corresponding to the color brightness data, corresponding to the instruction content of the operation input and the output of each time point of the attitude sensor, from the structure data called from the object data recording unit, the attribute data from the image adjustment unit, and the color brightness data called from the ambient light recording unit. The design support system according to claim 1.

7. A depth ranging unit capable of measuring the distance to an object in each direction of the real space, A foreground image rendering unit that generates a foreground image based on the output of the external camera in a direction where the distance to the object indicated by the depth ranging unit is smaller than the distance to the object in the virtual space, further comprising: The image integration unit outputs an image signal for the presentation image by additionally synthesizing the foreground image with the background image and the object image. The article design support system according to claim 1.

8. The object data recording unit stores the structure data and the attribute candidate data for each of a plurality of objects, The operation input unit receives from the operator operation inputs for selecting or adjusting the attributes for the plurality of objects and display instruction inputs for whether or not to display each object or the display mode. The image adjustment unit generates attribute data for the attribute selected or adjusted from the attribute candidate data for the items among the plurality of items that require display according to the instruction contents of the operation input received by the operation input unit and the display instruction input. The item image rendering unit generates item images under illumination corresponding to the brightness distribution data corresponding to the instruction content of the operation input and the output of the posture sensor at each time point for the plurality of items. The item design support system according to claim 1.

9. In the headset, the at least one display forms a pair corresponding to each of the operator's both eyes. In the headset, the at least one external camera forms a pair corresponding to each of the operator's both eyes. The item image rendering unit, the background image rendering unit, and the image integration unit are provided in two systems per headset. The design support system according to claim 1.

10. In the headset, the at least one display forms a pair corresponding to each of the operator's both eyes. In the headset, the at least one external camera forms a pair corresponding to each of the operator's both eyes. The item image rendering unit, the background image rendering unit, the foreground image rendering unit, and the image integration unit are provided in two systems per headset. The design support system according to claim 7.

11. A controller posture sensor that detects the posture of the operation input unit itself, A point position determination unit that determines a focus position based on the intersection of a virtual line in the virtual space determined according to the output of the controller posture sensor and the surface of the structure, A distance calculation unit that calculates the distance in the real space corresponding to two of the plurality of focus positions determined by the point position determination unit And further includes. The image integration unit generates the image signal by including in the presentation image a pointer display indicating the focus position updated in the virtual space according to the output of the controller posture sensor and a numerical value of the distance by the distance calculation unit. The design support system according to claim 1.

12. A controller attitude sensor that detects the attitude of the operation input unit itself, An intersection line determination unit that determines an intersection line between a virtual plane in the virtual space determined according to the output of the controller attitude sensor and a surface of the structure is further provided, The image integration unit includes, in the presentation image, a plane display showing the virtual plane updated in the virtual space according to the output of the controller attitude sensor and a display of a cut surface having the intersection line by the intersection line determination unit as a contour or a cutting line, and generates the image signal. The design support system according to claim 1.

13. A bounding box setting unit that sets a bounding box having a plurality of grip points on its surface, which is defined to enclose the article in the virtual space and can be gripped by an operator's operation using the operation input unit, A coordinate displacement unit that displaces at least a part of the coordinates inside the bounding box in correspondence with an operation in the operator's gripping state with respect to any one of the grip points, A structure deformation unit that deforms the structure that refers to at least a part of the coordinates within the bounding box in the virtual space by displacing the coordinates referred to by the structure data in accordance with the displacement by the coordinate displacement unit is further provided, The article image rendering unit generates the article image having the structure after deforming the structure data called from the article data recording unit in correspondence with the coordinates displaced by the structure deformation unit. The design support system according to claim 1.

14. The article data recording unit stores part structure data for determining the structures of a plurality of parts that make up the article, The structure deformation unit groups and deforms the plurality of parts by displacing the coordinates referred to by the part structure data for the plurality of parts in accordance with the displacement by the coordinate displacement unit. The design support system according to claim 13.

15. It further includes an auxiliary image rendering unit that generates an auxiliary image The image integration unit synthesizes the auxiliary image with the background image and the article image and outputs it as an image signal for the presentation image. The design support system according to claim 1.

16. A plurality of the headset devices, each worn by one of a plurality of operators, A plurality of the ambient light recording units for each of the plurality of operators; A plurality of the background image rendering units for each of the plurality of operators; A plurality of the article image rendering units for each of the plurality of operators; A plurality of the image integration units for each of the plurality of operators are provided, each ambient light recording unit records, in association with the posture of the headset, brightness distribution data indicating brightness in each direction around each operator photographed by the external camera in pre-shooting; each background image rendering unit generates a background image corresponding to the output at each time point of the posture sensor of each operator in the real space of each operator; each article image rendering unit generates an article image under illumination corresponding to the brightness distribution data of each operator, corresponding to the output at each time point of the posture sensor of each operator; each image integration unit synthesizes the background image and the article image and outputs an image signal for the presentation image for each operator; on the display of the headset worn by each operator, the presentation image for each operator is displayed The design support system according to any one of claims 1 to 15.

17. each ambient light recording unit stores brightness distribution data indicating brightness in each direction around each operator photographed by the external camera of each operator while changing the orientation of the headset in pre-shooting. The design support system according to claim 16.

18. A plurality of background update processing units for each of the plurality of operators are provided, each background update processing unit replaces data corresponding to at least a part of the visual field for each operator with brightness data at each time point acquired by photographing with the external camera. The design support system according to claim 16.

19. A plurality of depth ranging units for each of the plurality of operators capable of measuring the distance to an object in each direction of the real space, and a foreground image rendering unit for each of the plurality of operators are further provided, wherein the foreground image rendering unit generates a foreground image based on the output of the external camera in a direction in which the distance to the object indicated by the depth ranging unit is smaller than the distance to the article in the virtual space. The image integration unit adds and composites the foreground image to the background image and the article image to output an image signal for the presentation image of each operator. The design support system according to claim 16.

20. It further includes a plurality of the image adjustment units for each of the plurality of operators, The image adjustment unit for one of the plurality of operators generates attribute data for the attribute selected or adjusted from the attribute candidate data according to the instruction content of the operation input for the selection or adjustment of the attribute of the article received by the operation input unit for the other operator among the plurality of operators. The design support system according to claim 16.

21. The ambient light recording unit for one of the plurality of operators records the brightness distribution data for the other operator among the plurality of operators, The background image rendering unit and the article image rendering unit for the one operator generate the background image and the article image for the one operator based on the brightness distribution data for the other operator in the ambient light recording unit for the one operator. The design support system according to claim 16.

22. It further includes a background update processing unit for the one operator, The background update processing unit replaces the data corresponding to at least a part of the visual field for the one operator wearing the headset among the brightness distribution data for the other operator in the ambient light recording unit for the one operator with the brightness data obtained by photographing with the external camera of the other operator. The design support system according to claim 21.

23. The computer device used by a manager who is one of the plurality of operators includes an account management unit and a session management unit, The account management unit includes a permission table that determines what each user can access regarding the functions and data files of the design support system. The session management unit holds a list of users who can participate in a session as a session table, determines whether each of the plurality of operators is a legitimate user by referring to the session table, and establishes a connection for the session for the legitimate users among the plurality of operators. The design support system according to claim 16.

24. When the computer device used by the user serves as a client terminal when the computer device used by the one operator is a server device, it can communicate through a network. The user ID for identifying the user is used for establishing the session through the network. The design support system according to claim 23.

25. A computer including an arithmetic unit, a recording device, and a graphic device, At least one headset connected to the computer, the headset being worn on the head by an operator, the headset including a posture sensor capable of detecting the posture of the headset in the real space, at least one display capable of displaying a presented image to the operator, and at least one external camera capable of photographing at least a part of the external world corresponding to the field of view of the operator. At least one operation input unit connected to the computer and receiving an operation input from the operator A computer program that operates a computer system including the above to realize the design support system according to claim 1, Causing the arithmetic unit, the recording device, and the graphic device of the computer to function as the article data recording unit, the ambient light recording unit, the background image rendering unit, the image adjustment unit, the article image rendering unit, and the image integration unit. Computer program.

Images