Video output method, video output system, and program

Abstract

A video output method for performing display pertaining to one space in a facility having a real space having a plurality of spaces on an augmented reality space obtained by expanding the real space, the video output method comprising: acquiring status information indicating the status of each of the plurality of spaces (S10); in the present invention, a first space in a real space, which is a space in which a user (U) is currently located among a plurality of spaces, is specified (S30), a second space to be connected in the real space, which is a space in which a predetermined state is currently located among the plurality of spaces, is determined on the basis of state information for each of the plurality of spaces (S40), and presentation information is output. This presentation information is information for presenting to the user (U) via an XR device (10) worn by the user (U), the XR device being a cross-reality device (S60), an image in which the determined second space and the first space are connected in the augmented reality space.

Description

Technical Field

[0001] The present disclosure relates to an image output method, an image output system, and a program. Background Art

[0002] Conventionally, a multimedia communication system has been used in which data such as sound and images are transmitted and received between a plurality of communication devices for the purpose of mutually communicating the appearance of two geographically separated places by means of images and sounds. For example, Patent Document 1 discloses a multimedia communication system that can give a user a feeling that a plurality of geographically separated spaces are adjacent spaces.

[0003] (Prior art literature)

[0004] (Patent Document)

[0005] Patent Document 1: Japanese Patent Application Publication No. 2004-56161 Summary of the invention

[0006] Problems to be solved by the invention

[0007] Here, the user sometimes wants to confirm the appearance of a space in a predetermined state in a facility. In this case, the user hopes to be able to easily confirm the appearance of the space in the predetermined state. However, Patent Document 1 does not disclose a technology that allows the user to easily confirm the appearance of the space in the predetermined state.

[0008] Therefore, the present disclosure provides an image output method, an image output system, and a program that enable a user to easily check the appearance of a space in a predetermined situation.

[0009] Means for solving problems

[0010] An image output method involved in one embodiment of the present invention is an image output method for displaying a space related to a facility of a real space having multiple spaces on an extended real space, wherein the extended real space is a space that extends the real space. In the image output method, status information showing the status of each of the multiple spaces is obtained, a first space in the real space is determined, and the first space is a space where a user is currently located in the multiple spaces. Based on the status information of each of the multiple spaces, a second space in the real space that is a connection object is determined, and the second space is a space in the multiple spaces that is currently in a specified status. Prompt information is output, and the prompt information is information used to display an image that connects the determined second space and the first space on the extended real space to the user via an XR device worn by the user, and the XR device is a cross-reality device.

[0011] An image output system involved in one embodiment of the present invention is an image output system for displaying a space related to a facility of a real space having multiple spaces in an extended real space, wherein the extended real space is a space in which the real space is extended, and the image output system comprises: an acquisition unit, which acquires status information showing the status of each of the multiple spaces; a determination unit, which determines a first space in the real space, wherein the first space is a space in which a user is currently located among the multiple spaces; a determination unit, which determines a second space in the real space to be a connection object based on the status information of each of the multiple spaces, wherein the second space is a space in the multiple spaces that is currently in a specified status; and an output unit, which outputs prompt information, wherein the prompt information is information for prompting an image connecting the determined second space and the first space in the extended real space to the user via an XR device worn by the user, wherein the XR device is a cross-reality device.

[0012] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-mentioned video output method.

[0013] Effects of the Invention

[0014] According to one aspect of the present disclosure, it is possible to realize a video output method or the like that allows a user to easily check the appearance of a space in a predetermined situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram showing an overview of a house layout reconfiguration system according to an embodiment.

[0016] Figure 2 This is a block diagram showing the functional configuration of a server device according to the embodiment.

[0017] Figure 3A This is a diagram showing a first example of a predetermined situation according to the embodiment.

[0018] Figure 3B It is a diagram showing a second example of the predetermined situation according to the embodiment.

[0019] Figure 3C It is a diagram showing a third example of the predetermined situation according to the embodiment.

[0020] Figure 4 This is a flowchart showing the operation of the server device according to the embodiment.

[0021] Figure 5 It is a diagram showing a house layout before reconfiguration according to the embodiment.

[0022] Figure 6 It is shown Figure 4 The flowchart shows the details of the operation of step S40.

[0023] Figure 7 It is a diagram showing a house layout after reconfiguration according to an embodiment. DETAILED DESCRIPTION

[0024] An image output method involved in one embodiment of the present invention is an image output method for displaying a space related to a facility of a real space having multiple spaces on an extended real space, wherein the extended real space is a space that extends the real space. In the image output method, status information showing the status of each of the multiple spaces is obtained, a first space in the real space is determined, and the first space is a space where a user is currently located in the multiple spaces. Based on the status information of each of the multiple spaces, a second space in the real space that is a connection object is determined, and the second space is a space in the multiple spaces that is currently in a specified status. Prompt information is output, and the prompt information is information used to display an image that connects the determined second space and the first space on the extended real space to the user via an XR device worn by the user, and the XR device is a cross-reality device.

[0025] Accordingly, the user can confirm the appearance of the second space through the XR device while still in the first space, in other words, without moving. Moreover, since the XR device worn by the user is used, the user can determine the appearance of the second space from any space. Therefore, through the image output method involved in one embodiment of the present disclosure, the user can easily confirm the appearance of a space in a specified state.

[0026] Furthermore, for example, the status information may include information indicating an operating status of a device installed in the space, and the prescribed status may include the device being in a prescribed operating status.

[0027] This allows the user to easily check the appearance of a space in which the device is in a specified operating state.

[0028] Furthermore, for example, the predetermined operating status may include that the work executed by the device has been completed.

[0029] This allows the user to easily confirm that the job executed by the device has been completed.

[0030] Furthermore, for example, the predetermined operating status may include an abnormality occurring in the device.

[0031] This allows the user to easily check the appearance of the space where an abnormality has occurred in the device.

[0032] In addition, for example, the user may be further asked whether to connect the second space with the first space in the augmented reality space, and when an instruction to connect is obtained from the user, the prompt information is output.

[0033] According to this, since the user can select whether to connect the first space and the second space, the user can easily check the appearance of the space where the device is in a predetermined operating state as needed.

[0034] In addition, for example, the status information may include sensing data of a sensor set in the space, and based on the sensing data, it is determined whether a specified abnormality is occurring in the space, and the specified status includes the specified abnormality occurring in the space.

[0035] This allows the user to easily check the appearance of a space where a predetermined abnormality is occurring.

[0036] Furthermore, for example, the prescribed abnormality may include an abnormality in at least one of temperature, smoke, sound, and smell in the space.

[0037] This allows the user to easily check the state of a space where an abnormality in at least one of temperature, smoke, sound, and odor is occurring.

[0038] Furthermore, for example, the prompt information may include information for forcibly prompting the image to the user via the XR device worn by the user.

[0039] According to this, the user can be forced to confirm that a predetermined abnormality has occurred, so the user can easily confirm that the space is in a dangerous situation.

[0040] In addition, for example, the house layout information may be for showing the house layout of the plurality of spaces in the facility, and the determined second space may be virtually reconfigured next to the first space, and the prompt information may include information for prompting that, when the user in the real space views one side of the virtually reconfigured second space, an image obtained by photographing the second space is overlapped on the first space via the XR device.

[0041] According to this, no matter which space the user is in within the facility, the image of the second space can be superimposed on the first space and presented, so that the user can easily check the appearance of different spaces within the facility.

[0042] In addition, an image output system according to one embodiment of the present disclosure is an image output system for displaying a space related to a facility of a real space having multiple spaces on an extended real space, wherein the extended real space is a space in which the real space is extended, and the image output system comprises: an acquisition unit for acquiring status information showing the status of each of the multiple spaces; a determination unit for determining a first space in the real space, the first space being a space where a user is currently located in the multiple spaces; a determination unit for determining a second space in the real space to be connected based on the status information of each of the multiple spaces, the second space being a space currently in a specified status among the multiple spaces; and an output unit for outputting prompt information, the prompt information being information for prompting an image connecting the determined second space and the first space on the extended real space to the user via an XR device worn by the user, the XR device being a cross-reality device. In addition, a program according to one embodiment of the present disclosure is a program for causing a computer to execute the above-mentioned image output method.

[0043] This can achieve the same effect as the above-mentioned image output method.

[0044] In addition, these general or specific methods may be implemented by a system, method, integrated circuit, computer program, or non-transitory recording medium such as a computer-readable CD-ROM, or by any combination of systems, methods, integrated circuits, computer programs, or recording media. The program may be pre-stored in a recording medium or provided to a recording medium via a wide area network such as the Internet.

[0045] The embodiments will be described in detail below with reference to the drawings.

[0046] In addition, the embodiments to be described below are all general or specific examples. The numerical values, constituent elements, configuration positions of constituent elements, connection forms, steps, and the order of steps shown in the following embodiments are all examples, and their purpose is not to limit the present disclosure. Moreover, among the constituent elements of the following embodiments, the constituent elements that are not recorded in the independent technical solutions are described as arbitrary constituent elements.

[0047] In addition, each figure is a schematic diagram, not a strict illustration. Therefore, for example, the scales in each figure are not necessarily the same. In addition, in each figure, the same symbols are given to substantially the same structure, and repeated descriptions are omitted or simplified.

[0048] In the present specification, numerical values ​​and numerical ranges are not merely expressions of strict meanings but also include expressions indicating substantially equivalent ranges, for example, expressions with a difference of approximately several percent (or approximately 10%) may be included.

[0049] In addition, in this specification, ordinal numbers such as “first” and “second” do not mean the number or order of components unless otherwise specified, but are used for the purpose of distinguishing components to avoid confusion with the same kind.

[0050] (Implementation Method)

[0051] The following is a description of the house layout reconfiguration system according to the present embodiment. Figures 1 to 7 To explain.

[0052] [1. Composition of the house layout reconfiguration system]

[0053] First, regarding the structure of the house layout reconfiguration system involved in this embodiment, refer to Figures 1 to 3C To explain. Figure 1 This is a diagram showing an overview of a house layout reconfiguration system 1 according to the present embodiment.

[0054] like Figure 1 As shown, the house layout reconfiguration system 1 includes an XR device 10 and a server device 20. The house layout reconfiguration system 1 is an image output system (information processing system) for displaying a space (e.g., a second space described later) in a facility (facility in real space) having multiple spaces in a real space (physical space) in an extended real space that extends the real space. For example, the house layout reconfiguration system 1 is for reconfiguring the house layout in a facility in real space (e.g., referring to Figure 5 ) to virtually change the house layout after the virtual change (for example, refer to Figure 7 ) corresponding to an image displayed in an extended reality space. In this manual, virtually changing the layout of a house is also recorded as reconfiguration of the layout of the house (or simply reconfiguration). In addition, the image can be a moving image or a still image.

[0055] The XR device 10 is connected to the server device 20 in a communicable manner. In addition, an example in which the house layout reconfiguration system 1 is used in a house will be described below. Here, it is assumed that the user U is in a space (first space) in the house.

[0056] The XR device 10 is a device used by the user U to realize cross reality, such as a wearable device worn by the user U. The XR device 10 is realized by an AR (Augmented Reality) device such as dedicated glasses. In the present embodiment, the XR device 10 is realized by a glasses-type display (head-mounted display (HMD)) such as AR glasses, but it can also be realized by smart contact lenses (AR contact lenses) and the like. In addition, cross reality is a general term for technologies that combine the real world and the virtual world to perceive things and experiences that do not exist in reality, and also includes technologies such as AR.

[0057] The XR device 10 is an optically transmissive device having a display unit 11 having lenses equivalent to those of ordinary glasses, and enabling the user U to visually confirm the external scene while visually confirming the display on the display unit 11. The display unit 11 is formed of a light-transmitting material, and the display unit 11 is a transmissive display that does not block the field of view of the user U when the image is not displayed. The XR device 10 is configured to display an image of an area (the second space described later) in which the layout of the house in the residence is virtually reconfigured so that the first space is expanded on the display unit 11 as a transmissive display, and real objects (for example, objects in the first space) can be visually confirmed through the display unit 11. The reconfiguration of the house layout will be described later.

[0058] For example, the XR device 10 has a structure that projects the image from the display unit 11 directly to the eyes. In other words, the image is not directly projected on the walls, ceilings, and other construction components of the first space where the user U is located. Therefore, when a person other than the user U is present in the first space, the person other than the user U cannot see the image displayed on the display unit 11 after the house layout is reconfigured, in other words, cannot see the image including the second space.

[0059] The server device 20 is an information processing device for displaying a residence in a real space having multiple spaces in an extended real space. The server device 20 generates an image in which the layout of the house in the residence is virtually reconfigured, and outputs it to the XR device 10. The server device 20 generates an image for connecting a space of a specified condition in multiple spaces to the space where the user U is located in the extended real space, and outputs it to the XR device 10, and the details will be described later. When the status of the equipment or space in the residence of the server device 20 changes, the user U can grasp the status of the space without moving to the space where the status has changed.

[0060] Figure 22 is a block diagram showing the functional configuration of the server device 20 according to the present embodiment. The server device 20 is implemented by, for example, a PC (personal computer). Alternatively, the server device 20 may be implemented by a cloud server.

[0061] like Figure 2 As shown, the server device 20 includes an acquisition unit 21 , a determination unit 22 , a decision unit 23 , a generation unit 24 , and an output unit 25 .

[0062] The acquisition unit 21 acquires information from the XR device 10 and sensors installed in each space (room) in the house. The acquisition unit 21 is configured to include, for example, a communication circuit (communication module). In addition, examples of sensors include temperature sensors, heat sensors, smoke sensors, light and fire sensors, sound sensors, odor sensors, human sensors, or cameras, but are not limited to these.

[0063] The identification unit 22 identifies a first space in the real space, which is a space in a house where the user U is currently located among the plurality of spaces. The first space is a space where the user U is located.

[0064] The determination unit 23 determines a space (second space) of an object connected to the first space currently located by the user U in the augmented reality space from among the plurality of spaces in the house. The determination unit 23 determines the second space based on the information obtained by the obtaining unit 21. The second space is a space different from the first space and is a space currently in a predetermined state.

[0065] The specified condition includes at least one of a specified abnormality occurring and the device being in a specified operating condition. Furthermore, the specified condition may also include an event occurring that is different from the usual. The device is an electrical product (such as a household appliance) present in a residence, such as an induction cooker (such as an IH cooking heater), a microwave oven and other cooking equipment, air conditioners, electric fans or heaters and other cooling and heating air conditioning equipment, televisions, refrigerators, washing machines, etc., but are not limited to these. The device may also be, for example, a device having a battery (such as a lithium-ion battery). Furthermore, the device may also be a socket, etc.

[0066] The specified abnormality refers to a situation where the sensing data (sensing value) from the sensor exceeds a threshold value, for example, the sensing data of at least one of temperature, smoke, sound, and smell exceeds a threshold value. In other words, the specified abnormality includes an abnormality of at least one of the temperature, smoke, sound, and smell in the space. For example, the specified abnormality may include an abnormality of at least one of the temperature, smoke, and smell in the space. Examples of the specified abnormality include fire, abnormal heating of equipment, etc., specified abnormal sounds such as explosions, and specified abnormal smells.

[0067] The occurrence of these specified abnormalities can be determined based on sensing data from sensors such as temperature sensors, smoke sensors, light and fire sensors, sound sensors, odor sensors, or cameras installed in the space. The sensing data is temperature detection data, smoke detection results, sound data, odor detection results, and image data. Image data can be moving images or still images.

[0068] Figure 3A This is a diagram showing a first example of a predetermined situation according to the present embodiment.

[0069] like Figure 3A As shown, the specified anomaly may be a fire in a socket.

[0070] An event that is different from usual means that the space is in a different condition from usual, for example, detecting a sound louder than usual, detecting a person during a time period when there are usually no people, not detecting a person during a time period when there are usually people, detecting smoke in a room where people do not usually smoke, etc.

[0071] The occurrence of events different from these usual ones can be determined based on sensing data from a smoke sensor, a sound sensor, a human sensor, a camera, or the like.

[0072] Figure 3B This is a diagram showing a second example of the predetermined situation according to the present embodiment.

[0073] like Figure 3B As shown, an event different from usual events may refer to the occurrence of smoke from cigarettes or the like in a space where smoking is not usually allowed or a space where smoking is prohibited.

[0074] The prescribed operation status includes a status in which the operation status of the device has changed, for example, a status in which the operation performed by the device has been completed, and at least one of a status in which an abnormality or other unfavorable condition has occurred in the device. The status in which the operation performed by the device has been completed may be, for example, a status in which washing and drying of a washing machine have been completed, a status in which cooking has been completed, or a status in which heating of a microwave oven has been completed. Washing and drying, cooking, heating, etc. are examples of operations.

[0075] The occurrence of these predetermined operating conditions can be identified by information related to the operating condition of the device (for example, information indicating the current state of the device), a temperature sensor, a sound sensor, and the like.

[0076] In addition, the case where the operation status of the equipment changes does not include the mid-stage of a series of operations. For example, when a series of operations of a washing machine are washing and drying, the space connection is not performed at the time of washing completion, and the space connection is performed at the time of drying completion.

[0077] Figure 3C It is a diagram showing a third example of the predetermined situation according to the present embodiment.

[0078] like Figure 3C As shown, the change in the operating status of the device may be that cooking by the IH cooking heater, etc. has been completed.

[0079] Refer again Figure 2 The generating unit 24 generates content information that virtually connects the first space and the second space in the augmented reality space. The content information includes at least image information and may also include sound information. The content information is an example of prompt information to the user U.

[0080] The output unit 25 outputs the content information generated by the generation unit 24 to the XR device 10. The output unit 25 is configured to include, for example, a communication circuit (communication module).

[0081] Furthermore, as described above, the house layout reconfiguration system 1 does not include a projection device such as a projector.

[0082] [2. Work on the housing layout reconfiguration system]

[0083] Next, refer to Figures 4 to 7 The operation of the house layout reconfiguration system 1 constructed as described above will be described. Figure 4 This is a flowchart showing the operation (video output method) of the server device 20 according to the present embodiment. Figure 4 Each step shown in is executed by the server device 20. In addition, at least one of each device and sensor is arranged in each of the plurality of spaces.

[0084] like Figure 4 As shown, the acquisition unit 21 acquires at least one of the operation status of the device and the sensing data from at least one of each device and sensor arranged in each of the plurality of spaces (S10). At least one of the operation status and the sensing data is an example of the status information.

[0085] Here, refer to the attached Figure 5 Describe the house layout of the residence. Figure 5 It is a diagram showing a house layout before relocation according to the present embodiment. Figure 5 The house layout of the house shown is the house layout of the house in the real space. Also, it is assumed that the space where the user U is currently located is a Western-style room (3).

[0086] like Figure 5As shown, the multiple spaces in the house include a living room and a dining room, a kitchen, a western-style room (1) to (3), a toilet, a bathroom, a balcony, a corridor, and a porch. A walk-in closet can be included in the western-style room, for example. In addition, a small house or a garden in the yard of the house can also be included in the multiple spaces.

[0087] In each of the plurality of spaces, for example, at least one sensor (not shown) and at least one of the devices described above are arranged, and at least one of the sensor and the device can communicate with the server device 20 .

[0088] In step S10, from Figure 5 Each of the plurality of spaces in the house shown obtains at least one of an operating status of a device and sensing data.

[0089] like Figure 5 The house layout information shown may be pre-acquired and stored by the server device 20. The house layout information may include location information (eg, latitude, longitude, altitude) of each space.

[0090] Refer again Figure 4 Next, the decision unit 23 determines whether each of the plurality of spaces is in a predetermined state based on at least one of the operating state of the equipment in each of the plurality of spaces and the sensing data of the sensor (S20).

[0091] Next, when the determination unit 23 determines that the space is in a prescribed state (“Yes” in S20), the determination unit 22 determines the first space where the user U wearing the XR device 10 is currently located based on the information obtained by the acquisition unit 21 (S30). For example, when the acquisition unit 21 obtains information about the space where the user U is currently located, the determination unit 22 determines the space indicated by the information as the first space where the user U is currently located. Moreover, when the acquisition unit 21 obtains information including sensing data of each of a plurality of spaces, the determination unit 22 determines the first space where the user U is currently located based on the information. For example, the determination unit 22 can determine the space where the user U wearing the XR device 10 is located as the first space through image analysis, or can determine that the user U is currently located in the living room and dining room by picking up sound information of the user U’s speech (e.g., “I am in the living room”, etc.), and determine the living room and dining room as the first space. Moreover, when the acquisition unit 21 obtains the current position information of the user U, the determination unit 22 determines the first space where the user U is currently located based on the position information and the position information of each of the plurality of spaces included in the house layout information. The current location information of the user U is, for example, information indicating the current location of the user U (eg, latitude, longitude, altitude) measured by a location sensor (eg, a GPS (Global Positioning System) sensor) mounted on the XR device 10 worn by the user U.

[0092] When the determination unit 23 determines that the space is not in the predetermined state (No in S20 ), the identification unit 22 does not execute the process for identifying the first space where the user U is currently located, and returns to step S10 .

[0093] In addition, when it is determined that each of the multiple spaces is not in the specified state, it is determined as "No" in step S20, and when it is determined that at least one space among the multiple spaces is in the specified state, it is determined as "Yes" in step S20.

[0094] In addition, the process of step S30 may be executed regardless of whether the determination of step S20 is executed or not.

[0095] Next, the decision unit 23 determines a second space to be connected to the first space (S40). The decision unit 23 determines the second space to be connected in the real space based on the status information of each of the plurality of spaces, the second space being a space currently in a specified status among the plurality of spaces. For example, the decision unit 23 determines a space in the plurality of spaces that is connected to the first space currently located by the user U in the augmented real space and is in a specified status as the second space.

[0096] For the operation of step S40, please refer to the attached Figure 6 to explain. Figure 6 It is shown Figure 4 Flowchart showing details of the operation of step S40 (image output method) shown.

[0097] like Figure 6 As shown, the decision unit 23 determines whether the urgency of the space in the specified state (the space determined as "yes" in S20) is high (S41). For example, the decision unit 23 determines that the urgency of the space is high when a specified abnormality is occurring in the space, and determines that the urgency of the space is low when an event different from the usual is occurring or the operating status of the equipment has changed. The decision unit 23 can make the determination in step S41 based on a table that establishes a correspondence between the specified state and the level of urgency (e.g., presence or absence) for example.

[0098] When the decision unit 23 determines that the urgency of the space is high (“Yes” in S41), the decision unit 23 determines the space in the specified state (the space determined as “Yes” in S41) as the second space (S42). Furthermore, when the decision unit 23 determines that the urgency of the space is low (“No” in S41), the decision unit 23 inquires the user U whether to connect to the space in the specified state (the space determined as “No” in S41) (S43). The decision unit 23 outputs a notification for inquiring whether to connect, for example, to the XR device 10 worn by the user U or to a terminal device (such as a dedicated remote control, a smartphone, a tablet terminal, etc.) held by the user U.

[0099] Next, when receiving a response from the user U to connect to a space in a predetermined state ( S44 ), the determination unit 23 determines the space in the predetermined state selected or permitted by the user U as the second space ( S42 ).

[0100] In addition, Figure 6 In the example in which a space with high urgency is automatically determined as the second space, the present invention is not limited thereto, and a space with low urgency may be automatically determined as the second space. In other words, the determination in step S41 may not be performed.

[0101] In addition, the following description will be given of an example in which a kitchen is determined as the second space.

[0102] Refer again Figure 4, the generation unit 24 generates content information of the second space connected to the first space in the augmented reality space (S50). The generation unit 24 virtually reconfigures the house layout of the residence so that the second space exists next to the first space, and generates content information for displaying the virtually reconfigured house layout to the user U. For example, the generation unit 24 generates content information that connects the determined second space and the first space in the augmented reality space and prompts the user U via the XR device 10 worn by the user U.

[0103] The generator 24 may generate the following prompt information when the judgment in step S41 is “yes”, which is information for forcibly prompting the user U with an image via the XR device 10 worn by the user U. When such prompt information is obtained by the XR device 10, for example, the image may be forcibly displayed to the user U. For example, the XR device 10 may forcibly superimpose the image of the second space on the wall of the real space that the user U is looking at, etc., regardless of where the user U looks.

[0104] Figure 7 is a diagram showing the layout of the house after the reconfiguration involved in this embodiment. Figure 7 , an example is shown in which the generation unit 24 virtually reconfigures the layout of a house by connecting the wall W1 of the Western-style room (3) where the user U is currently located to the wall W2 of the kitchen where a predetermined abnormality has occurred. Also, it is assumed that a socket is catching fire in the kitchen.

[0105] like Figure 7 As shown, the generator 24 reconfigures the house layout so that the wall W1 and the wall W2 overlap (for example, become the same coordinate in the house layout). The generator 24 virtually reconfigures the house layout by adding a kitchen at a position adjacent to the left side of the Western-style room (3).

[0106] For example, when the user U looks at the wall W1 side in the real space, the generator 24 generates content information that overlaps the image (digital information) of the inside of the kitchen seen from the wall W2 side on the wall W1 for display. The content information is information that virtually connects the first space of the real space to the second space to expand the first space. In addition, it can be said that the content information includes information for overlapping the image obtained by shooting the second space on the first space via the XR device 10 to display when the user U looks at the second space side of the reconfigured space in the real space.

[0107] The generator 24 generates content information based on the image captured by the camera installed in the second space. Thus, it is possible to realize an image in which the kitchen is located adjacent to the left side of the Western-style room (3) in the augmented reality space.

[0108] In addition, the generation unit 24 may also generate content information such as connecting any one of the multiple walls of the first space to any one of the multiple walls of the second space in the extended real space. For example, the generation unit 24 may also generate content information such as connecting the wall with the largest area among the multiple walls of the first space (e.g., wall W1) to the wall with the largest area among the multiple walls of the second space (e.g., wall W2) in the extended real space. Such content information includes, for example, information for prompting the wall W1 with the largest area of ​​the first space in the real space by overlapping the image obtained by shooting the second space on the wall W1 via the XR device 10. In this case, the content information may be generated such that, for example, when the user U sees the wall W1 via the XR device 10, the wall W1 cannot be seen. For example, the content information may be an image of a Western-style room (3) connected to a kitchen (there is no wall between the Western-style room (3) and the kitchen).

[0109] In addition, the user U may select how the second space is connected to the first space, in other words, which building component of the first space to connect to and in what direction the second space is connected. The generator 24 may connect the first space and the second space according to the instruction from the user U.

[0110] In addition, the generating unit 24 may also include information showing the status of the second space in the content information. For example, information showing where the abnormality occurred in the kitchen and what kind of abnormality (temperature, smoke, fire, etc.) occurred can be superimposed and displayed on the image of the kitchen seen by the user U via the XR device 10. In addition, the image of the kitchen is an image of the place where the abnormality occurred in the kitchen. In addition, information showing whether the user U should rush to the kitchen for confirmation can also be superimposed on the image of the kitchen. The generating unit 24 can also, for example, use the details of the abnormality (temperature, smoke, fire, etc.) and the information showing whether the scene should be rushed to for confirmation to establish a corresponding table to generate content information superimposed with information showing whether the scene should be rushed to for confirmation. In addition, information showing the details of the response that should be performed by the user U can also be superimposed on the image of the kitchen. The generating unit 24 can also, for example, use the details of the abnormality (temperature, smoke, fire, etc.) and the information showing the details of the response to establish a corresponding table to generate content information superimposed with information showing the details of the response. The details of the response can be contacting the fire department, performing fire extinguishing work, and escaping.

[0111] Furthermore, the generator 24 may include information indicating which space the first space is connected to in the content information. In other words, not only the image of the kitchen but also information indicating the connection to the kitchen may be superimposed on the image viewed by the user U via the XR device 10 .

[0112] Refer again Figure 4 Next, the output unit 25 outputs the content information generated by the generation unit 24 to the XR device 10 (S60).

[0113] The display unit 11 of the XR device 10 displays the content information from the server device 20. For example, when the user U looks at the wall W1, the display unit 11 displays the image of the kitchen on the wall W1. The display unit 11 displays the image of the second space that cannot be seen in the first space in reality (does not exist in the first space) superimposed on the wall W1 in the real world that can be directly seen by the naked eye. The image of the kitchen displayed on the display unit 11 is, for example, an image showing the appearance of the kitchen in real time.

[0114] Accordingly, user U can confirm the appearance of a space in a specified condition in the extended reality space without having to move to the space. For example, in the case of a socket fire, user U can know that the socket fire has occurred through the extended reality space, so that initial fire extinguishing and other measures can be quickly taken. In addition, for example, in the case where smoke is detected due to a certain user smoking, user U can know that a certain user is smoking through the extended reality space (in other words, the smoke is not caused by the fire), so there is no need to move to the space where the certain user is located for confirmation. For example, in the case where smoke is detected while user U is performing a task, user U can know that the smoke is not a problem without moving to the space where a certain user is located, so he can continue to perform the task, etc.

[0115] (Other embodiments)

[0116] The above is an explanation of the house layout reconfiguration system and the like involved in one or more embodiments of the present disclosure based on the implementation mode, however, the present disclosure is not limited to the implementation mode. Within the scope of the main purpose of the present disclosure, the various modifications that can be thought of by those skilled in the art are implemented in the present embodiment, and the methods of combining the constituent elements in different embodiments are all included in the scope of the present disclosure.

[0117] For example, although the above embodiment describes an example in which the facility is a residence, the facility may be any building for which building layout information can be obtained, such as a school, a hospital, a nursing facility, an office building, etc.

[0118] In addition, although the above-mentioned embodiment describes an example in which the number of determined second spaces is one, it is not limited to this, and the determined second spaces may be more than two. When the number of determined second spaces is more than two, the server device 20 may, for example, connect multiple walls of the first space to different second spaces respectively. And, when the number of determined second spaces is more than two, the server device 20 may, for example, display more than two second spaces on the wall in a time-sharing manner. For example, the server device 20 may switch the second space connected to the first space at a predetermined time (for example, every few seconds). In this way, the house layout reconfiguration system 1 can be configured to be able to freely change the house layout (reconfigured house layout) of the facilities in the extended real space in space or time series.

[0119] In addition, although the above embodiment describes an example in which the XR device 10 is an optically transmissive device, it is not limited to this. The XR device 10 may also be, for example, a non-transmissive binocular HMD display. In this case, the XR device 10 has a camera, and the image data of the second space is superimposed on the image data of the first space captured by the camera for display. In this way, the situation in which the image data of the second space (based on the digital information of the first space) is superimposed on the image data of the first space (based on the digital information of the first space) for display also includes connecting the first space and the second space in the extended reality space.

[0120] In addition, although the above-mentioned embodiment describes an example of connecting the walls of the first space and the second space, it is not limited to this, and for example, doors, windows, etc. may also be connected. When taking the door as an example, the generation unit 24 may generate content information that connects the door of the first space and the door of the second space in the extended real space. The generation unit 24 may also generate such content information: when the user U looks at one side of the door of the first space in the real space, the image (digital information) of the inner side of the second space seen from the side of the door of the second space is superimposed on the door of the first space for display. Such content information, for example, includes information for prompting the door of the first space in the real space by superimposing the image obtained by shooting the second space on the door via the XR device 10.

[0121] In addition, the communication between the XR device 10 and the server device 20 in the above-mentioned embodiment is performed by wireless communication, for example. Regarding the communication between the XR device 10 and the server device 20, for example, wireless communication using a wide area communication network such as the Internet, but it can also be near-field wireless communication such as ZigBee (registered trademark), Bluetooth (registered trademark), or wireless LAN (Local Area Network). And, regarding the communication between the XR device 10 and the server device 20, for example, it can be performed by wired communication.

[0122] In addition, in each of the above-mentioned embodiments, each component may be formed by dedicated hardware, or may be implemented by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or a processor, reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0123] In addition, the order in which each step in the flowchart is executed is an example shown for the purpose of specifically explaining the present disclosure, and may be an order other than the above. Furthermore, part of the above steps may be executed simultaneously (in parallel) with other steps, or part of the above steps may not be executed.

[0124] In addition, the division of the functional blocks in the block diagram is an example, and multiple functional blocks can also be implemented as one functional block, or one functional block can be divided into multiple blocks, or a part of the function can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can also be processed in parallel or in time-sharing by a single hardware or software.

[0125] In addition, the server device 20 involved in the above-mentioned embodiment can be implemented by a single device or by multiple devices. In the case where the server device 20 is implemented by multiple devices, the various components of the server device 20 can be arbitrarily distributed to multiple devices. In the case where the server device 20 is implemented by multiple devices, the communication method between the multiple devices is not specifically limited, and it can be wireless communication or wired communication. And wireless communication and wired communication can be combined between devices. In addition, at least part of the functions of the server device 20 can be implemented by the XR device 10.

[0126] In addition, each component element described in the above embodiment can be implemented as software, typically as an integrated circuit, i.e., LSI. These can be made into a chip separately, or part or all of them can be included in one chip. Although it is called system LSI here, it is also called IC, system LSI, super LSI, and extra-large LSI according to the degree of integration. In addition, the method of integrated circuitization is not limited to LSI, and can also be implemented by a dedicated circuit (a general circuit that executes a dedicated program) or a general-purpose processor. It is also possible to use a programmable FPGA (Field Programmable Gate Array) after LSI manufacturing, or a reconfigurable processor that can rebuild the connection or setting of the circuit unit inside the LSI. Furthermore, with the advancement of semiconductor technology or other derived technologies, when a technology for integrated circuitization that can replace LSI appears, of course, the technology can be used to integrate the components.

[0127] System LSI is a multifunctional LSI manufactured by integrating multiple processing units on a single chip. Specifically, it is a computer system composed of a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in ROM. The microprocessor operates according to the computer program, so that the system LSI achieves its functions.

[0128] In addition, one embodiment of the present disclosure may also be to enable a computer to execute Figure 4 as well as Figure 6 A computer program comprising the characteristic steps of any of the image output methods shown in .

[0129] In addition, for example, the program may be a program for causing a computer to execute. Furthermore, one embodiment of the present disclosure may also be a computer-readable non-temporary recording medium recording such a program. For example, the recording medium recording such a program may be distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the processor may execute the program, thereby enabling the device to perform the above-mentioned processing.

[0130] (Note)

[0131] The following techniques are disclosed through the description of the above embodiments and the like.

[0132] (Technology 1)

[0133] A method for outputting an image is provided for displaying, in an extended real space, a space related to a facility of a real space having a plurality of spaces, wherein the extended real space is a space in which the real space is extended. In the method for outputting an image,

[0134] obtaining status information showing a status of each of the plurality of spaces,

[0135] determining a first space in the real space, the first space being a space in which the user is currently located among the multiple spaces,

[0136] determining a second space in the real space to be connected based on the status information of each of the plurality of spaces, the second space being a space currently in a predetermined status among the plurality of spaces;

[0137] Output prompt information, where the prompt information is information for prompting the user via an XR device worn by the user, the image of the second space and the first space being determined to be connected in the extended reality space, wherein the XR device is a cross-reality device.

[0138] (Technique 2)

[0139] As the image output method described in technology 1,

[0140] The status information includes information showing the operating status of equipment set in the space,

[0141] The specified condition includes the device being in a specified operating condition.

[0142] (Technique 3)

[0143] As the image output method described in technology 2,

[0144] The specified operating status includes that the work performed by the device has been completed.

[0145] (Technique 4)

[0146] The image output method as described in technology 2 or 3,

[0147] The predetermined operating condition includes an abnormality occurring in the device.

[0148] (Technique 5)

[0149] The image output method as described in any one of techniques 1 to 4,

[0150] further asking the user whether to connect the second space with the first space in the extended reality space,

[0151] The prompt information is output when an instruction to connect is obtained from the user.

[0152] (Technique 6)

[0153] The image output method as described in any one of techniques 1 to 5,

[0154] The status information includes sensing data of sensors arranged in the space,

[0155] Based on the sensing data, determine whether a specified abnormality is occurring in the space,

[0156] The specified condition includes that the specified abnormality is occurring in the space.

[0157] (Technique 7)

[0158] As the image output method described in technology 6,

[0159] The specified abnormality includes an abnormality in at least one of temperature, smoke, sound, and smell in the space.

[0160] (Technology 8)

[0161] The image output method as described in technology 6 or 7,

[0162] The prompt information includes information for forcibly prompting the image to the user via the XR device worn by the user.

[0163] (Technique 9)

[0164] The image output method as described in any one of techniques 1 to 8,

[0165] with respect to the house layout information showing the house layout of the plurality of spaces in the facility, virtually relocating the determined second space next to the first space,

[0166] The presentation information includes information for presenting the second space by superimposing an image obtained by photographing the second space on the first space via the XR device when the user views the second space that is virtually reconfigured in the real space.

[0167] (Technology 10)

[0168] An image output system is an image output system for displaying, in an extended real space, a space related to a facility of a real space having a plurality of spaces, wherein the extended real space is a space in which the real space is extended, and the image output system comprises:

[0169] an obtaining unit that obtains status information indicating the status of each of the plurality of spaces;

[0170] a determining unit, which determines a first space in the real space, where the first space is a space where the user is currently located among the multiple spaces;

[0171] a determination unit that determines, based on the status information of each of the plurality of spaces, a second space in the real space to be connected, the second space being a space currently in a predetermined status among the plurality of spaces; and

[0172] An output unit outputs prompt information, wherein the prompt information is used to prompt the user via an XR device worn by the user, the XR device being a cross-reality device, with the image of the second space and the first space being connected in the extended reality space.

[0173] (Technology 11)

[0174] A program for causing a computer to execute the image output method described in any one of Techniques 1 to 9.

[0175] Industrial Applicability

[0176] The present disclosure is useful for systems and the like that use XR devices.

[0177] Description of Reference Numerals

[0178] 1House layout reconfiguration system (image output system)

[0179] 10 XR Device

[0180] 11 Display

[0181] 20 Server Devices

[0182] 21 Get Department

[0183] 22 Determination

[0184] 23 Decision Department

[0185] 24 Generation Department

[0186] 25 Output

[0187] U User

[0188] W1, W2 walls

Claims

1. A method for outputting an image, the method being for displaying, in an extended real space, a space related to a facility in a real space having a plurality of spaces, wherein the extended real space is a space in which the real space is extended, wherein: obtaining status information showing a status of each of the plurality of spaces, determining a first space in the real space, the first space being a space in which the user is currently located among the multiple spaces, determining a second space in the real space to be connected based on the status information of each of the plurality of spaces, the second space being a space currently in a predetermined status among the plurality of spaces; Output prompt information, where the prompt information is information for prompting the user via an XR device worn by the user, the image of the second space and the first space being determined to be connected in the extended reality space, wherein the XR device is a cross-reality device.

2. The image output method according to claim 1, The status information includes information showing the operating status of equipment set in the space, The specified condition includes the device being in a specified operating condition.

3. The image output method according to claim 2, The specified operating status includes that the work performed by the device has been completed.

4. The image output method according to claim 2, The predetermined operating condition includes an abnormality occurring in the device.

5. The image output method according to any one of claims 2 to 4, further asking the user whether to connect the second space with the first space in the extended reality space, The prompt information is output when an instruction to connect is obtained from the user.

6. The image output method according to claim 1, The status information includes sensing data of sensors arranged in the space, Based on the sensing data, determine whether a specified abnormality is occurring in the space, The specified condition includes that the specified abnormality is occurring in the space.

7. The image output method according to claim 6, The specified abnormality includes an abnormality in at least one of temperature, smoke, sound, and smell in the space.

8. The image output method according to claim 6 or 7, The prompt information includes information for forcibly prompting the image to the user via the XR device worn by the user.

9. The image output method according to any one of claims 1 to 4, 6 and 7, with respect to the house layout information showing the house layout of the plurality of spaces in the facility, virtually relocating the determined second space next to the first space, The presentation information includes information for presenting the second space by superimposing an image obtained by photographing the second space on the first space via the XR device when the user views the second space that is virtually reconfigured in the real space.

10. An image output system for displaying, in an augmented reality space, a space related to a facility in a real space having a plurality of spaces, wherein the augmented reality space is a space in which the real space is expanded, the image output system comprising: an obtaining unit that obtains status information indicating the status of each of the plurality of spaces; a determining unit, which determines a first space in the real space, where the first space is a space where the user is currently located among the multiple spaces; a determination unit that determines, based on the status information of each of the plurality of spaces, a second space in the real space to be connected, the second space being a space currently in a predetermined status among the plurality of spaces; as well as An output unit outputs prompt information, wherein the prompt information is used to prompt the user via an XR device worn by the user, the XR device being a cross-reality device, with the image of the second space and the first space being connected in the extended reality space.

11. A program for causing a computer to execute the image output method according to any one of claims 1 to 4, 6, and 7.

Images