Augmented reality display device and recording medium

Abstract

The present invention provides an augmented reality technology capable of combining images in real time in a virtual space. The augmented reality display device comprises: a shooting unit for obtaining a background image of the real world; a model control unit for controlling a two-dimensional or three-dimensional model in a virtual space; and a display unit for displaying the model together with the background image. The model has one or more connection units that can be connected to other models, and the model control unit controls each model so that the connection unit of the first model is connected to the connection unit of the second model in the virtual space.

Description

Technical Field

[0001] The present invention relates to a virtual reality display device for displaying a virtual model superimposed on a background image of the real world and a recording medium recording a computer program for realizing the virtual reality display device. Background Art

[0002] In recent years, with the development of computer graphics rendering technology and the popularization of mobile devices such as smartphones and head-mounted displays, experiential video content known as VR (Virtual Reality) or AR (Augmented Reality) has attracted much attention. VR is mainly a technology that presents only images of virtual space to users wearing head-mounted displays while shielding the visual recognition of real space, and can provide the user with highly immersive video content. In contrast, AR is a technology that presents the real space as a background image for users of smartphones or head-mounted displays, while overlapping a two-dimensional model or a three-dimensional model on the background image, and can provide the user with highly realistic video content in which the real space and the virtual model are mixed. The present invention is particularly related to augmented reality (AR) technology among these experiential video technologies.

[0003] Regarding AR technology, Patent Document 1 discloses an augmented reality presentation device that can synthesize and display images and image data in real space. The device described in Patent Document 1 recognizes a predetermined part of a person based on the three-dimensional image data of the person obtained by photography, and aligns the three-dimensional image data of the person with the three-dimensional image data of multiple objects based on the recognition result. The device described in Patent Document 1 is envisioned to be used, for example, in a service that simulates the experience of trying on clothes.

[0004] In addition, Patent Document 2 discloses a vehicle installation trial fitting system that recognizes a predetermined reference portion of a vehicle in a vehicle image, calculates a reference position for installing a vehicle installation product, and synthesizes the vehicle installation product with an appropriately adjusted proportion at the reference position. According to the system described in Patent Document 2, various vehicle installation products (roof luggage boxes, child seats, etc.) can be trial fitted and displayed on a screen on a vehicle according to the user's preferences.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent document 1: Japanese Patent Application Publication No. 2012-118948;

[0008] Patent document 2: Japanese Patent Application Publication No. 2017-220059.

[0009] However, the inventions described in Patent Documents 1 and 2 analyze the image of an object (a person or a vehicle, etc.) in real space and synthesize a three-dimensional image at a specific part of the image of the object. The above invention attracts the interest of users by combining (synthesizing) the image of a real object with a three-dimensional image. However, the above invention requires the existence of an object that becomes the object of synthesizing a three-dimensional image in real space. If such an object does not exist, the unique fun of AR technology cannot be provided to users.

[0010] In addition, in the above invention, as a pre-stage process for synthesizing objects in real space and three-dimensional images, it is necessary to analyze the image in real space, confirm the existence of a specified object in the image, or identify features such as shape in a specified object to determine the location of the synthesized three-dimensional image. Therefore, depending on the processing power of the computing device (smartphone, etc.), the image processing load is too large and the image display may be delayed. In this case, it is difficult to overlap the objects in real space with the three-dimensional image in real time, and there is a deviation between the object and the image, which also causes a problem of loss of realism. Summary of the invention

[0011] The main purpose of the present invention is to provide an augmented reality technology that is independent of the existence of a specified object in the real space and can give users the pleasure of combining images in real time in a virtual space.

[0012] As a result of the inventor's intensive research on the means for achieving the above-mentioned purpose, it is determined that one or more joint portions (Joint Portions) are pre-set on a two-dimensional or three-dimensional virtual model, and the image obtained by combining multiple models through the joint portions is overlapped and displayed with the background image of the real world. As a result, a game method such as assembling models or scene (Diorama) models can be realized in real time through the augmented reality technology that utilizes the model of the virtual space. The inventor thought that the above-mentioned purpose can be achieved as long as it is based on the above-mentioned idea, and thus completed the present invention. If described in detail, the present invention has the following structure.

[0013] The first aspect of the present invention relates to an augmented reality display device. The augmented reality display device of the present invention has a shooting unit, a control unit and a display unit. The shooting unit obtains a background image of the real world. The model control unit controls a two-dimensional or three-dimensional model in a virtual space. The display unit displays the model together with the background image. In the present invention, the model has one or more connecting parts that can be connected to another model. The model control unit controls each model so that the connecting part of the first model is connected to the connecting part of the second model in the virtual space. In addition, in the present application specification, the real space and the virtual space are combined together, and the space in which the virtual model overlaps on the real background image is conveniently referred to as "augmented reality space."

[0014] As described above, by overlapping and displaying the image obtained by connecting multiple models in a virtual space with a real background image, it is possible to realize experiential content such as assembly models or scene models through augmented reality technology. In addition, the present invention combines virtual models and does not rely on actual objects that exist in the real world, so it is possible to provide users with new game methods under any circumstances. In addition, a connection part is pre-set on the model of the present invention. Therefore, when connecting multiple models, there is no need for pre-processing such as analyzing the shape or characteristics of each model. In this way, the load in the connection processing of the model can be reduced, so the delay in the image output processing can be suppressed, and even a computing device with general processing power can output the image of the augmented reality space in real time.

[0015] In the augmented reality display device of the present invention, it is preferred that the model control unit controls each model in such a manner that the first model and the second model are connected through the connection part when the connection part of the first model and the connection part of the second model are close to within a specified range in the virtual space. In this way, by automatically connecting the two models when the connection parts of the two models are close to within a specified range, the user can easily connect the models. On the other hand, by requiring the user to perform an operation to bring the connection parts of each model close to within a specified range, a simulated method of enjoying the fun of assembling models can be provided.

[0016] In the augmented reality display device of the present invention, preferably, metadata (Metadata) is assigned to each connection part to determine the model or connection part that can be connected thereto. In this case, the model control unit analyzes the metadata of two connection parts that are close to each other in the virtual space. When the model control unit determines that the two close connection parts can be connected to each other, the first model and the second model are connected at these two connection parts. As such, in the present invention, not all connection parts can be connected to each other without restriction, as long as the other connection parts that can be connected are predetermined (restricted) for each connection part. As a result, the individuality of each model is strengthened, so it is possible to provide a game method that includes elements of finding or collecting a collection of models that can be connected to the connection part of a certain model.

[0017] The augmented reality display device of the present invention may further include a model generation unit. The model generation unit analyzes an image containing a real object and generates a two-dimensional or three-dimensional model based on the real object contained in the image. In addition, the model generation unit sets one or more connection parts that can be connected to another model for the generated model. In this way, by generating a model with a connection part based on an arbitrary image received and provided from the user, the user can perform a game method of combining his own original model with another model in the augmented reality space.

[0018] The augmented reality display device of the present invention may further include a model acquisition unit. The model acquisition unit analyzes an image containing a real object, and based on the information obtained from the real object contained in the image, reads a model corresponding to the real object from a storage unit, or obtains a model corresponding to the model from a server through a communication line. In this way, characteristic information such as the shape and color of the real object or coding information attached to the object is obtained from the real object contained in the image, and based on this information, a two-dimensional or three-dimensional existing model corresponding to the object is determined. It is sufficient to read the existing model from the storage unit of the augmented reality display device itself, or download it from a server through the Internet, etc. In this way, the existing model can be effectively utilized, so that a virtual model corresponding to the real object can be simply and quickly made to appear in the augmented reality space.

[0019] In the augmented reality display device of the present invention, it is preferred that the model control unit automatically corrects at least one of the size, shape and texture of at least part of the first model and the second model when connecting the connecting portion of the first model with the connecting portion of the second model. In this way, by automatically correcting the size and the like when connecting the models, the connection between the models can be achieved in a more natural form.

[0020] The augmented reality display device of the present invention may further include an operating unit and a candidate model screening unit. The operating unit receives the selection of a connection portion of a model through an input operation by a user. The candidate model selection unit screens out one or more candidates of other virtual models having a connection portion that can be connected to the connection portion selected by the operating unit, and displays information about the screened virtual models on the display unit. In this way, by presenting information about models that can be connected to the selected connection portion to the user, the user can easily find the desired model. In addition, for example, in the case where the present invention is used in a service that sells models as commodities, the candidate models are presented through the above-mentioned operation, thereby prompting the user to purchase the model.

[0021] The second aspect of the present invention relates to a recording medium having a computer program recorded thereon. The program recorded in the recording medium of the present invention is used to enable a general-purpose portable information terminal to function as the augmented reality display device involved in the first aspect described above. Examples of portable information terminals are smartphones, tablet terminals, laptop terminals, head-mounted displays, and heads-up displays. The program of the present invention can be recorded in a computer-readable medium such as a CD-ROM.

[0022] According to the augmented reality technology of the present invention, it is possible to combine images in real time in a virtual space regardless of whether or not a predetermined object exists in a real space. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a diagram showing an embodiment of an augmented reality display device of the present invention, schematically showing a state where a model is displayed on a display unit of the device.

[0024] Figure 2 This is a diagram schematically showing the relationship between multiple models that can be connected.

[0025] Figure 3 This is a block diagram showing an example of the performance structure of an augmented reality display device and a management server.

[0026] Figure 4 This is a diagram showing a data structure example of a database that records various information related to a model.

[0027] Figure 5A , Figure 5B This is a diagram schematically showing an operation for connecting models.

[0028] Figure 6 The diagram schematically shows a process of generating a virtual model from an image of a real object.

[0029] Figure 7 This is a diagram schematically showing a process of reading a virtual model corresponding to an image of a real object.

[0030] Figure 8 This is a diagram schematically showing a process of displaying candidates of models connectable to a certain connection portion.

[0031] Fig. 9 It is a diagram showing another embodiment of the augmented reality display device of the present invention, schematically showing a state in which a model of a scene model is displayed on the display unit of the device.

[0032] Description of Reference Numerals

[0033] 10...Augmented reality display device 11...Control unit

[0034] 11a ... Model control unit 11b ... Model generation unit

[0035] 11c ... Model acquisition unit 11d ... Candidate model screening unit

[0036] 12 ... Storage unit 13 ... Photographing unit

[0037] 14 ...Display unit 15 ...Operation unit

[0038] 16 ... sensor unit 17 ... communication unit

[0039] 20 ... Management server 21 ... Control unit

[0040] 22…Database 23…Communications DETAILED DESCRIPTION

[0041] The following describes specific embodiments with reference to the drawings. The present invention is not limited to the embodiments described below, and includes appropriate changes that can be made by a person skilled in the art within an obvious range based on the following embodiments.

[0042] Figure 1 FIG. 1 shows a first embodiment of an augmented reality display device 10 of the present invention. In this embodiment, the augmented reality display device 10 is implemented by a so-called smart phone. Figure 1 As shown, the augmented reality display device 10 uses the real space photographed by the shooting unit (camera) as a background image, and displays the image of the two-dimensional or three-dimensional model superimposed on the background image on the display unit (display). In addition, in each figure of the present application, the virtual model drawn by computer graphics technology is represented by a diagonal hatching line.

[0043] Figure 2 The following is a diagram showing the connection structure of the assembled model which is a characteristic point of the present invention. Figure 2 As shown, in the present invention, one or more connection parts are set in each model. Figure 2 In the example shown, four connections B1, B2, B3, and B4 are set on the "Body" model. In addition, the connection B1 of the "Body" model can be connected to the connection H1 set on the "Head" model. Similarly, the connections B2, B3, and B4 of the "Body" model can be connected to the connection H1 set on the "Head" model. Figure 2As shown, the connection part RA1 of the "Right Arm" model, the connection part LA1 of the "Left Arm" model, and the connection part L1 of the "Leg" model can be connected respectively. In this way, in this embodiment, a game method such as assembling models can be realized in an augmented reality space.

[0044] exist Figure 1 as well as Figure 2 In the example shown, as an example of an assembled model, a three-dimensional human-shaped model such as a figure or a plamodel (registered trademark) is shown. The human-shaped model includes a torso, a head, a right arm, a left arm, and a leg, each of which is independent, and all the models are connected by the aforementioned connecting parts to form a completed form. However, in the present invention, the assembled model is not limited to the human-shaped model, for example, it can also be an equipment or decoration worn on the human-shaped model, it can also be other products that imitate characters, or it can be a vehicle or a train, aircraft, ship, or other transportation vehicle type. In addition, the details will be described later, but the assembled model can also be a model of a scene (Diorama) model that expresses the landscape, such as a train track or station, a crossing, a building, a tree, a rock, a mountain, or other natural objects. In addition, the assembled model is not limited to the models listed here, and can include various types of models.

[0045] Figure 3 It is a block diagram showing the functional structure of a server-client system including an augmented reality display device 10 and a management server 20. The augmented reality display device 10 and the management server 20 are connected via a communication line such as the Internet, and can send and receive information to each other. In the present embodiment, the management server 20 stores information about a plurality of models (modeling data and its metadata), and the augmented reality display device 10 downloads the information about the models from the management server 20 as needed and displays it on the display unit 14. However, although it is omitted in the figure, in another embodiment, the augmented reality display device 10 itself may store information about a plurality of models in advance. In this case, the management server 20 is not required.

[0046] The augmented reality display device 10 is a device that has at least the function of photographing the real space and displaying the photographed image and the virtual model in an overlapping manner. The augmented reality display device 10 is implemented, for example, by a well-known smart phone or a head-mounted display. Specifically, in a general-purpose portable information terminal such as a smart phone, an application is stored in order to execute the processing unique to the augmented reality display device 10 of the present invention. The terminal functions as the augmented reality display device 10 of the present invention by executing the program. Figure 3As shown, the augmented reality display device 10 basically includes a control unit 11 , a storage unit 12 , a photographing unit 13 , a display unit 14 , an operation unit 15 , a sensor unit 16 , and a communication unit 17 .

[0047] The control unit 11 of the augmented reality display device 10 performs overall control of other elements 12 to 17 of the augmented reality display device 10. As the control unit 11, a well-known processor such as a CPU or a GPU can be used. The control unit 11 reads an application stored in the storage unit 12, and controls other elements according to the application. Specifically, the control unit 11 causes the background image of the real space photographed by the shooting unit 13 to be displayed on the display unit 14. In addition, the control unit 11 causes the model received from the management server 20 through the communication unit 17 to be displayed on the display unit 14 together with the background image, and controls the movement of the model according to the input information from the operation unit 15 or the sensor unit 16. In addition, the control unit 11 has a model control unit 11a, a model generation unit 11b, a model acquisition unit 11c, and a candidate model screening unit 11d as main functional blocks. For details of these functional blocks 11a to 11d, refer to Figures 4 to 8 Described later.

[0048] The storage unit 12 is an element for storing information used in the display control of augmented reality. Specifically, the storage unit 12 stores an application for enabling a portable information terminal such as a general smart phone to function as an augmented reality display device 10. The application stored in the storage unit 12 is read by the control unit 11 when the augmented reality is displayed, and processing according to the program is executed. In addition, a plurality of other applications may also be stored in the storage unit 12. The storage function of the storage unit 12 can be implemented, for example, by a non-volatile memory such as an HDD and an SDD. In addition, the storage unit 12 may also have a function as a memory for writing or reading the process of the calculation processing performed by the control unit 11. The memory function of the storage unit 12 can be implemented by a volatile memory such as a RAM or a DRAM.

[0049] The photographing unit 13 is a camera for obtaining image data of a still image or a moving image. The camera constituting the photographing unit 13 is a camera built into the augmented reality display device 10. The image data obtained by the photographing unit 13 is sent to the control unit 11, and after a predetermined calculation process, it is displayed on the display unit 14. In addition, the image data can also be saved in the storage unit 12. The camera is composed of, for example, a lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a CCD image sensor unit or a CMOS image sensor unit, a digital signal processor (DSP) that reads the charge amount from the photoelectric conversion element to generate image data, and an IC memory.

[0050] The display unit 14 is an element for displaying a predetermined image. The display unit 14 is composed of a known display device such as a liquid crystal display or an organic EL display. In the present invention, the display unit 14 mainly displays a two-dimensional or three-dimensional model together with a background image of a real space.

[0051] The operation unit 15 is an element for inputting the user's operation information into the augmented reality display device 10. As the operation unit 15, any known input device such as a touch panel, a mouse, a keyboard, a stylus, etc. can be used. In addition, it is also possible to configure a touch panel display by arranging the touch panel (operation unit 15) on the front of the display (display unit 14). In addition, the operation unit 15 may also be a structure that can be physically separated from the augmented reality display device 10. In this case, the operation unit 15 is connected to the augmented reality display device 10 via a short-range wireless communication standard such as Bluetooth (registered trademark)).

[0052] The sensor unit 16 is an element for inputting sensor information into the augmented reality display device 10. Examples of the sensor unit 16 include an acceleration sensor, a gyro sensor, a microphone, a GPS sensor, a proximity sensor, a brightness sensor, etc. The augmented reality display device 10 has one or more sensors. In the present invention, the sensor unit 16 is used to detect the posture of the augmented reality display device 10. That is, through the sensor unit 16 including the acceleration sensor and the gyro sensor, the change in the photographing direction or photographing range of the photographing unit 13 of the augmented reality display device 10 is detected as sensor information. The control unit 11 (mainly the model control unit 11a) uses the sensor information to control the display position and display direction of the model. For example, the control unit 11 only needs to be able to perform the following control: make the model appear at a specific position in the real space, and when the model existing at the specific position enters the photographing range of the photographing unit 13, the model is displayed on the display unit 14.

[0053] The communication unit 17 of the augmented reality display device 10 is an element for communicating mainly with the management server 20 through a communication line such as the Internet. Specifically, when a download request for a specified model is generated by the control unit 11, the communication unit 17 sends the request to the management server 20. In addition, the communication unit 17 receives various model-related information stored in the database 22 of the management server 20 and transmits it to the control unit 11.

[0054] The management server 20 is a network server that is mainly responsible for managing the model provided to the augmented reality display device 10. The model and related information stored in the management server 20 can be updated (added, changed, deleted) at any time. In addition, the management server 20 can be composed of a single network server, or it can be constructed by sharing its functions among multiple network servers. The management server 20 is basically managed by an operator that provides users with services related to augmented reality. Figure 3 As shown, the management server 20 basically includes a control unit 21 , a database 22 , and a communication unit 23 .

[0055] The control unit 21 of the management server 20 performs overall control of other elements 22 and 23 of the management server 20. A well-known processor such as a CPU or a GPU can be used as the control unit 21. When receiving a download request from the augmented reality display device 10, the control unit 21 mainly reads the model corresponding to the download request and the information related thereto from the database 22, and sends the information to the augmented reality display device 10 via the communication unit 23.

[0056] The database 22 stores information related to various models provided to the augmented reality display device 10 . Figure 4 The data structure of a relational database is shown as an example of data included in the database 22. For example, the database 22 includes a model table (a), a connection condition table (b), and an owned model table (c).

[0057] The model table records information related to various models displayed on the augmented reality display device 10. In particular, the model table records information about the connection parts of each model. Specifically, the model table includes a model ID field (Field) (column), a series field, a part field, a model name field, and a connection part field.

[0058] The model ID field records the identification information (ID) inherent to the model. The series field records the name of the series to which each model belongs. For example, in the example shown in the figure, model IDs 0001 to 0005 belong to Series A (Note: Series A), and model IDs 0006 to 0010 belong to Series B (Note: Series B). For example, models belonging to the same series have the same design or are made with the same design concept and have common points such as a sense of unity. The part of the model is recorded in the part field. For example, in Figure 4 In the example shown, since it is assumed that Figure 2The assembled humanoid model shown in the figure is divided into the head, body, left arm, right arm and leg as an example. The model name field records the name of the model. The connection part field records the identification information of one or more connection parts of each model.

[0059] like Figure 4 As shown in the example, in the present invention, it is preferred that each model does not have a separate connection part, but the connection part is standardized, so that a connection part common to multiple models can be provided. That is, model ID0002 and model ID0007 are different models, but have common connection parts such as B1, B2, B3, and B4. For example, connection part RA1 can be connected to connection part B2, and by standardizing the connection part, another model having connection part RA1 can be connected to connection part B2 of model ID0002 or connection part B2 of model ID0007. By standardizing the connection part in this way, it is possible to replace the model.

[0060] The connection condition table records the connectable conditions of the connection part. For example, the connection part H1 is set to be connectable to the connection part B1. In addition, the connection part B1 is set to be connectable to the connection part H*. Here, "*" means that the connectable conditions only need to be partially consistent, which means that, for example, as long as the prefix of the identification information of the connection part is "H" like "H1" or "H2", the connectable conditions are met regardless of the subsequent values. Therefore, the connection part B1 can be connected to either the connection part H1 or H2. And like the relationship between the connection part H1 and the connection part B1, the connection parts that satisfy the connectable conditions are judged to be connectable. For example, the connection part H1 can be connected only to the connection part B1. In addition to the connection parts H1 and H2, the connection part B1 can also be connected to other connection parts with the prefix H (such as H3 and H4 not shown).

[0061] Furthermore, the connectable conditions may include additional conditions other than the conditions indicating the connectable connection portion. Figure 4 In the example shown, the information of the series field recorded in the model table is set as an additional condition. For example, the link H2 is set to be connectable to the link B1 and the link of the model belonging to Series B. In other words, the link H2 cannot be connected to the link B1 of the model belonging to Series A, and can only be connected to the link B1 of the model belonging to Series B. It is also possible to set additional conditions in this way to limit the connectable links.

[0062] The owned model table is a table used to record information related to the models owned by each user. For example, a user can use a model that he or she owns by making it appear freely in the augmented reality space. On the other hand, a model that is not owned cannot be used, and in order to use such a model, it is necessary to obtain it separately (purchase, create, exchange with other users, receive it as a gift from other users, etc.). The owned model table includes a user ID field for recording the user's unique ID, an owned model ID field for recording the ID of the model owned by the user, and a source field for recording the source of each model. Regarding the source of the model, for example, "created" means a model created by the user himself or herself (for example, refer to Figure 6 ). In addition, "acquisition" means downloading a model from a server when a user takes a photo of a predetermined real object (for example, referring to Figure 7 ). In addition, the so-called "download" means that the user downloads the model from the server (for example, refer to Figure 8 ). In addition, the source of the model may also be an exchange with other users or a gift from other users.

[0063] The communication unit 23 of the management server 20 is an element for communicating mainly with the augmented reality display device 10 via a communication line such as the Internet. The communication unit 23 mainly transmits information related to various models stored in the database 22 to the augmented reality display device 10 .

[0064] Next, refer to Figure 5A , Figure 5B An example of a method of linking models will be described. Figure 5A As an example, the case where the left arm model (Left Arm) is connected to the torso model (Body) is shown. Figure 5A As shown, the display unit 14 displays a model obtained by connecting a head model (Head), a right arm model (Right Arm) and a leg model (Leg) to a torso in an augmented reality space (a space in which a real space and a virtual space are combined and a virtual model is superimposed on a real background image). In addition, in the augmented reality space, the left arm model (Left Arm) is displayed in a state separated from other models.

[0065] exist Figure 5A , Figure 5BIn the example shown, the display unit 14 and the operation unit 15 constitute a touch panel display, and the user can freely operate by touching each model. The model control unit 11a included in the control unit 11 controls these models existing in the augmented reality space (specifically, the virtual space). For example, by sliding the finger up, down, left, and right while touching each model with one finger, each model can be moved. In addition, by sliding the finger up, down, left, and right while touching each model with two fingers, the direction of each model can be changed. In addition, by widening the distance between the two fingers (enlarging (Pinch Out)) or narrowing the distance between the two fingers (reducing (Pinch In)) while touching each model with two fingers, the size of each model can be changed. In addition, there are operations such as touching each model twice or touching each model three times with one finger, and the model control unit 11 can control the movement or change of the model in the virtual space in response to various operations of the user.

[0066] exist Figure 5A In the example shown, the user performs the following operation: the connection part LA1 of the left arm model is brought close to the connection part B3 of the torso model. In addition, in this example, the range of a distance r1 away from the connection part B3 of the torso model is represented by a dashed line, and the range of a distance r2 away from the connection part B3 of the torso model is represented by a dotted line (wherein, distance r2 < distance r1). In this case, when the connection part LA1 of the left arm approaches the connection part B3 of the torso, the connection part LA1 first enters the range of distance r1. When the connection part LA1 enters the range of distance r1, the model control unit 11a determines whether the connection part LA1 and the connection part B3 satisfy the connectable conditions with each other. The connectable conditions are, for example, Figure 4 As shown in FIG. 1 , the connection condition table is recorded, so the model control unit 11a only needs to refer to the table. When the model control unit 11a determines that the connection part LA1 and the connection part B3 can be connected, a guidance display is performed to show that these connection parts LA1 and B3 can be connected. Figure 5A In the figure, an "arrow" is displayed as an example of a guide display. That is, in order to show that the connection part LA1 of the left arm and the connection part B3 of the trunk can be connected, the "arrow" is used to indicate the direction in which the connection part B3 of the trunk exists from the left arm currently being moved. Thus, the user can easily understand that the two models are connected by moving the left arm toward the trunk along the "arrow".

[0067] Next, the connection part LA1 of the left arm further approaches the connection part B3 of the trunk, and when entering the range of distance r2, the model control unit 11a automatically connects the connection part LA1 and the connection part B3. As a result, the model connection operation becomes simple for the user. In addition, by performing the connection processing corresponding to the distance of the connection part, multiple models can be connected without relying on the shape of the model, etc., so that the processing load on the computer can be reduced. Moreover, when connecting the models of the left arm and the trunk, the model control unit 11a can also automatically correct one or more of the size, shape and texture (Texture) of each model in its entirety or in part. For example, when connecting, if the size (proportion) of the left arm and the trunk model deviates, the two models are connected on the basis of making the size of the two models consistent. In addition, in order to make the state of the two models after connection look more natural, it is also possible to change the shape of both or one of the models of the left arm and the trunk. Furthermore, when connecting, even if there is a difference in texture (color or texture, etc.) between the left arm model and the torso model, the two models may be connected after making the textures of the two models consistent.

[0068] On the other hand, Figure 5B In the example shown, the user performs an operation to bring the joint RA1 of the right arm model closer to the joint B3 of the trunk model. Here, the joint RA1 of the right arm cannot be connected to the joint B3 of the trunk. In this case, when the joint RA1 of the right arm approaches the joint B3 of the trunk and enters the range of distance r1, the model control unit 11a performs a guidance display indicating that these joints RA1 and B3 cannot be connected. For example, Figure 5B As shown, an "×" is displayed between the connection part RA1 of the right arm and the connection part B3 of the trunk, and it is sufficient to show that the two models cannot be connected.

[0069] In addition, Figure 5A , Figure 5B , it is of course possible to separate the models from the connected models. In this case, when the operation opposite to the operation of connecting the models is performed, the model control unit 11a only needs to control so that the models are separated. As described above, the models can be freely connected and separated via the connection unit.

[0070] Next, refer to Figures 6 to 8 , an example of how users can obtain models is given below. Figure 6 A method of generating a two-dimensional or three-dimensional virtual model based on a real object included in a photographic image is described. Figure 6(a) in the figure shows a photographic image of a real object such as a figurine or a plastic assembly model. The real object is not limited to a figurine or an assembly model, but may also be a train track or station, a crossing, a building or other construction, or a tree, a rock, a mountain or other natural object. The augmented reality display device 10 obtains a photographic image of such a real object. The photographic image may be obtained by photographing the camera unit 13, or may be obtained by downloading the image through the Internet or the like. Figure 6 (a) in FIG. 1 shows a photographic image of the front side of the object. When making a two-dimensional model, only a photographic image of the front side of the object is sufficient. On the other hand, when making a three-dimensional model, it is preferable to obtain multiple images of the same object taken from different angles. Figure 6 In addition to the front side of the object shown in (a), photographic images of the object from multiple angles, such as the back side, left side, right side, top view, top view, and other stereoscopic images, can be obtained.

[0071] Next, the model generation unit 11b performs Figure 6 One or more photographic images are analyzed as shown in (b) in the figure, and a two-dimensional or three-dimensional model is generated based on the real object contained in the image. For example, in the case of generating a two-dimensional model, the outline of the real object contained in a photographic image can be determined, the outer edge of the plane model can be formed along the outline, and the color and pattern of the model can be set in a manner corresponding to the color and pattern of the real object. In addition, in the case of generating a three-dimensional model, the outer edge of the three-dimensional model can be formed based on multiple photographic images captured from different angles of the same object, and the color and pattern of the model can be set in a manner corresponding to the color and pattern of each image. In addition, the process of generating a two-dimensional or three-dimensional model based on a photographic image is not limited to the method described here, and a known process can be appropriately adopted.

[0072] Next, the model generation unit 11b performs Figure 6 As shown in (c) in the figure, a connection part is set in the model generated as described above. Specifically, the model generating unit 11b first extracts the point (hereinafter referred to as "separation point") at which the model of the component (Part) is separated from the model in the completed form according to the prescribed rule. As the prescribed rule, for example, parts of the completed model such as the base of the arm, the neck, and the thinning part of the torso, which are thinner or narrower than other parts, may be used as separation points. In addition, parts of the completed model whose color or pattern changes significantly compared to other parts may be used as separation points. In addition, a data set of a plurality of pre-made models and their separation points may be used as teacher data to implement machine learning, and the separation points of a certain model may be automatically formed by referring to the learned model obtained thereby.

[0073] Next, the model generation unit 11b sets a connection part for the separation point formed as described above. Thus, the models of each component can be freely assembled and disassembled on the completed model. Figure 4 When the parts are standardized as shown in (b) in the figure, the model generation unit 11b can select an appropriate connection part from the standardized connection parts and set it at the separation point of the model. As a result, not only the connection / separation between the models of the parts separated from the same model, but also the models of the parts separated from another model can be replaced. For example, there is the following situation: the completed form model A is separated into the models of the parts A1 and A2, and the completed form model B is separated into the models of the parts B1 and B2. In this case, by using the standardized connection part, not only A1 and A2 can be reconnected and B1 and B2 can be reconnected, but A1 and B2 can also be connected, or B1 and A2 can be connected.

[0074] The model generation unit 11b registers information about the model generated as described above in the database 22 of the management server 20. Specifically, for the newly generated model, the model of each component is set to Figure 4 The information corresponding to each field (column) of the model table shown in (a) can be recorded. Model ID, part, model name, and connection part can be automatically assigned. In addition, for models of parts separated from a certain completed model, they can be registered as the same series. In addition, by centrally managing the models generated by each user by the management server 20, for example, a model generated by a certain user can be downloaded and used by another user. In this case, the exchange or sale of models between users can also be carried out.

[0075] Figure 7 The present invention shows a method of reading an existing model pre-stored in the storage unit 12 or the database 22 based on a real object included in a photographic image. Figure 7 In the example shown, a two-dimensional or three-dimensional model is not newly generated, but a real object included in the photographic image is used as a trigger to read an existing model corresponding to the real object from the database 22 or the like. Specifically, the model acquisition unit 11c of the control unit 11 Figure 7 The photographic image shown in (a) in the figure is analyzed to extract the feature points of the real object contained in the image. The outline, color, pattern, etc. of the object can be listed as the feature points. In addition, when a code such as a two-dimensional code (barcode) or a three-dimensional code (QR code (registered trademark)) is attached to the real object, the model acquisition unit 11c can also extract the code from the photographic image as a feature point to obtain the information embedded in the code. In this case, it is sufficient to embed the ID information of the model corresponding to the real object in the code.

[0076] Next, the model acquisition unit 11c performs Figure 7 As shown in (b) of FIG. 1 , based on the feature points or codes (ID information) extracted from the photographic image, the corresponding model is read from the storage unit 12 of the augmented reality display device 10 or the database 22 of the management server 20. The model stored in the storage unit 12 or the database 22 is as follows: Figure 7 As shown in (c) in the figure, a connection part has been set. According to such an embodiment, for example, after a user purchases a physical figure or assembled model, by photographing it with a smartphone (augmented reality display device 10), a service such as downloading a two-dimensional or three-dimensional model corresponding to the figure, etc. from the management server 20 can be deployed. In addition, in the present invention, the model can be freely assembled in the augmented reality space or exchanged with another model, so the game method of the actually purchased figure, etc. can be expanded.

[0077] Figure 8 A method for specifying a connection of a model and obtaining another model that can be connected to the specified connection is shown. Figure 8 As shown in FIG. 1 , when a user operates the touch panel display or the like to designate a connection portion of a certain model, the candidate model selection unit 11d of the control unit 11 selects candidates for models having a connection portion that can be connected to the connection portion designated by the user from among the multiple models stored in the database 22 of the management server 20 (or the storage unit 12 of the augmented reality display device 10). In addition, the candidate model selection unit 11d displays the selected candidate models on the display unit 14. For example, in Figure 8 In the example shown, the connection part B3 of the torso model (Body) is specified by the user. The connection part LA1 of the left arm model (Left Arm) can be connected to the connection part B3 of the torso. Therefore, the candidate model screening unit 11d screens a plurality of left arm models (Left Arm 1 to 4) having the connection part LA1 as candidates, and displays the candidates on the display unit 14.

[0078] Next, the user selects any model from among the multiple models displayed as candidates. As a result, the model selected by the user is read from the database 22 of the management server 20 (or the storage unit 12 of the augmented reality display device 10) and appears in the augmented reality space. Figure 5A The order shown is the same and can be connected to another model. In this way, the user can simply search and obtain another model that can be connected thereto by specifying the connection part.

[0079] Fig. 9A second embodiment of the augmented reality display device 10 of the present invention is shown. In this embodiment, the augmented reality display device 10 is implemented by a so-called head mounted display (not shown). The augmented reality display device 10 of the second embodiment has basically the same functional configuration as the first embodiment described above (see Figure 3 ). However, the second embodiment is different from the first embodiment in that the model appearing in the augmented reality space is a so-called scene model, which is assumed to be a human-shaped model.

[0080] like Fig. 9 As shown, in this embodiment, multiple rail models (Rail (note: rail)) are displayed in the augmented reality space, and these rails can be connected at the connection parts R1 and R2. That is, a convex connection part R1 is set at one end of each rail, and a concave connection part R2 is set at the other end, and the connection parts R1 and R2 are set to be connected. In addition, the rails are prepared with, for example, Fig. 9 Straight tracks (Straight (Note: Straight)) and curved tracks (Gurve (Note: Curved)) as shown, or tracks of other shapes. Therefore, by connecting the connection part R1 of a certain track model with the connection part R2 of another track model in the augmented reality space, the track can be laid in a free shape or the track can be freely extended.

[0081] In addition, Fig. 9 In the example shown, a model of a train running on a model of a track is prepared. The train model can be Figure 6 (a) and Figure 6 The sequence shown in (b) is based on the photographic image of the actual train, and can also be based on Figure 7 (a) and Figure 7 In the order shown in (b) in the figure, the data is read from the storage unit 12 or the database 22 according to the photographic image of the actual train. In addition, the train model may be an existing model read from the storage unit 12 or the database 22. The train model is programmed to run on a model of a track in the augmented reality space. Therefore, the user can make a favorite train model run along the model of the track he or she has created. In addition, although the illustration is omitted, the models that can appear in the augmented reality space are not limited to tracks and trains, as long as models of scene models that represent the landscape, such as stations, crossings, buildings, natural objects such as trees or rocks, and mountains, are appropriately prepared.

[0082] In the present invention, the term augmented reality (AR) is used, but according to the embodiments of the present invention, it is sometimes classified as virtual reality (VR) or mixed reality (MR) or XR, which is a general term for VR and MR.

[0083] In the above, in order to express the content of the present invention, the present invention is described with reference to the attached drawings. Figure 1 The embodiments of the present invention have been described. However, the present invention is not limited to the above-described embodiments, and includes modifications and improvements that are obvious to those skilled in the art based on the matters described in the present specification.

Claims

1. An augmented reality display device, comprising: A photographing unit for obtaining a background image of the real world; A model control unit for controlling a two-dimensional or three-dimensional model in a virtual space; A display unit for displaying the model together with the background image; in, The model has one or more connecting parts that can be connected to another model, The model control unit controls the models so that a first connection portion of the first model is connected to a connection portion of the second model, and a second connection portion of the first model is connected to a connection portion of the third model in the virtual space.

2. The augmented reality display device according to claim 1, wherein: The model control unit controls each model so that the first connection portion of the first model and the connection portion of the second model are connected when the first connection portion of the first model and the connection portion of the second model approach each other within a predetermined range in the virtual space.

3. The augmented reality display device according to claim 1, wherein: Metadata for specifying connectable models or connection parts is allocated to the connection parts. The model control unit analyzes the metadata of two connection parts that are close to each other in the virtual space, and when it is determined that the two connection parts can be connected to each other, connects the models at the two connection parts.

4. The augmented reality display device according to claim 1, wherein: The invention also includes a model generating unit that analyzes an image containing a real object and generates a two-dimensional or three-dimensional model based on the real object contained in the image. The model generation unit sets one or more connection parts that can be connected to another model for the generated model.

5. The augmented reality display device according to claim 1, wherein: It also has a model acquisition unit, which analyzes an image containing a real object, and reads the model corresponding to the real object from the storage unit based on information obtained from the real object contained in the image, or obtains the model corresponding to the real object from the server through a communication line.

6. The augmented reality display device according to claim 1, wherein: When the model control unit connects the first connection part of the first model with the connection part of the second model, the model control unit automatically corrects at least one of the size, shape and texture of both or either of the first model and the second model in part.

7. The augmented reality display device according to claim 1, wherein: Also features: an operating portion capable of accepting selection of the connecting portion of the model; The candidate model selection unit selects a plurality of candidates of the other models having a connection portion that can be connected to the connection portion selected by the operation unit, and displays information on the selected models on the display unit. 8 . A recording medium recording a program for causing a portable information terminal to function as the augmented reality display device according to claim 1 .

Images