Data structure, data display program, data generation program, and data accumulation processor

Abstract

To generate an AR scenario for explaining an object to an end user by more simple operation.SOLUTION: There is provided a data structure of object data having a content such that explanation for an object as an explanation object is AR displayed on a computer. The data structure of the object data includes mapping data obtained by mapping the surface of the object, and AR display data including one or a plurality of sets of provision information including marker information indicating the position of a marker arranged on the surface of the object, in a coordination system based on the mapping data, and a content related to the object corresponding to the marker information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a data structure, a data display program, a data generation program, and a data storage and processing device, and can be applied to a system that performs, for example, AR (Augmented Reality) processing. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there exists a system disclosed in Patent Document 1 as a technology for guiding a user to operate a product by using AR processing.

[0003] The system of Patent Document 1 holds an "AR scenario" in which AR content including display of still images and videos and audio output is described in a set order. In the system of Patent Document 1, a wearable device equipped with a camera, display, headphones, microphone, optical sensor, etc. is worn on the user's head, and the AR scenario is provided to the user using the wearable device as a user interface. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-200920 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional AR processing systems, creating an AR scenario (such as an AR scenario as described in Patent Document 1) that explains to a user an object to be explained to the user (such as an object that can be operated or used; hereinafter simply referred to as an "object") required complicated operations, such as 3D modeling of each part that makes up the object.

[0006] In view of the above problems, there is a need for an information processing device, an information processing program, an information processing method, and a data structure that are capable of generating an AR scenario that explains an object to an end user with simpler operations. [Means for solving the problem]

[0007] The first invention is characterized in that a data structure of object data describing content for AR display on a computer of an explanation of an object to be explained is provided, the data structure comprising mapping data obtained by mapping a surface of the object, and AR display data having one or more sets of marker information indicating positions of markers to be placed on the surface of the object in a coordinate system based on the mapping data, and additional information including content related to the object corresponding to the marker information.

[0008] A data display program according to the second aspect of the present invention causes a computer to function as AR display means for performing AR display based on object data having the data structure according to the first aspect of the present invention.

[0009] A data generation program according to a third aspect of the present invention is characterized in that it causes a computer to function as object data generation means for generating object data having the data structure according to the first aspect of the present invention. Effect of the Invention

[0010] According to the present invention, it is possible to generate an AR scenario that explains an object to an end user with simpler operations. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram showing a connection configuration of each device according to the embodiment. [Diagram 2] 2 is a block diagram showing the overall configuration of a user company terminal according to the embodiment; FIG. [Diagram 3] 2 is a block diagram showing the overall configuration of an end user terminal according to the embodiment. FIG. [Figure 4] FIG. 2 is a block diagram showing an internal configuration of an application server according to the embodiment. [Diagram 5] 4 is a diagram showing an example of a data configuration of user company management data managed by a user company management data management unit according to the embodiment. FIG. [Figure 6] 11 is a diagram showing how an AR scenario based on object data is played (displayed) on an end user terminal according to an embodiment. FIG. [Figure 7] FIG. 2 is a block diagram showing an example of a logical structure of object data according to the embodiment. [Figure 8] FIG. 2 is a diagram showing an example of the configuration of target object management data according to the embodiment. [Figure 9] 1 is an explanatory diagram showing an example of the configuration of AR sequence management data of AR sequence data according to an embodiment; [Figure 10] 1A to 1C are explanatory diagrams showing configuration examples of marker information and attachment information according to an embodiment. [Figure 11] In the embodiment, an example in which a target object is arranged in an XYZ coordinate system is shown. [Figure 12] 1 illustrates an example of setting an origin in a coordinate system of an AR space according to an embodiment. [Figure 13] 13 is a flowchart showing an example of an operation of reproducing an AR scenario based on object data in an end user terminal (object data display application) according to the embodiment. [Figure 14] FIG. 13 is a diagram (part 1) showing an example of an AR display screen displayed on a touch panel display of an end-user terminal by an object data display application according to an embodiment. [Figure 15] FIG. 2 is a diagram (part 2) showing an example of an AR display screen displayed on a touch panel display of an end-user terminal by an object data display application according to an embodiment. [Figure 16]13 is a flowchart showing an example of an operation of a user company terminal (object data generation application) according to an embodiment, where an AR scenario (object data) related to an object is generated in response to an operation by an operator of a user company. [Figure 17] FIG. 11 is a diagram (part 1) showing an example of an AR generation screen displayed on a touch panel display 11 of a user company terminal by an object data generation application according to the embodiment. [Figure 18] FIG. 2 is a diagram (part 2) showing an example of an AR generation screen displayed on the touch panel display 11 of the user company terminal by the target object data generation application according to the embodiment. [Figure 19] FIG. 11 is a diagram (part 3) showing an example of an AR generation screen displayed on the touch panel display 11 of the user company terminal by the target object data generation application according to the embodiment. [Figure 20] FIG. 11 is a block diagram showing an example of a logical structure of object data according to a modified example of the embodiment. [Figure 21] 13 is a flowchart showing an example of an operation of a user company terminal (object data generation application) according to a modified embodiment (part 1) for generating an AR scenario (object data) related to an object in response to an operation by an operator of a user company. [Figure 22] 13 is a flowchart showing a modified example of the operation of editing object data TD in response to an operator's operation in a user company terminal (object data generation application) according to a modified embodiment (part 2). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (A) Main embodiment Hereinafter, an embodiment of a data structure, a data display program, a data generation program, and a data storage processing device according to the present invention will be described in detail with reference to the drawings.

[0013] (A-1) Configuration of the embodiment FIG. 1 is a block diagram showing the connection configuration of each device according to this embodiment.

[0014] 1, there are arranged an information processing system 1, a user company terminal 10, and an end user terminal 20. It is assumed that the devices shown in FIG. 1 are configured to be able to communicate with each other via the Internet NW.

[0015] The information processing system 1 is a system that processes data (hereinafter referred to as "object data" or "AR scenario data") including an AR scenario (hereinafter referred to simply as an "AR scenario" or "explanatory AR scenario") that explains an object TO that is to be explained to an end user.

[0016] In this embodiment, a user who receives an explanation based on object data about an object TO (hereinafter also referred to as an "object data provided user") is referred to as an "end user," and a user who generates and provides object data including object data about an object TO (hereinafter also referred to as an "object data providing user") is referred to as a "user company." Here, a user company is described as a company that mainly manufactures or sells the object TO itself, and generates object data about the object TO that it manufactures or sells and provides it on the Internet NW. Here, an end user is described as a user who uses an object TO, and receives an explanation based on object data when using the object TO.

[0017] The user company terminal 10 may be configured by installing a program (including an "AR scenario generation program" or a "data generation program" according to the embodiment) in an information processing terminal (e.g., various PCs, tablet PCs, smartphones, etc.) equipped with a computer capable of processing an AR scenario. The end user terminal 20 may be configured by installing a program (including an "AR scenario display program" or a "data display program" according to the embodiment) in an information processing terminal (e.g., various PCs, tablet PCs, smartphones, etc.) equipped with a computer capable of processing an AR scenario.

[0018] Next, the configuration of the information processing system 1 will be described.

[0019] The information processing system 1 shown in FIG. 1 has an application server 30. The number of devices (computers) constituting the information processing system 1 is not limited. For example, the information processing system 1 may be configured to have the functions of the application server 30 using a plurality of devices (computers). The application server 30 shown in FIG. 1 is capable of communicating with other devices (terminals) via the Internet NW. In this embodiment example, the user company terminal 10 and the end user terminal 20 are both described as tablet PCs equipped with various devices capable of processing object data.

[0020] The application server 30 is a device that mainly processes object data between each terminal. The application server 30 also manages information on user companies (AR scenario providing users) (hereinafter referred to as "user company information") and information on end users (AR scenario provided users) (hereinafter referred to as "end user information").

[0021] FIG. 2 is a block diagram showing the overall configuration of the user company terminal 10 of this embodiment.

[0022] As shown in FIG. 2, the user company terminal 10 includes a touch panel display 11, a camera 12, an information processing unit 13, a sensor unit 14, a microphone 15, and a speaker 16.

[0023] The information processing unit 13 has a function of controlling the entire user company terminal 10, and has an object data generation application 131 (including a "data generation program" according to the embodiment) that generates an AR scenario.

[0024] The information processing unit 13 may be configured as software by installing a program (including a "data generation program" according to the embodiment) in a computer (not shown), or may be configured as hardware (for example, using a dedicated semiconductor chip, etc.). In this embodiment, the information processing unit 13 will be described as being configured as software using a computer.

[0025] The camera 12 is an imaging means capable of imaging the real space in which the object TO is placed. In this embodiment, the camera 12 is described as having a configuration capable of stereoscopic scanning (3D scanning) of the object TO by including multiple image sensors (compound eyes). Note that if the user company terminal 10 itself does not need to perform 3D scanning, the camera 12 may be a monocular camera.

[0026] The touch panel display 11 has a display means (display function) capable of displaying and outputting (presenting) images (e.g., images of an operation screen, etc.) to a user, and an operation means (touch panel function) capable of accepting operations from the user.

[0027] The sensor unit 14 is a group of sensors for detecting the movement of the user company terminal 10. For example, a group of sensors including an acceleration sensor and a gyro sensor (a group of sensors required for 3D scanning and AR processing) can be applied to the sensor unit 14.

[0028] FIG. 3 is a block diagram showing the overall configuration of the end user terminal 20 of this embodiment.

[0029] The end user terminal 20 includes a touch panel display 21, a camera 22, an information processing unit 23, a sensor unit 24, a microphone 25, and a speaker 26.

[0030] The information processing unit 23 has a function of controlling the entire end user terminal 20, and has an object data display application 231 (a program equivalent to the "data display program" according to the embodiment) that performs playback processing of the AR scenario.

[0031] The information processing unit 23 may be configured as software by installing a program (a program corresponding to the "data display program" according to the embodiment) in a computer (not shown), or may be configured as hardware (for example, using a dedicated semiconductor chip, etc.). In this embodiment, the information processing unit 23 will be described as being configured as software using a computer.

[0032] In this embodiment, other devices constituting the end user terminal 20 (touch panel display 21, camera 22, sensor unit 24, microphone 25 and speaker 26) are similar to those of the user company terminal 10, so detailed description will be omitted.

[0033] As described above, in terms of hardware, the user company terminal 10 and the end user terminal 20 can be configured using information processing devices such as PCs and tablet PCs.

[0034] FIG. 4 is a block diagram showing the internal configuration of the application server 30. As shown in FIG.

[0035] The application server 30 includes an AR processing server application 31 and a data management unit 32. That is, the application server 30 includes the AR processing server application 31 and the data management unit 32 as a data storage processing unit that stores and processes object data.

[0036] The internal configuration of the application server 30 can be functionally shown in Fig. 5. The application server 30 can be constructed, for example, by installing software (including the "data storage processing program" according to the embodiment) in a computer (e.g., a PC or a workstation) having a memory and a processor.

[0037] The AR processing server application 31 processes object data between each terminal. The AR processing server application 31 accumulates the object data supplied from the user company terminal 10 in a data management unit 32 and processes it. The AR processing server application 31 also performs a process of providing object data in response to a request from the end user terminal 20.

[0038] The data management unit 32 accumulates and manages data including object data under the control of the AR processing server application 31. The data management unit 32 accumulates object data and the like supplied from the AR processing server application 31. In addition, the data management unit 32 reads object data and the like requested by the AR processing server application 31 and supplies the data to the AR processing server application 31. The data management unit 32 can be constructed using, for example, various DBMSs (Database Management Systems).

[0039] As shown in Figure 4, the data management unit 32 includes at least a user company management data management unit 321 that manages user companies, an object data management unit 322 for managing object data TD (TD-1, TD-2, ...) including object data for each object TO, and reference data RD (RD-1, RD-2, ...) that is referenced when mapping the shape of the object TO.

[0040] In this embodiment, it is assumed that IDs 1, 2, ... are assigned to the object data TD-1, TD-2, ... as IDs for identifying the object data (hereinafter referred to as "object data IDs"), respectively. Also, in this embodiment, it is assumed that IDs 1, 2, ... are assigned to the reference data RD-1, RD-2, ... as IDs for identifying the reference data (hereinafter referred to as "reference IDs"), respectively.

[0041] FIG. 5 is a diagram showing an example of the data configuration of the user company management data managed by the user company management data management unit 321. As shown in FIG.

[0042] As shown in Fig. 5, the user company management data includes, for each user company, at least a user company ID for identifying the user company, a user company name, and data items of an object data list that is a list of object data IDs belonging to the user company. For example, in Fig. 5, for company A, a user company ID of 1 is assigned, and it is registered that object data TD-1, TD-2, and TD-3 belong to the company.

[0043] Next, an overview of the object data in this embodiment will be described with reference to FIG.

[0044] FIG. 6 is a diagram showing how an AR scenario based on object data relating to an object TO is reproduced on the end user terminal 20 of this embodiment.

[0045] Fig. 6 illustrates an end user terminal 20 configured as a tablet terminal and an object TO. In Fig. 2, a screen based on object data corresponding to the object TO (hereinafter, referred to as an "AR display screen") is displayed on the touch panel display 21 of the end user terminal 20.

[0046] Specifically, in FIG. 6, the object data display application 231 of the end user terminal 20 synthesizes an image captured by the camera 22 (an image capturing the object TO) with an image of a marker or content based on the object data of the object TO (hereinafter referred to as an “AR image”), and displays it on the touch panel display 21.

[0047] As described above, the object TO is an object that can be the subject of explanation to an end user, and is, for example, an industrial product that requires various operations (e.g., industrial equipment, audio equipment, etc.). Objects that require operations (e.g., industrial products, etc.) usually include parts that can be the subject of explanation (hereinafter referred to as "parts to be explained"), such as operating parts (e.g., buttons, handles, levers, knobs, etc.), display parts (e.g., lamps, displays, various meters, etc.), movable parts (e.g., openable covers, drive mechanisms, etc.), and dangerous parts (e.g., parts that generate high heat, etc.).

[0048] In this embodiment, the object data includes content that visually displays (as an AR image) an explanation of the operation part of the object TO in an easy-to-understand manner. Specifically, in this embodiment, the object data includes at least data (hereinafter referred to as "mapping data") obtained by mapping (for example, 3D mapping by a point cloud or the like) the reference data of the object TO in a predetermined coordinate system, information (hereinafter referred to as "marker information") that defines a marker placed in an explanation part on a 3D object (an object in a 3D space represented by a point cloud or the like) of the object TO represented by the mapping data, and data (hereinafter referred to as "attachment information") that includes content (for example, content for explaining the explanation part) that is attached (linked) to each marker (explanation part). That is, in this embodiment, the object data includes mapping data corresponding to the object TO, one or more pieces of marker information, and one or more pieces of attached information (content) corresponding to each piece of marker information. Note that there may be cases where a plurality of pieces of attached information are associated with one marker.

[0049] Next, an example of a logical data structure of the object data TD will be described with reference to FIG.

[0050] FIG. 7 is a block diagram showing an example of the logical structure of the object data TD.

[0051] As shown in FIG. 7, the object data includes object management data DD, which is information for systematically managing the object TO, one or more pieces of mapping data MD, and one or more pieces of AR sequence data AD.

[0052] The AR sequence data AD is associated with any mapping data MD, and is data having one or more pairs of marker information PD and attached information SD on the mapping data MD. As shown in FIG. 7, the AR sequence data AD has a structure in which a plurality of pairs of marker information PD and attached information SD are logically arranged and linked in a predetermined order, forming a series of flows (hereinafter, this flow is referred to as an "AR sequence"). In other words, by taking the above-mentioned AR sequence structure in the AR sequence data AD, it is possible to play an AR scenario that comprehensively displays AR contents (contents based on the attached information SD) in a predetermined order. In other words, the above-mentioned AR sequence can be said to show one pattern of an AR scenario.

[0053] 7 shows a pattern in which the mapping data MD and the AR sequence data AD are in one-to-one correspondence, but a configuration in which a plurality of AR sequence data AD are linked to one mapping data MD may be used. This is because it is sufficient that the mapping data MD corresponding to each AR sequence data AD is uniquely determined. In the example of this embodiment, a pattern in which the mapping data MD and the AR sequence data AD are in one-to-one correspondence will be described.

[0054] In addition, in order to simplify the explanation, FIG. 7 also illustrates the reference data RD that was used as the basis for generating each mapping data MD, but if the mapping data MD has already been generated, the presence of the reference data RD is not essential for processing the object data TD.

[0055] In the object data TD shown in Fig. 7, the mapping data MD-1, MD-2, ... are assigned IDs of 1, 2, ..., respectively, and the AR sequence data AD-1, AD-2, ... are assigned IDs of 1, 2, ..., respectively. In the AR sequence data AD-1 shown in Fig. 7, the marker information PD1-1, PD, 1-2, ..., PD-1-N (N is an integer of 1 or more) are assigned IDs of 1, 2, ..., N, respectively, and the assignment information SD-1-1, SD-1-2, SD-1-M (M is an integer of 1 or more) are assigned IDs of 1, 2, ..., M, respectively.

[0056] FIG. 8 is a diagram showing an example of the configuration of the object management data DD.

[0057] As shown in Figure 8, the object management data DD includes an "object ID", which is an identifier for the object data TD, a "user company ID" (the user company ID of the user company to which the object data TD belongs), and an "object name", which is the name of the object TO corresponding to the object data TD.

[0058] FIG. 9 is an explanatory diagram showing an example of the configuration of the AR sequence management data ARM of the AR sequence data AD-1.

[0059] As shown in FIG. 9, in the AR sequence management data ARM, an "AR sequence ID" that is an identifier of the AR sequence data AD, and an ID of the mapping data corresponding to the AR sequence data AD (mapping data ID) are set.

[0060] For example, Figure 9 shows the AR sequence management data ARM of the AR sequence data AD-1 shown in Figure 8, so the AR sequence ID is set to "1" and the mapping data ID is set to "1", which is the identifier of the corresponding mapping data MD-1.

[0061] FIG. 10 is an explanatory diagram showing an example of the configuration of the marker information PD and the attachment information SD.

[0062] FIG. 10 shows a specific configuration example of the marker information PD-1-2 and two pieces of attachment information SD-1-2 and SD-1-3 associated with the marker information PD-1-2.

[0063] As shown in FIG. 10, the marker information PD includes a "marker ID" which is an identifier of the corresponding marker, a "marker position" (X, Y, Z values) which indicates the position of the corresponding marker (position in the coordinate system of the corresponding mapping data MD; coordinates), a "marker size" which indicates the size of the marker, and an "attachment information list" which is a list of IDs of the attached information corresponding to the marker. The marker defined by the marker information in this embodiment is a circular area (an area along the surface of the 3D object of the target TO) with the marker position as the center and the marker size as the radius. The unit of the marker size is not limited, but may be, for example, the unit of the coordinate system of the mapping data MD. For example, if the marker position is (X, Y, Z)=(0, 0, 0) and the marker size is 10, the marker becomes, for example, a circle with a radius of 10 units centered on the origin (0, 0, 0). The marker size may be a fixed value or may be a value which varies according to the user's operation. In addition, the shape of the marker is not limited to a circle, and may be a polygon such as a rectangle.

[0064] As shown in Figure 10, the grant information SD includes an "grant information ID" which is an identifier of the grant information, a "content type" which indicates the type of content corresponding to the grant information, a "content link destination" which indicates the link destination for accessing the content, and a "next sequence" which indicates the identifier of the grant information next in sequence to the grant information.

[0065] The content type may be, for example, text (e.g., a txt file), image (e.g., a JPEG or TIFF file), video (e.g., an MPEG file), or a homepage (e.g., an HTML file). There is no limit to the number or combination of types that can be specified as the content type. In this embodiment example, the content type is described as being set to "text" indicating text, "image" indicating an image, "movie" indicating a video, "html" indicating a homepage, etc.

[0066] The content link destination may be a URL on the Internet, or a path for accessing a specific file corresponding to the content (for example, a path on a file server, etc.).

[0067] The next sequence indicates the order (sequence) when multiple pieces of attachment information SD correspond to the same marker information PD. In other words, the next sequence indicates the order (sequence) when multiple pieces of attachment information SD corresponding to the same marker are reproduced in the end user terminal 20, for example. The next sequence of the attachment information SD in the last order (sequence) is "END". In addition, when only one piece of attachment information SD corresponds to one piece of marker information PD, the next sequence of the attachment information SD is END. For example, in FIG. 10, the next sequence of the attachment information SD1-2 is 3 (ID of the attachment information SD-1-3), and the next sequence of the attachment information SD-1-3 is END. In other words, in the AR sequence data AD, a tree-structured sequence is formed by the sequence structure of the marker information PD and the sequence structure of the attachment information SD under each piece of marker information PD.

[0068] Next, an example of a coordinate system (a coordinate system of an AR space centered on an object TO) defined in the mapping data MD of each object data TD will be described with reference to Figures 11 and 12. In the example of Figures 11 and 12, the object TO is an audio interface equipped with four knobs (dials) as operation parts.

[0069] Fig. 11 shows an example in which an object TO is placed in an XYZ coordinate system with a position P0 as the origin. Fig. 12 shows an example in which an origin P0 is set in the coordinate system of an AR space. Fig. 12 shows an example in which the origin P0 is set to one of the vertices of a bounding box BB of the object TO.

[0070] (A-2) Operation of the embodiment The operation of each device (each program) according to this embodiment will be described below.

[0071] FIG. 13 is a flowchart showing an example of an operation of reproducing an AR scenario related to an object TO in the end user terminal 20 (object data display application 231).

[0072] The processing of the object data display application 231 shown in FIG. 13 may be realized using middleware related to AR (for example, middleware such as ARKit).

[0073] 14 and 15 are diagrams showing examples of the AR display screen displayed on the touch panel display 21 of the end user terminal 20 by the object data display application 231. In FIG.

[0074] First, it is assumed here that the end user operates the end user terminal 20 and the object data display application 231 is started. When the object data display application 231 is started, it displays an image captured by the camera 22 on the touch panel display 21. At this time, the object data display application 231 displays a message (such as a message "point the camera so that the object is captured") to guide the end user to bring the object TO into the field of view of the camera 22, as shown in FIG. 14(a). It is assumed here that the object data display application 231 creates 3D scanning data of the object TO based on an image captured by the camera 22 (for example, an image captured by a compound eye camera) and a detection result of the sensor unit 24, and acquires the result (S101). At this time, the object data display application 231 may acquire mapping data as a result of scanning the object TO. In FIG. 14 and FIG. 15, the image (captured image) of the object TO displayed on the touch panel display 21 is TOG.

[0075] Next, the object data display application 231 identifies candidate object data TD based on the acquired scanning result (mapping data), presents the object names of the candidate object data TD to the user, and accepts selection of the object TO to be processed this time (S102).

[0076] For example, the object data display application 231 may transmit the scanning result to the AR processing server application 31, and the AR processing server application 31 may specify the candidate object data TD. The method of specifying the candidate object data TD from the scanning result is not limited, but for example, a feature amount (vector data) may be acquired from the scanning result, and one or more object data having similar (higher than a predetermined degree of match) mapping data may be extracted from the object data management unit 322, and the extracted object data TD may be set as a candidate. The object data display application 231 may request the AR processing server application 31 to extract the object data having mapping data similar to the scanning result from the object data management unit 322. At this time, the object data display application 231 may present an operation screen (operation screen for selecting one from a list of object names) as shown in FIG. 14(b) to the end user, and allow the end user to select one of the object names. When there is only one candidate object data TD, the object data display application 231 may omit the process of making the end user select it.

[0077] The object data display application 231 may receive input of identification information (such as model number or name) for identifying the object TO, or may receive selection of the object TO via a menu screen or the like.

[0078] Through the above-described processing, the object data display application 231 identifies the object data TD corresponding to the scanned object TO and acquires it from the AR processing server application 31 (S103).

[0079] Then, the object data display application 231 selects one of the AR sequence data AD from the acquired object data TD, and further acquires mapping data MD corresponding to the selected AR sequence data AD. The object data display application 231 then maps the surface of the object TO based on the acquired mapping data MD, and causes the AR display screen (touch panel display 21) to compositely display images of the markers according to each piece of marker information PD acquired from the AR sequence data AD (S104).

[0080] FIG. 15(a) illustrates a state in which markers M101 to M104 (hatched areas) are superimposed and displayed on each of the four knobs (operation parts) of the object TO on the AR display screen.

[0081] Then, when any of the markers M101 to M104 is operated (for example, by touching on the touch panel display 21), the object data display application 231 displays the content (content linked to the content) of the attachment information SD corresponding to the operated marker on the AR display screen (S105).

[0082] FIG. 15(b) illustrates a state in which the marker M101 is operated, and the content of the attached information SD corresponding to the marker information PD of the marker M101 (a pop-up window displaying text) is synthesized and displayed on the AR display screen.

[0083] FIG. 16 is a flowchart showing an example of an operation of the user company terminal 10 (object data generation application 131) to generate object data (AR scenario data) related to the object TO in response to an operation by an operator of the user company.

[0084] 17 to 19 are diagrams showing examples of operation screens (hereinafter referred to as "AR generation screens") displayed on the touch panel display 11 of the user company terminal 10 by the object data generation application 131. FIG.

[0085] Note that the processing of the object data generation application 131 shown in FIG. 16 may be realized using middleware related to AR (for example, middleware such as ARKit).

[0086] First, it is assumed here that the operator operates the user company terminal 10, launches the object data generation application 131, and starts the object data generation process (for example, by an arbitrary menu operation by the operator). When the object data generation application 131 starts the object data generation process, it first accepts an input reception process (hereinafter referred to as "information input process") of information on the object TO for which the object data TD is to be generated (for example, each item of the object management data DD) (S200). For the information input process, it is sufficient to be able to receive information input for each item of the object management data DD, so detailed screen configurations are not shown in the figures.

[0087] Next, the object data generation application 131 displays the image captured by the camera 12 on the touch panel display 11. At this time, the object data generation application 131 displays a message (such as a message "point the camera so that the object is captured") to guide the end user to bring the object TO into the field of view of the camera 12, as shown in FIG. 17(a). Here, the object data generation application 131 3D scans the object TO based on the image captured by the camera 12 (for example, an image captured by a compound eye camera) and the detection result of the sensor unit 14, and acquires mapping data (S201, S202). In FIG. 17 to FIG. 19, the image (captured image) of the object TO displayed on the touch panel display 11 is TOG.

[0088] Next, the object data generation application 131 performs AR processing based on the acquired mapping data to display an AR generation screen such as that shown in FIG. 17(b) and accepts input of the marker position (3D coordinates) to be set by the operator (S203).

[0089] Specifically, the object data generation application 131 maps the feature points of the point group constituting the mapping data on the surface of the object TO currently photographed by the camera 22, and further enables conversion between the position (2D coordinates) on the AR generation screen (touch panel display 21) and the 3D coordinates on the surface of the object TO. This allows the object data generation application 131 to accept an input of the 3D coordinates (position on the surface of the object TO) corresponding to the position (2D coordinates) touched on the AR generation screen (touch panel display 21) as the coordinates of the marker position input by the operator. Here, as shown in FIG. 18(a), it is assumed that the operator touches the position P201 where the knob (dial) constituting the object TO is displayed. Note that the processing of such coordinate conversion may be performed using the functions of various AR processing middleware.

[0090] Next, the object data generation application 131 performs a process of accepting input of content (additional data) corresponding to the marker position input by the operator (S204).

[0091] At this time, the object data generation application 131 may display a GUI window on the AR generation screen that accepts input of the type and details of the content (additional data) corresponding to the marker position. For example, the object data generation application 131 may accept the type of content (additional data) corresponding to the marker position by displaying a pop-up window in which multiple types of media options (in FIG. 18(b), buttons such as "text", "image", "video", "audio" etc.) are arranged, as shown in FIG. 18(b), and accept data input according to the selected type. For example, in FIG. 18(b), when text is selected as the type of content (additional data) corresponding to the marker position, the object data generation application 131 may additionally display a pop-up window in which text can be input as shown in FIG. 19(a) and FIG. 19(b) to accept input. The object data generation application 131 temporarily stores the input marker position and the content (additional data) in association with each other.

[0092] Next, the object data generation application 131 determines whether or not the next marker position has been input (S205), and if the next marker position has been input, proceeds to the processing of step S203 described above, and if not, proceeds to the processing of step S204. For example, as shown in Fig. 17(b), the object data generation application 131 may provide a button (a button labeled "End") for accepting whether or not to end the generation of the AR scenario when accepting the input of the marker position, and use the button to accept an input that the next marker position has not been input.

[0093] If the next marker position is not input, the object data generation application 131 generates AR sequence data based on the data (a set of marker position and content (additional data)) input in steps S203 and S204, and further generates object data TD having a structure as shown in Fig. 7 together with mapping data (mapping data generated in step S202) and other information (information acquired in the information input process in step S201), and transmits (uploads) it to the application server 30 (AR processing server application 31) (S206). At this time, the object data generation application 131 may also upload the result of 3D scanning itself, which is the source of the generated mapping data, to the application server 30 as reference data RD. At this time, when the application server 30 (AR processing server application 31) receives an upload of new object data TD, it assigns a new ID to the object data TD and additionally registers it in the object data management unit 322, and also performs a process of updating the user company managed data in the user company managed data management unit 321 (adding and registering the ID of the object data TD added to the object data list of the corresponding user company ID). Also, at this time, if the application server 30 (AR processing server application 31) also receives an upload of reference data RD, it additionally registers the reference data RD in the reference data management unit 323, and manages it in association with the object data uploaded at the same time.

[0094] As described above, the object data generation application 131 generates the mapping data MD and the AR sequence data AD, and transmits them to the application server 30 (AR processing server application 31) as object data TD.

[0095] (A-3) Effects of the embodiment According to this embodiment, the following effects can be obtained.

[0096] In the information processing system 1 of this embodiment, as shown in FIG. 7, the AR scenario is generated and held using the object data TD, so that the generation reception process can be performed only by a simple operation (repeated marker setting and attachment information setting) as shown in FIG. 16. In particular, the object data TD of this embodiment has a simple data structure in which the object is defined by mapping data MD capturing the whole as a single object without mapping or modeling each part of the object, and then marker information is overlaid and attachment information SD (content) is linked to each marker information, so that the object data can be created by the above-mentioned simple operation. For example, in the data structure that defines the conventional AR scenario, mapping data for each part is defined for an object having multiple parts, and content is attached to the mapping data for each part, so that an extremely complicated process of acquiring mapping data for each part is required. In contrast, the object data TD of this embodiment is structured such that, as described above, the object TO is defined by mapping data MD that captures the object TO as one object, a marker (marker information PD) is mapped onto one object defined by one mapping data MD, and each marker is linked to attached information SD (contents). Therefore, the object data TD of this embodiment has a simple structure and yet can be given a sufficient amount of information when configuring data of an AR scenario that explains the object TO to an end user.

[0097] In particular, in the object data TD of this embodiment, the object management data DD, mapping data MD, and AR sequence data are managed separately, and are structured to be easily replaced or edited individually. Furthermore, the contents of the AR sequence data AD are also structured to be easily replaced or edited individually, and are structured to be easily managed separately for the AR sequence management data ARM, marker information PD, and attachment information SD. In the object data TD of this embodiment, the structure allows easy replacement of individual data as described above, making it possible to create object data with the above-mentioned simple operation. Note that data constituting a normal AR scenario (for example, a data format normally processed on a platform such as ARKit) is a block of binary data, and is therefore not structured to be easily replaceable like the object data TD of this embodiment.

[0098] In addition, in this embodiment, by providing virtual information to the object TO, which is a real product, by the object data TD, it is possible to promote practical use of the object TO by end users (for example, in a factory or a production site). For example, an end user who receives an AR scenario based on the object data TD of this embodiment can efficiently operate and maintain the object TO. Specifically, for example, a worker who conventionally operates the object TO (for example, various equipment and products) by checking the procedure while referring to a PDF or paper manual can work while checking the procedure using contents such as videos, drawings, audio, and text virtually projected on the product by AR. In this way, the end user terminal 20 instantly displays necessary information by the AR scenario based on the object data TD, so that the end user can efficiently operate and maintain the object TO, leading to improved productivity. Furthermore, by using the AR scenario (AR service) based on the object data TD, the end user can promote training in work using the object TO. For example, the end user can actually train while touching the object TO by working on the object TO displayed in AR based on the object data TD. As described above, in this embodiment, since the object data TD (AR scenario) can be easily created in the user company terminal 10, know-how regarding the object TO can be easily shared within a company, etc. As described above, for the end user of this embodiment, the AR scenario based on the object data TD promotes employee training and knowledge sharing within the company, leading to skill improvement at the production site.

[0099] (B) Other embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments such as those exemplified below.

[0100] (B-1) In the AR sequence data AD constituting the object data TD in the above embodiment, all the attached information SD is linked to one of the marker information PD, but some or all of the attached information SD may be structured not to be linked to the marker information (that is, the attached information SD may be structured directly linked to the mapping data MD). For example, in FIG. 21, in the AR sequence data AD-1, none of the marker information PD is linked to the attached information SD-1-2 and SD-1-3. In this case, in the end user terminal 20 (object data display application 231), the attached information SD (content) not linked to the marker information PD may be displayed on the AR display screen by switching (for example, switching in an order determined by an identification number, a next sequence, etc.) one after another in response to a predetermined action (for example, pressing a predetermined button or key). In this case, the user company terminal 10 (object data generation application 131) may be configured to accept input of content (attachment information SD) that is not linked to the marker in response to a specified action (e.g., pressing a specified button or key) when accepting the selection of the marker position on the AR generation screen.

[0101] (B-2) In the above embodiment, the user company terminal 10 (object data generation application 131) performs a process of 3D scanning the object TO to obtain mapping data (see steps S201 and S202 in Figure 16), but the mapping data may be reused (for example, by downloading and obtaining from the application server 30) from the mapping data TD of the object TO that is identical or has the same shape as the object TO that is the target of the AR scenario creation.

[0102] FIG. 21 is a flowchart showing a modified example (part 1) of the operation of generating object data (AR scenario data) regarding an object TO in response to an operation by an operator of a user company in the user company terminal 10 (object data generation application 131).

[0103] In FIG. 21, the same process steps as those in FIG. 16 above are denoted by the same reference numerals (step numbers).

[0104] First, it is assumed here that an operator operates the user company terminal 10, launches the object data generation application 131, and starts the object data generation process. When the object data generation application 131 starts the object data generation process, it first accepts information input processing (input processing for each item of the object management data DD) (S200). Note that the process of step S200 here is the same as that in FIG. 16 above, and therefore a detailed description will be omitted.

[0105] Next, the object data generation application 131 displays the image captured by the camera 22 on the touch panel display 21. At this time, the object data generation application 131 also displays a message (such as a message "point the camera so that the object is captured") to guide the operator to bring the object TO into the field of view of the camera 22. Here, the object data generation application 131 creates 3D scanning data of the object TO based on the image captured by the camera 22 (for example, an image captured by a compound eye camera) and the detection result of the sensor unit 24, and acquires the result (S301).

[0106] Next, the object data generation application 131 identifies candidate object data TD based on the acquired scanning results, presents the object names of the candidate object data TD to the user, and accepts selection of the object TO to be processed this time (S302).

[0107] The processes in steps S301 and S303 can be similar to the processes (steps S101 and S102) in the end-user terminal 20 (the object data display application 231) shown in FIG. 13 described above, and therefore detailed explanations will be omitted.

[0108] Through the above-described processing, the object data generation application 131 identifies object data TD (a set of mapping data MD and AR sequence data AD) corresponding to the scanned object TO and acquires it from the AR processing server application 31, and further acquires only the corresponding mapping data MD from the acquired object data TD (S303). Note that, if multiple mapping data exist in the identified object data TD, the mapping data MD closest to the scanning result (the scanning data acquired in step S301) may be acquired.

[0109] The subsequent processes in FIG. 21 (processing after acquisition of mapping data), that is, steps S203 to S206, are similar to the processes in the same steps in FIG. 16 described above, and therefore detailed description thereof will be omitted.

[0110] As described above, when creating an AR scenario in the user company terminal 10 (object data generation application 131), mapping data of other object data TD can be used.

[0111] The process of acquiring mapping data by 3D scanning requires a very large amount of processing if it is to be executed with high accuracy, and it is expected that depending on the specifications of the user company terminal 10, it may take a very long time or the storage capacity of the working memory or the like may be insufficient. Therefore, as described above, in the user company terminal 10 (object data generation application 131), when creating an AR scenario, efficient processing is possible by downloading mapping data of the object data TD of the object TO that is the same as or has the same shape as the object TO that is the target of AR scenario creation. That is, as shown in FIG. 7, by managing the object management data DD, mapping data MD, and AR sequence data AD in the object data TD, it is possible to reuse the mapping data as described above. When creating an AR scenario in the user company terminal 10 (object data generation application 131), the mapping data of other object data TD may be reused, for example, by the processing of the flowchart shown in FIG. 21.

[0112] (B-3) In the above embodiment (FIGS. 16 and 21 above), the process in which the user company terminal 10 (object data generation application 131) generates new object data TD has been described, but it may also be possible to edit already registered object data TD, or to allow multiple operators (user company terminals 10) to simultaneously edit a single object data TD (for example, adding, editing, deleting marker information and additional information, etc.).

[0113] FIG. 22 is a flowchart showing a modified example of the operation of editing the object data TD in the user company terminal 10 (object data generation application 131) in response to an operation by an operator of the user company.

[0114] First, it is assumed here that the operator operates the user company terminal 10, the object data generation application 131 is started, and the editing process of the object data is started. When the object data generation application 131 starts the editing process of the object data, it displays an image captured by the camera 22 on the touch panel display 21. At this time, the object data generation application 131 also displays a message (such as a message "point the camera so that the object is captured") to guide the operator to bring the object TO into the field of view of the camera 22. Then, it is assumed here that the object data generation application 131 creates 3D scanning data of the object TO based on the image captured by the camera 22 (for example, an image captured by a compound eye camera) and the detection result of the sensor unit 24, and acquires the result (S401).

[0115] Next, the object data generation application 131 identifies candidate object data TD based on the acquired scanning results, presents the object names of the candidate object data TD to the user, and accepts selection of the object TO to be processed this time (S402).

[0116] Through the above-described processing, the object data generation application 131 identifies object data TD corresponding to the scanned object TO and acquires it from the AR processing server application 31, and further reads corresponding mapping data MD and AR sequence data linked to the mapping data MD from the acquired object data TD (S403). Note that, if multiple mapping data exist for the identified object data TD, it may be possible to read the mapping data MD that is closest to the scanning result (scanning data acquired in step S301) and the AR sequence data AD linked to the mapping data MD.

[0117] The processing in steps S401 to S403 can be similar to the processing in steps S101 to S103 of the end-user terminal 20 (the object data display application 231) shown in FIG. 13 described above, and therefore detailed description thereof will be omitted.

[0118] Next, the object data generation application 131 performs AR processing based on the acquired mapping data MD and AR sequence data AD to display an AR generation screen as shown in Fig. 17(b) and to receive input of a marker position (3D coordinates) set by the operator (S404). The processing in step S404 is the same as that in step S203 described above, and therefore a detailed description thereof will be omitted.

[0119] Next, the object data generation application 131 performs a process of accepting input of the content (additional data) corresponding to the marker position input by the operator, and the object data generation application 131 temporarily stores the input marker position and the content (additional data) in association with each other (S405). The process of step S405 is similar to the process of step S204 described above, and therefore a detailed description thereof will be omitted.

[0120] Next, the object data generation application 131 generates marker information PD corresponding to the marker position retained in step S404 and attachment information SD corresponding to the content retained in step S404, links the generated marker information PD with the generated attachment information SD (for example, links them to each other as shown in Figure 10), transmits them to the application server 30 (AR processing server application 31), and registers them in the AR sequence data AD of the object data TD to be edited (S406).

[0121] Next, the object data generation application 131 judges whether the next marker position is input (S407), and if the next marker position is input, the process proceeds to step S404 described above, and if not, the process of editing the object data TD ends. The process of step S407 is the same as step S205 described above, and therefore a detailed description thereof will be omitted.

[0122] As described above, the user company terminal 10 (object data generation application 131) can accept editing of the object data TD (AR sequence data AD). Although Fig. 22 shows an example of adding a set of new marker information PD and attached information SD to the object data TD (AR sequence data AD), various editing processes such as deletion (e.g., deletion of registered marker information PD and attached information SD), addition of only attached information SD (e.g., addition of attached information SD to registered marker information PD), and change (e.g., change of marker position set in marker information PD, change of attached information SD added to marker information PD) may be accepted and reflected in the data on the application server 30 (AR processing server application 31).

[0123] In the user company terminal 10 (object data generation application 131), editing (addition, deletion, change) of each data in the object data TD (AR sequence data AD) may be performed simply by accepting data editing operations for each item in a database (data editing operations for a database having a structure as shown in Fig. 7) instead of an operation screen involving AR processing as shown in the flowchart of Fig. 22. In this case, the application server 30 (AR processing server application 31) may accept database editing processing (mainly processing for adding, changing, and deleting marker information and attached information constituting the AR sequence data) in accordance with a request from the user company terminal 10 (object data generation application 131).

[0124] As described above, the object data TD of this embodiment can accept edits on the object data TD (AR sequence data AD) with a simple operation in the user company terminal 10 (object data generation application 131) because the object data TD of this embodiment has a structure that allows easy replacement of individual data as described above. Also, since the object data TD of this embodiment has a structure that allows easy replacement of individual data as described above, edits (addition, change, deletion, etc. of the marker information PD and the attachment information SD) can be accepted from multiple user company terminals 10 at the same time.

[0125] (B-4) When the user company terminal 10 (object data generation application 131) performs the process of generating the object data TD (the process of FIG. 16 described above), 3D scanning data (scanning data based on the detection results of a camera, a sensor, etc.) acquired by the user company terminal 10 itself is used as the reference data RD from which the mapping data MD is generated. Instead of this, 3D modeling data (e.g., 3D modeling data such as CAD data from which the object TO is manufactured) stored in advance may be used. For example, in steps S201 and S202, the user company terminal 10 (object data generation application 131) may receive input of 3D modeling data from an operator. The 3D modeling data modeled after the object TO can be said to be more precise reference data RD than the 3D scanning data (scanning data based on the detection results of a camera, a sensor, etc.) acquired by the user company terminal 10 itself, and is therefore suitable for generating the mapping data MD. In the user company terminal 10 (object data generation application 131), the format of the 3D modeling data is not limited, and various formats (for example, various CAD format data, etc.) can be applied. In addition, the process of converting the 3D modeling data into the format of the mapping data MD can be the same as that of various AR middleware. [Explanation of symbols]

[0126] 1...information processing system, 10...user company terminal, 11...touch panel display, 12...camera, 13...information processing unit, 14...sensor unit, 15...microphone, 16...speaker, 20...end user terminal, 21...touch panel display, 22...camera, 23...information processing unit, 24...sensor unit, 25...microphone, 26...speaker, 30...application server, 31...AR processing server application, 32...data management unit, 131...object data generation application, 231...object data display application, 321...user company management data management unit, 322...object data management unit, AD...AR sequence data, AR...object description, AR...description, ARM...AR sequence management data, DD...object management data, MD...mapping data, PD...marker information, RD...reference data, SD...attachment information, TD...object data, TO...object.

Claims

1. In a data structure of object data in which an explanation of an object to be explained is described, the explanation being displayed on a computer in an AR format, Mapping data obtained as a result of mapping the surface of the object; and and AR display data including one or more sets of marker information indicating a position of a marker placed on the surface of the object in a coordinate system based on the mapping data and additional information including content related to the object corresponding to the marker information.

1. A data structure comprising:

2. 2. The data structure according to claim 1, wherein the mapping data is data obtained by mapping the entire object as a single model.

3. The data structure according to claim 1 , wherein the AR display data can include additional information that does not correspond to any of the marker information.

4. The data structure according to claim 1 , wherein the object data is a structure capable of including a plurality of the mapping data and the AR display data relating to the object.

5. A data display program that causes a computer to function as an AR display means for performing AR display based on object data having the data structure according to any one of claims 1 to 4.

6. A data generating program that causes a computer to function as object data generating means for generating object data having the data structure according to any one of claims 1 to 4.

7. 5. A data storage processing device comprising: a storage processing means for storing and processing object data having the data structure according to claim 1.

Images