Content display system, content display method, content display device, and content display program

Abstract

A content display system includes a content display device that displays content including at least one virtual object, and a determination processing device. The content display device includes a parameter calculation unit that calculates the initial position and initial orientation of the content display device as initial parameters, a parameter update unit that updates the position and orientation of the content display device, a display processing unit that superimposes and displays the content on a real object, a determination information transmission unit that transmits determination information to the determination processing device, and a determination result information reception unit that receives determination result information indicating a determination result from the determination processing device. The determination processing device includes a determination information reception unit that receives the determination information, a determination processing unit that determines whether the position of the object to be determined is abnormal, and a determination result information transmission unit that transmits the determination result information to the content display device.

Description

Technical Field

[0001] Embodiments of the present invention relate to a technique for displaying content.

Background Art

[0002] Conventionally, MR (Mixed Reality) technology for superimposing the real world and the virtual world and AR (Augmented Reality) technology for adding virtual information to the real world are known. Also, attempts have been made to introduce MR technology and AR technology into the workplace.

[0003] As this type of technology, there is known a content display device that MR-displays content including at least one virtual object, based on design information of a real object including a virtual object in which each component constituting the real object is represented as a three-dimensional model, and a captured image of the real object existing at the work site, and superimposes the content arranged in a virtual space with the origin at the position of the real object on the real object for MR display (Patent Document 1).

[0004] Further, when a virtual object included in the displayed content is selected and operated, this content display device changes the determination state for the virtual object including the normal state and the abnormal state. According to such a content display device, it is possible to realize a determination operation for a determination target using MR technology.

[0005] In the determination operation by this content display device, it is necessary for the operator to visually compare the determination target in the real object with the virtual object corresponding to this determination target, and the determination operation has been complicated.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] One problem is to more easily perform a determination operation on a determination object using MR technology.

[0008] Embodiments of the present invention are made to solve the above-described problems, and an object thereof is to provide a technology capable of more easily performing a determination operation on a determination object using MR technology.

Means for Solving the Problems

[0009] To solve the above problems, an embodiment of the present invention is a content display system including a content display device that displays content including at least one or more virtual objects and a determination processing device. The content display device includes a parameter calculation unit that calculates, as initial parameters, an initial position and an initial orientation of the content display device with the position of the real object as the origin based on a captured image of the real object; a parameter update unit that calculates displacements of the position and the orientation of the content display device and updates the position and the orientation of the content display device starting from the initial position and the initial orientation; a display processing unit that superimposes and displays the content arranged in a virtual space with the position of the real object as the origin on the real object; a determination information transmission unit that transmits determination information including the captured image and a display area of the virtual object superimposed on the captured image to the determination processing device; and a determination result information reception unit that receives determination result information indicating a determination result based on the determination information. The determination processing device includes a determination information reception unit that receives the determination information from the content display device; a determination processing unit that recognizes a determination target object in the real object and determines whether the position of the determination target object is abnormal based on a degree of overlap between a region of the determination target object and a region of a virtual object corresponding to the determination target object based on the received determination information; and a determination result information transmission unit that transmits the determination result information indicating the determined determination result to the content display device.

[0010] Further, an embodiment of the present invention is a content display device that displays content including at least one virtual object, and based on a captured image of a real object, calculates, as initial parameters, an initial position and an initial orientation of the content display device with the position of the real object as the origin, a parameter calculation unit, calculates a displacement of the position and orientation of the content display device, and updates the position and orientation of the content display device starting from the initial position and the initial orientation. A parameter update unit, and a display processing unit that superimposes and displays the content arranged in a virtual space with the position of the real object as the origin on the real object, and based on the captured image, recognizes a determination target object in the real object, and based on the degree of overlap between the region of the determination target object and the region of the virtual object corresponding to the determination target object, a determination processing unit that determines whether the position of the determination target object is abnormal.

Brief Description of Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] <First Embodiment> (Hardware Configuration) The hardware configuration of the content display system according to the first embodiment will be described. FIG. 1 is a schematic diagram showing the configuration of the content display system according to this embodiment. FIGS. 2 and 3 are block diagrams showing the hardware configurations of the content display device and the determination processing device, respectively.

[0014] As shown in FIG. 1, the content display system 1 according to this embodiment includes a content display device 10, a content management device 20, and a determination processing device 30. In this embodiment, the content display device 10 is a terminal device used by a user who views content related to real objects, and is configured as a portable terminal device, specifically, a tablet-type terminal device. The content management device 20 is a device that manages the content viewed using the content display device 10, and is configured as a PC (personal computer) in this embodiment. The determination processing device 30 is a device that performs determination and evaluation on the object to be determined in the real object, and is a device with higher data processing ability than the content display device 10. In this embodiment, the determination processing device 30 is configured as a PC in the same manner as the content management device 20.

[0015] Note that the content display device 10 may be configured as a smartphone or a head-mounted display, or may be configured as a device capable of performing display on a display or projection by a projector. Also, various functions executed by the content management device 20 and the determination processing device 30 described later may be executed by the content display device 10, or may be executed by a server device connected to the content display device 10 via the Internet.

[0016] An actual object is an object on which a user performs arbitrary operations, and is an actual existing object marked with a marker to be described in detail later. The actual object according to this embodiment is a member having at least one determination object that requires state determination and position evaluation. In this embodiment, the determination object is a welding dot which is a spot-welded part, and it is determined whether the position of the welding dot is abnormal. Here, the welding dot is determined to be abnormal when it is not at the original position defined in the design of the actual object, and when it is determined to be abnormal, it is evaluated how far it is separated from the original position.

[0017] The content display device 10 is connected to a network via a wireless LAN (Local Area Network) installed at the work site where the actual object is placed, and communicates with the content management device 20 and the determination processing device 30 via the network. Note that the communication between the content display device 10 and the content management device 20 and the determination processing device 30 may be performed via a wired connection using a USB (Univer Serial Bus) cable or the like.

[0018] As shown in FIG. 2, as hardware, the content display device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage device 13, a touch panel 14, a network I / F (Interface) 15, a camera 16, and a depth sensor 17. The CPU 11 and the RAM 12 cooperate to execute various functions, and the storage device 13 stores various data used for the processing executed by the various functions. The touch panel 14 has a display and a touch sensor, and is an input / output device provided on the front side of the content display device 10. The network I / F 15 performs wireless communication with the content management device 20 and the determination processing device 30.

[0019] The camera 16 is an imaging device provided on the back side of the content display device 10, and captures two-dimensional images and videos using visible light. The depth sensor 17 is provided on the back side of the content display device 10, and is a sensor that measures the distance from the content display device 10 to surrounding objects by projecting a laser and receiving the reflected light with a light receiving element. In this embodiment, this depth sensor 17 is a sensor that measures the distance from the content display device 10 to surrounding objects and generates a three-dimensional point cloud using the ToF (Time of Flight) method that converts the delay time of the received light pulse with respect to the projected light pulse into a distance. It is assumed that the imaging direction by the camera 16 and the measurement direction by the depth sensor 17 are the same.

[0020] As shown in FIG. 3, as hardware, the determination processing device 30 includes a CPU 21, a RAM 22, a storage device 23, an input / output I / F 24, and a network I / F 25. The CPU 21 and the RAM 22 cooperate to execute various functions, and the storage device 23 stores various data used for the processes executed by the various functions. The input / output I / F 24 performs input / output of data with input devices such as a keyboard and output devices such as a display connected to the determination processing device 30. The network I / F 25 performs wired communication and wireless communication with other devices including the content display device 10. The content management device 20 is configured in the same manner as the determination processing device 30 as hardware. Therefore, the description of the hardware configuration of the content management device 20 is omitted.

[0021] (Functional Configuration of Content Display Device) The functional configuration of the content display device according to the first embodiment will be described. FIG. 4 is a block diagram showing the functional configuration of the content display device according to the present embodiment.

[0022] As shown in FIG. 4, the content display device 10 includes, as functions, a parameter calculation unit 101, a parameter update unit 102, a content reception unit 103, a display processing unit 104, an imaging processing unit 105, a determination information transmission unit 106, a determination result reception unit 107, an evaluation information transmission unit 108, an evaluation result reception unit 109, and a memory processing unit 110.

[0023] The imaging processing unit 105 acquires time-series data of the captured image captured by the camera 16 and time-series data of depth information including the three-dimensional point cloud measured by the depth sensor 17. The parameter calculation unit 101 recognizes a marker in the captured image in which the real object is captured, which is acquired by the imaging processing unit 105, and calculates, as parameters, the initial position and the initial orientation of the content display device 10 in the marker coordinates with the marker as the origin based on the recognized marker.

[0024] The parameter update unit 102 calculates, for each frame, the displacement of the position and the orientation of the content display device 10 based on the displacement between the frames before and after the depth information acquired by the imaging processing unit 105, and updates the position and the orientation of the content display device 10 starting from the initial position and the initial orientation calculated by the parameter calculation unit 101 based on this displacement.

[0025] The content reception unit 103 receives the content transmitted from the content management device 20. The display processing unit 104 displays the content received by the content reception unit 103 on the touch panel 14 based on the position and the orientation of the content display device 10.

[0026] The determination information transmission unit 106 transmits determination information including the captured image acquired by the imaging processing unit 105 and the virtual object corresponding to the determination target to the determination processing device 30. The determination result reception unit 107 receives, from the determination processing device 30, a determination result indicating whether the position of the determination target is normal or abnormal.

[0027] The evaluation information transmission unit 108 transmits evaluation information including the captured image and depth information acquired by the imaging processing unit 105 and the virtual object corresponding to the object to be determined to the determination processing device 30. The evaluation result reception unit 109 receives evaluation result information indicating the position error of the object to be determined from the determination processing device 30. The storage processing unit 110 stores the evaluation result information in the storage device 13.

[0028] (Functional configuration of the content management device) The functional configuration of the content management device according to the first embodiment will be described. FIG. 5 is a block diagram showing the functional configuration of the content management device according to the present embodiment.

[0029] As shown in FIG. 5, the content management device 20 includes, as functions, a conversion unit 201, a management unit 202, and a content transmission unit 203. The conversion unit 201 converts three-dimensional data related to the real object into content. Note that examples of the three-dimensional data to be converted into content include CAD (Computer Aided Design) data, which is the design information of the real object. The management unit 202 manages the content converted by the conversion unit 201. The content transmission unit 203 transmits the content managed by the management unit 202 to the content display device 10.

[0030] The content includes at least one or more virtual objects. All virtual objects included in the content are assigned an object ID, which is a unique identifier. Among the virtual objects included in the content, at least one or more virtual objects are set with a determination flag indicating that they are objects to be determined. The virtual objects are all three-dimensional data, and for the virtual objects with the determination flag set, reference coordinates are set. The reference coordinates are three-axis coordinates with a point representing the virtual object as the origin, and as will be described in detail later, they are used to evaluate the position error of the object to be determined with respect to the virtual object. In this embodiment, the virtual object to be determined is the three-dimensional data of welding dots, and the reference coordinates are set with the center of the circular object exposed to the outside as the origin, two of the three axes (x, y axes) are in the radial direction, and the remaining one axis (z axis) is perpendicular to the two axes in the radial direction (see Fig. 16).

[0031] (Functional Configuration of the Determination Processing Device) The functional configuration of the determination processing device according to the first embodiment will be described. Fig. 6 is a block diagram showing the functional configuration of the determination processing device according to this embodiment.

[0032] As shown in Fig. 6, the determination processing device 30 includes, as functions, a determination information receiving unit 301, a determination processing unit 302, a determination result transmitting unit 303, an evaluation information receiving unit 304, an evaluation processing unit 305, and an evaluation result transmitting unit 306.

[0033] The determination information receiving unit 301 receives the determination information transmitted by the content display device 10. The determination processing unit 302 determines whether the position of the object to be determined is normal based on the determination information received by the determination information receiving unit 301. The determination result transmitting unit 303 transmits determination result information indicating the determination result for the object to be determined by the determination processing unit 302 to the content display device 10.

[0034] The evaluation information receiving unit 304 receives the evaluation information transmitted by the content display device 10. The evaluation processing unit 305 evaluates the position error of the object to be determined based on the evaluation information received by the evaluation information receiving unit 304. The evaluation result transmitting unit 306 transmits evaluation result information indicating the evaluation result for the object to be determined by the evaluation processing unit 305 to the content display device 10.

[0035] (Operation of the content display device) The operation of the content display device according to the first embodiment will be described. FIG. 7 is a flowchart showing the operation of the content display device according to the present embodiment. FIG. 8 is a schematic diagram showing the actual object according to the present embodiment. FIG. 9 is a schematic diagram for explaining the calculation of the initial position and the initial posture according to the present embodiment. FIG. 10 is a schematic diagram showing the content superimposed on the actual object according to the embodiment. In the flowchart shown in FIG. 7, it is assumed that the content has already been transmitted from the content management device to the content display device.

[0036] As shown in FIG. 7, first, the parameter calculation unit 101 determines whether or not the marker M is recognized in the captured image by the camera 16 (S101).

[0037] As shown in FIG. 8, the marker M is a two-dimensional image attached to the actual object RO, and defines the origin of the virtual space where the content is arranged in the real space. Examples of such a marker M include a two-dimensional image attached to the actual object RO, and concavo-convex in the form of a two-dimensional image engraved on the actual object RO. Note that the marker M does not necessarily have to be attached to the actual object RO, and may be attached to the floor surface, wall surface, ceiling surface, etc. around the actual object RO.

[0038] Alternatively, without using the marker M, the origin of the virtual space may be determined in the real space by a three-point measurement method or a position fixing method. The three-point measurement method is a method of selecting the shapes of at least three real objects and using the plane connecting the selected locations instead of the marker M. The position fixing method is a method of previously fixing the arrangement position of the real object and the usage position of the content display system 1, specifically the initial position of the content display device 10, and always using the content display system 1 from the same position to determine the origin of the virtual space in the real space. Thus, the origin of the virtual space may be determined in the real space by any method.

[0039] When the marker M is not recognized (S101, NO), the parameter calculation unit 101 determines again whether the marker M is recognized in the captured image by the camera 16 (S101). On the other hand, when the marker M is recognized (S101, YES), the parameter calculation unit 101 calculates the initial position and the initial orientation of the content display device 10 with the marker M as the origin as parameters (S102).

[0040] Here, the calculation of the initial position and the initial orientation will be described. As shown in FIG. 9, when the imaging range S including the marker M attached to the real object RO is imaged by the camera 16, the marker m on the captured image is enlarged, reduced, and deformed according to the positional relationship between the marker M and the content display device 10. The parameter calculation unit 101 calculates the relative position and the relative orientation of the content display device 10 with respect to the marker M from the size and the shape of the marker m based on the marker information indicating the shape and the size of the reference marker M. Further, the parameter calculation unit 101 calculates the position of the content display device 10 on the three axes (X, Y, Z in the figure) with the marker M as the origin and the rotation of the content display device 10 around the rotation axes (X1, Y1, Z1) parallel to these three axes from this relative position and the relative orientation, thereby calculating the initial position and the initial orientation of the content display device 10. In the present embodiment, the center point of the marker M is set as the origin O, and the center of the camera 16, specifically the center point of the image sensor, is set as the origin I of the rotation axis.

[0041] After calculating the initial position and the initial orientation, the current position and the current orientation of the content display device 10 are updated at a predetermined period by the parameter update unit 102 based on the displacement of the three-dimensional feature point group detected by the depth sensor 17. The parameter update unit 102 calculates the displacement of the position and the orientation of the content display device 10 based on the displacement of the three-dimensional feature point group before and after in time series, and accumulates this displacement in the initial position and the initial orientation or the position and the orientation updated last time, thereby sequentially updating the current position and the current orientation of the content display device 10. Note that, for the calculation of the displacement of the self-position and the self-orientation based on the displacement of the three-dimensional feature point group, known methods such as the SLAM (Simultaneous Localization and Mapping) method and the SfM (Structure from Motion) method are assumed to be used.

[0042] After calculating the initial position and the initial orientation, one of at least one or more contents transmitted by the content management device 20 is selected by the user of the content display device 10 (S103). Note that the selection of the content may be made based on the captured image of the real object RO. For example, based on the captured image of the real object RO when the position of the content display device 10 with respect to the real object RO is on the front side or the side surface side of the real object RO, the content corresponding to the real object RO may be selected.

[0043] After selecting the content, as shown in FIG. 10, the display processing unit 104 superimposes the content CN on the real object RO in the captured image and displays it on the touch panel 14 (S104). Here, it is assumed that the relative position with respect to the origin O in the virtual space is set in advance so that the content CN is displayed on the touch panel 14 in substantially the same orientation and size as the real object RO. In addition, the virtual object JVO set as the determination target may be displayed in a color different from other virtual objects so that the user can more easily visually recognize the determination target object JRO. Note that FIG. 10 shows the determination target objects JRO_1 and JRO_2 and the virtual objects JVO_1 and JVO_2 corresponding thereto.

[0044] After the content CN is superimposed, the determination information transmission unit 106 transmits the determination information to the determination processing device 30 (S105). The determination information includes the captured image acquired by the imaging processing unit 105 and the superimposition information regarding the virtual object JVO included in the content CN superimposed on this captured image. The superimposition information indicates the display area of the virtual object JVO superimposed on the captured image.

[0045] Next, the determination result receiving unit 107 determines whether it has received the determination result information from the determination processing device 30 (S106). The determination result information indicates the determination results for all the determination target objects JRO included in the captured image transmitted by the determination information transmission unit 106. The determination result indicates the object ID of the virtual object JVO corresponding to the determination target object JRO and whether the position of the determination target object JRO compared with this virtual object JVO is normal or abnormal.

[0046] When the determination result receiving unit 107 receives the determination result information (S106, YES), the evaluation information transmission unit 108 determines whether the virtual object in the captured image has been selected as an evaluation target by the user via the touch panel 14 (S107). Here, the evaluation information transmission unit 108 determines whether the virtual object JVO corresponding to the determination target object JRO determined to be abnormal among the virtual objects JVO superimposed on the captured image has been selected.

[0047] When the evaluation target is selected (S107, YES), the evaluation information transmission unit 108 transmits the evaluation information regarding the selected evaluation target to the determination processing device 30 (S108). The evaluation information includes the object ID of the selected virtual object JVO, the reference coordinates on the captured image of the selected virtual object JVO, the captured image, and the depth information.

[0048] Next, the evaluation result receiving unit 109 determines whether it has received the determination result information transmitted by the determination processing device 30 (S109). The determination result information indicates the relative position of the determination target object JRO with respect to the selected virtual object JVO as the position error of the determination target object JRO.

[0049] When the evaluation result receiving unit 109 receives the evaluation result information (S109, YES), the storage processing unit 110 stores the evaluation result information in the storage device 13 in association with the object ID of the selected virtual object JVO (S110). Next, the evaluation information transmission unit 108 determines again whether the virtual object has been selected by the user (S107).

[0050] On the other hand, when the evaluation result receiving unit 109 has not received the evaluation result information (S109, NO), the evaluation result receiving unit 109 determines again whether it has received the determination result information transmitted by the determination processing device 30 (S109).

[0051] Also, in step S107, when the evaluation target has not been selected (S107, NO), the evaluation information transmission unit 108 determines again whether the virtual object in the captured image has been selected as the evaluation target by the user via the touch panel 14 (S107).

[0052] (Operation of the determination processing device) The operation of the determination processing device according to the first embodiment will be described. FIG. 11 is a flowchart showing the operation of the determination processing device according to the present embodiment.

[0053] As shown in FIG. 11, first, the determination information receiving unit 301 determines whether it has received the determination information transmitted from the content display device 10 (S201).

[0054] When the determination information receiving unit 301 receives the determination information (S201, YES), the determination processing unit 302 executes a determination process described later (S202), and determines whether or not the number of times of the determination process is equal to or greater than a preset specified number of times (S203). On the other hand, when the determination information receiving unit 301 has not received the determination information (S201, NO), the determination information receiving unit 301 determines whether or not it has received the determination information transmitted again from the content display device 10 (S201).

[0055] When the number of times of the determination process is equal to or greater than the specified number of times (S203, YES), the determination result transmission unit 303 transmits determination result information indicating the determination result by the determination process to the content display device 10 (S204), and the evaluation information receiving unit 304 determines whether or not it has received the evaluation information transmitted from the content display device 10 (S205). On the other hand, when the number of times of the determination process is less than the specified number of times (S203, NO), the determination information receiving unit 301 determines whether or not it has received the determination information transmitted again from the content display device 10 (S201).

[0056] When the evaluation information receiving unit 304 receives the evaluation information (S205, YES), the evaluation processing unit 305 executes an evaluation process described later (S206), and the evaluation result transmission unit 306 transmits evaluation result information indicating the evaluation result by the evaluation process to the content display device 10 (S207). On the other hand, when the evaluation information receiving unit 304 has not received the evaluation information (S205, NO), the evaluation information receiving unit 304 determines whether or not it has received the evaluation information again (S205).

[0057] (Determination Process) The operation of the determination process according to the first embodiment will be described. FIG. 12 is a flowchart showing the determination process according to the present embodiment. FIG. 13 is a schematic diagram showing an object to be determined and a virtual object corresponding to the object to be determined according to the present embodiment.

[0058] As shown in FIG. 12, first, the determination processing unit 302 recognizes the area of the welding spot as the object to be determined JRO from the captured image included in the determination information (S301). Here, the recognition of the welding spot in the captured image is performed by a machine learning model that has been trained in advance based on the captured image including the welding spot.

[0059] Next, the determination processing unit 302 selects one unselected virtual object JVO from among the virtual objects JVO corresponding to the recognized object to be determined JRO (S302). Here, the virtual object JVO that is superimposed so as to be closest to the object to be determined JRO is selected as the virtual object JVO corresponding to the object to be determined JRO.

[0060] After the selection of the virtual object JVO, the determination processing unit 302 determines whether the degree of overlap between the selected virtual object JVO and the object to be determined JRO corresponding to this virtual object JVO is equal to or greater than a preset overlap threshold (S303). Here, as shown in FIG. 13, the degree of overlap indicates the degree to which the area occupied by the object to be determined JRO in the captured image and the area occupied by the virtual object JVO superimposed on the captured image overlap. Examples of the index used as the degree of overlap include the area of the overlapping region (overlap area) between the object to be determined JRO and the virtual object JVO, the ratio of the overlap area to the total area of the region occupied by the virtual object JVO, and the ratio of the overlap area to the total area of the region occupied by the object to be determined JRO.

[0061] When the degree of overlap is equal to or greater than the overlap threshold (S303, YES), the determination processing unit 302 determines that the position of the object to be determined JRO corresponding to the selected virtual object JVO is normal (S305), and determines whether there is a virtual object JVO that corresponds to the recognized object to be determined JRO and is unselected (S305). On the other hand, when the degree of overlap is less than the overlap threshold (S303, NO), the determination processing unit 302 determines that the position of the object to be determined JRO corresponding to the selected virtual object JVO is abnormal (S306), and determines whether there is an unselected virtual object JVO (S305).

[0062] When there is an unselected virtual object JVO (S305, YES), the determination processing unit 302 recognizes again the area of the welding mark as the determination target object JRO from the captured image included in the determination information (S301). On the other hand, when there is no unselected virtual object JVO (S305, NO), the determination processing unit 302 ends the determination processing.

[0063] Thus, according to the determination processing, the determination of the determination target object JRO is made based on the degree of overlap between the determination target object JRO and the virtual object JVO. In addition, when the determination target object JRO is not recognized, more specifically, when there is no determination target object JRO recognized within a predetermined distance range from the selected virtual object JVO in the first place, it may be determined that the determination target object JRO that should exist corresponding to the virtual object JVO is abnormal.

[0064] (Evaluation Processing) The operation of the evaluation processing according to the first embodiment will be described. FIG. 14 is a flowchart showing the evaluation processing according to the present embodiment. FIG. 15 is a schematic diagram showing the reference point of the determination target object according to the present embodiment. FIG. 16 is a schematic diagram showing the evaluation method for the determination target object according to the present embodiment.

[0065] As shown in FIG. 14, first, the evaluation processing unit 305 calculates the reference point of the welding mark, which is the determination target object JRO corresponding to the virtual object JVO indicated by the object ID included in the evaluation information (S401). Here, since the welding mark as the determination target object JRO is substantially circular, as shown in FIG. 15, with respect to the bounding box surrounding the area of the determination target object JRO in the captured image, the midpoint in the vertical direction of the bounding box and the position that is the midpoint in the horizontal direction of the bounding box are calculated as the reference point P1. Note that the center of gravity position of the area of the determination target object JRO may be calculated as the reference point.

[0066] Next, the evaluation processing unit 305 calculates the relative position of the determination object JRO with respect to the virtual object JVO (S402). Here, as shown in FIG. 16, the evaluation processing unit 305 calculates the position of the reference point P1 of the determination object JRO with respect to the origin P0 of the reference coordinates based on the reference point P1 of the determination object JRO, the depth information included in the evaluation information, and the reference coordinates of the virtual object JVO. Since the reference coordinates have three axes including the depth direction (z-axis), the deviation of the position of the determination object JRO provided at the recessed position with respect to the virtual object JVO_2 shown in FIG. 10 can be evaluated more accurately.

[0067] <Second Embodiment> (Overall Configuration and Hardware Configuration) The overall configuration of the content display system according to the second embodiment and the hardware of the content display device will be described. FIG. 17 is a schematic diagram showing the configuration of the content display system according to the present embodiment. FIG. 18 is a block diagram showing the hardware configuration of the content display device according to the present embodiment.

[0068] As shown in FIG. 17, the content display system 1A according to the present embodiment is different from the content display system 1 according to the first embodiment in that it does not include the determination processing device 30 and includes the content display device 10A instead of the content display device 10. As shown in FIG. 18, the content display device 10A is different from the content display device 10 according to the first embodiment in that it does not include the depth sensor 17.

[0069] (Functional Configuration of Content Display Device) The functional configuration of the content display device according to the second embodiment will be described. FIG. 19 is a block diagram showing the functional configuration of the content display device according to the present embodiment.

[0070] As shown in FIG. 19, the content display device 10A according to the present embodiment is different from the content display device 10 in that it does not include a determination information transmission unit 106, a determination result reception unit 107, an evaluation information transmission unit 108, and an evaluation result reception unit 109 as functions. Further, the content display device 10A is different from the content display device 10 in that it includes a parameter update unit 102A and an imaging processing unit 105A instead of the parameter update unit 102 and the imaging processing unit 105. Further, the content display device 10A is different from the content display device 10 in that it further includes a determination processing unit 111 and an evaluation processing unit 112.

[0071] The imaging processing unit 105A is different from the imaging processing unit 105 in that it only acquires the time-series data of the captured image captured by the camera 16. The parameter update unit 102A is different from the parameter update unit 102 in that it uses the VSLAM (Visual SLAM) method to calculate the displacement of the position and orientation of the content display device 10A for each frame based on the displacement between the frames before and after the captured image acquired by the imaging processing unit 105A.

[0072] The determination processing unit 111 corresponds to the determination processing unit 302 of the determination processing device 30 according to the first embodiment and executes determination processing. The evaluation processing unit 112 corresponds to the evaluation processing unit 305 of the determination processing device 30 and executes evaluation processing.

[0073] (Operation of the content display device) The operation of the content display device according to the second embodiment will be described. FIG. 20 is a flowchart showing the operation of the content display device according to the present embodiment. FIG. 21 is a schematic diagram showing a method for determining direct facing with respect to the determination object according to the present embodiment. FIG. 22 is a schematic diagram showing a method for evaluating the determination object according to the present embodiment.

[0074] As shown in FIG. 20, the content display device 10A according to the present embodiment is different from the content display device 10 in that it executes steps S501 to S505 instead of steps S105 to S110. Therefore, only the operations of steps S501 to S505 will be described below.

[0075] After step S104 is executed, the determination processing unit 111 executes determination processing (S501). Since this determination processing is the same as the determination processing executed by the determination processing unit 302 of the determination processing apparatus 30 according to the first embodiment, the description thereof is omitted.

[0076] After the execution of the determination processing, the evaluation processing unit 112 determines whether a virtual object JVO corresponding to the determination target object JRO determined to be abnormal has been selected by the user (S502). If the virtual object JVO has not been selected (S502, NO), the evaluation processing unit 112 determines again whether the virtual object JVO has been selected (S502).

[0077] On the other hand, when the virtual object JVO is selected (S502, YES), the evaluation processing unit 112 determines whether the content display device 10A is facing the determination target object JRO directly (S503). Here, the evaluation processing unit 112 determines that the content display device 10A is facing the determination target object JRO directly when the difference between the horizontal width LY and the vertical width LX of the bounding box surrounding the determination target object JRO corresponding to the selected virtual object JVO shown in FIG. 21 is equal to or less than a predetermined threshold value. That is, it is determined whether the welding dot, which is the determination target object JRO, is substantially circular on the captured image.

[0078] When the content display device 10A is not facing the determination target object JRO directly (S503, NO), the evaluation processing unit 112 determines whether the content display device 10A is facing the determination target object JRO directly based on the newly acquired current captured image (S503).

[0079] On the other hand, when the content display device 10A is facing the object to be determined JRO directly (S503, YES), the evaluation processing unit 112 executes evaluation processing (S504). As shown in FIG. 22, this evaluation processing is different from the evaluation processing according to the first embodiment in that, for only two axes (x and y axes) excluding the depth direction (z axis) among the three axes at the reference coordinates set for the selected virtual object JVO, the point for calculating the position of the reference point P1 with respect to the origin P0 is different.

[0080] After the execution of the evaluation processing, the storage processing unit 110 stores in the storage device 13 the relative position of the object to be determined JRO with respect to the virtual object JVO calculated by the evaluation processing in association with the object ID of the virtual object JVO (S505), and the evaluation processing unit 112 determines again whether the virtual object JVO has been selected (S502).

[0081] Thus, according to the content display device 10A according to the present embodiment, especially when the content display device 10A has sufficient information processing capabilities, it is possible to perform the determination and evaluation of the object to be determined JRO without interlocking with the determination processing device 30.

[0082] Also, by performing the evaluation processing when the object to be determined JRO is circular, it is possible to evaluate the deviation of the position of the object to be determined JRO_2 as shown in FIG. 10 with respect to the position of the virtual object JVO_2 using the content display device 10A not equipped with the depth sensor 17. Note that the determination in step S503 may be executed in the determination processing device 30.

[0083] Although the embodiments of the invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0084] 10, 10A Content Display Device 101 Parameter Calculation Unit 102 Parameter Update Unit 104 Display Processing Unit 106 Determination Information Sending Unit 107 Determination Result Receiving Unit 108 Evaluation Information Sending Unit 109 Evaluation Result Receiving Unit 111 Determination Processing Unit 112 Evaluation Processing Unit 301 Determination Information Receiving Unit 302 Determination Processing Unit 303 Determination Result Sending Unit 304 Evaluation Information Receiving Unit 305 Evaluation Processing Unit 306 Evaluation Result Sending Unit

Claims

1. A content display system including a content display device that displays content including at least one virtual object, and a determination processing device, The content display device comprises: a parameter calculation unit that calculates, as initial parameters, an initial position and an initial attitude of the content display device, with the position of the real object as an origin, based on a captured image of the real object; a parameter update unit that calculates a displacement of a position and an attitude of the content display device and updates the position and attitude of the content display device from the initial position and the initial attitude; a display processing unit that displays the content arranged in a virtual space having the position of the real object as an origin, the content being superimposed on the real object; a determination information transmitting unit configured to transmit, to the determination processing device, determination information including the captured image and a display area of ​​the virtual object superimposed on the captured image; a judgment result information receiving unit that receives judgment result information indicating a judgment result based on the judgment information from the judgment processing device; The determination processing device includes: a determination information receiving unit that receives the determination information from the content display device; a determination processing unit that recognizes a determination target in the real object based on the received determination information, and determines whether or not a position of the determination target is abnormal based on an overlapping degree between a region of the determination target and a region of a virtual object corresponding to the determination target; a determination result information transmission unit that transmits the determination result information indicating the determined determination result to the content display device.

2. The content display device comprises: an evaluation information transmitting unit that transmits evaluation information including a reference coordinate having an origin at a predetermined position in a virtual object corresponding to the determination target object determined to be abnormal and the captured image to the determination processing device; an evaluation result information receiving unit that receives evaluation result information indicating an evaluation result based on the evaluation information from the judgment processing device; The determination processing device includes: an evaluation information receiving unit that receives the evaluation information from the content display device; an evaluation processing unit that calculates, as the evaluation result, a relative position of an abnormality determination target that is the determination target determined to be abnormal with respect to a virtual object corresponding to the abnormality determination target based on the received evaluation information; The content display system according to claim 1 , further comprising an evaluation result transmission unit that transmits the evaluation result information indicating the calculated evaluation result to the content display device.

3. The content display system according to claim 2 , wherein the evaluation processing unit calculates a reference point within a region of the object to be determined as an abnormality, and calculates a position of the calculated reference point in the reference coordinates as the evaluation result.

4. 4. The content display system according to claim 3, wherein the reference point is a center position of a bounding box that surrounds the area of ​​the object for abnormality determination, or a center of gravity position of the area of ​​the object for abnormality determination.

5. The object to be determined is a welding point, The content display system according to any one of claims 2 to 4, characterized in that the evaluation processing unit calculates the evaluation result based on an captured image in which the difference between the vertical width and horizontal width of a bounding box surrounding the area of ​​the object to be determined as abnormal is less than a predetermined threshold value.

6. A content display method executed by a content display device that displays content including at least one virtual object and a determination processing device, comprising: The content display device, calculating, as initial parameters, an initial position and an initial attitude of the content display device with the position of the real object as an origin, based on a captured image of the real object; calculating a displacement of a position and an attitude of the content display device, and updating a position and an attitude of the content display device based on the initial position and the initial attitude; displaying the content arranged in a virtual space with the position of the real object as the origin, superimposed on the real object; transmitting, to the determination processing device, determination information including the captured image and a display area of ​​the virtual object superimposed on the captured image; The determination processing device, receiving the determination information from the content display device; recognize a determination target in the real object based on the received determination information, and determine whether or not a position of the determination target is abnormal based on an overlapping degree between a region of the determination target and a region of a virtual object corresponding to the determination target; transmitting determination result information indicating the determined determination result to the content display device; The content display device, The content display method further comprises receiving the determination result information from the determination processing device.

7. A content display device that displays content including at least one virtual object, a parameter calculation unit that calculates, as initial parameters, an initial position and an initial attitude of the content display device, with the position of the real object as an origin, based on a captured image of the real object; a parameter update unit that calculates a displacement of a position and an attitude of the content display device and updates the position and attitude of the content display device from the initial position and the initial attitude; a display processing unit that displays the content arranged in a virtual space having the position of the real object as an origin, the content being superimposed on the real object; a determination processing unit that recognizes a determination target in the real object based on the captured image, and determines whether or not a position of the determination target is abnormal based on an overlapping degree between a region of the determination target and a region of a virtual object corresponding to the determination target; A content display device comprising:

8. A content display program executed on a computer that displays content including at least one virtual object, The computer a parameter calculation unit that calculates, as initial parameters, an initial position and an initial orientation of the computer with the position of the real object as an origin, based on a captured image of the real object; a parameter update unit that calculates a displacement of a position and an attitude of the computer and updates the position and the attitude of the computer from the initial position and the initial attitude; a display processing unit that displays the content arranged in a virtual space having the position of the real object as an origin, the content being superimposed on the real object; A content display program that functions as a judgment processing unit that recognizes a judgment target in the real object based on the captured image, and judges whether or not the position of the judgment target is abnormal based on the degree of overlap between the area of ​​the judgment target and the area of ​​a virtual object corresponding to the judgment target.

Images