Content display system, content display method, and content display device

Abstract

One embodiment of the present invention is a content display system including: a content display device that displays content including at least one or more virtual objects; and a determination processing device. The content display device includes: a parameter calculation unit that calculates an initial position and an initial posture of the content display device as initial parameters; a parameter updating unit that updates the position and the posture of the content display device; a display processing unit that displays the content that is superimposed on a real object; a determination information transmission unit that transmits information for determination to the determination processing device; and a determination result information reception unit that receives determination result information indicating a determination result from the determination processing device. The determination processing device includes: a determination information reception unit that receives the information for determination; a determination processing unit that determines whether the position of an object for determination is abnormal; and a determination result information transmission unit that transmits the determination result information to the content display device.

Description

Content display system, content display method, and content display device

[0001] TECHNICAL FIELD Embodiments of the present invention relate to a technique for displaying content.

[0002] Conventionally, mixed reality (MR) technology, which overlays the real world with a virtual world, and augmented reality (AR) technology, which adds virtual information to the real world, are known. Furthermore, attempts are being made to introduce MR technology and AR technology into work sites.

[0003] One such technology is a content display device that displays content including at least one virtual object in mixed reality (MR), and that displays content arranged in a virtual space with the position of the real object as the origin by superimposing it on the real object based on design information of the real object, including virtual objects in which each part constituting the real object is represented as a three-dimensional model, and on an image of the real object at a work site (Patent Document 1).

[0004] Furthermore, when a virtual object included in the displayed content is selected, the content display device changes the judgment state of the virtual object, which includes a normal state and an abnormal state. With this content display device, it is possible to perform a judgment operation on an object to be judged using MR technology.

[0005] In the determination work using this content display device, the worker must visually compare the determination target in the real object with the virtual object corresponding to this determination target, making the determination work cumbersome.

[0006] JP 2023-60799 A

[0007] One of the challenges is to make it easier to perform the evaluation of an object to be evaluated using MR technology.

[0008] The embodiments of the present invention have been made to solve the above-mentioned problems, and aim to provide a technology that makes it easier to perform evaluation work on an object to be evaluated using MR technology.

[0009] In order to solve the above-described problems, an embodiment of the present invention is a content display system including a content display device that displays content including at least one or more virtual objects, and a determination processing device, wherein the content display device includes a parameter calculation unit that calculates, as initial parameters, an initial position and initial orientation of the content display device, based on a captured image of the real object, with the position of the real object as the origin; a parameter update unit that calculates displacement of the position and orientation of the content display device and updates the position and orientation of the content display device, based on the initial position and initial orientation; a display processing unit that displays content arranged in a virtual space, with the position of the real object as the origin, superimposed on the real object; and a judgment result information receiving unit that receives, from the judgment processing device, judgment information indicating a judgment result based on the judgment information. The judgment processing device includes: a judgment information receiving unit that receives the judgment information from the content display device; a judgment processing unit that recognizes a judgment target in the real object based on the received judgment information, and determines whether or not the position of the judgment target is abnormal based on the degree of overlap between the area of ​​the judgment target and the area of ​​a virtual object corresponding to the judgment target; and a judgment result information transmitting unit that transmits the judgment result information indicating the judgment result to the content display device.

[0010] Moreover, an embodiment of the present invention is a content display device that displays content including at least one or more virtual objects, and includes a parameter calculation unit that calculates, based on a captured image of a real object, an initial position and initial attitude of the content display device as initial parameters, with the position of the real object as an origin; a parameter update unit that calculates displacement of the position and attitude of the content display device and updates the position and attitude of the content display device with the initial position and initial attitude as a starting point; a display processing unit that displays the content arranged in a virtual space with the position of the real object as an origin, superimposed on the real object; and a determination processing unit that recognizes a determination target in the real object based on the captured image, and determines whether the position of the determination target is abnormal based on the degree of overlap between the area of ​​the determination target and the area of ​​a virtual object corresponding to the determination target.

[0011] 1 is a schematic diagram showing the configuration of a content display system according to a first embodiment. FIG. 1 is a block diagram showing the hardware configuration of a content display device according to the first embodiment. FIG. 2 is a block diagram showing the hardware configuration of a determination processing device according to the first embodiment. FIG. 3 is a block diagram showing the functional configuration of a content display device according to the first embodiment. FIG. 4 is a block diagram showing the functional configuration of a content management device according to the first embodiment. FIG. 5 is a block diagram showing the functional configuration of a determination processing device according to the first embodiment. FIG. 6 is a flowchart showing the operation of a content display device according to the first embodiment. FIG. 7 is a schematic diagram showing a real object according to the first embodiment. FIG. 8 is a schematic diagram for explaining calculation of an initial position and an initial attitude according to the first embodiment. FIG. 9 is a schematic diagram showing content superimposed on a real object according to the first embodiment. FIG. 10 is a flowchart showing the operation of a determination processing device according to the first embodiment. FIG. 11 is a flowchart showing determination processing according to the first embodiment. FIG. 12 is a schematic diagram showing a determination target and a virtual object corresponding to this determination target according to the first embodiment. FIG. 13 is a flowchart showing evaluation processing according to the first embodiment. FIG. 14 is a schematic diagram showing a reference point of a determination target according to the first embodiment. FIG. 15 is a schematic diagram showing an evaluation method for a determination target according to the first embodiment. FIG. 16 is a schematic diagram showing the configuration of a content display system according to a second embodiment. FIG. 17 is a block diagram showing the hardware configuration of a content display device according to the second embodiment. FIG. 18 is a block diagram showing the functional configuration of a content display device according to the second embodiment. FIG. 19 is a flowchart showing the operation of a content display device according to the second embodiment. FIG. 19 is a schematic diagram showing a facing determination method for a determination target according to the second embodiment. FIG. 19 is a schematic diagram showing an evaluation method for a determination target according to the second embodiment.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] First Embodiment (Hardware Configuration) The hardware configuration of a content display system according to the first embodiment will be described. Fig. 1 is a schematic diagram showing the configuration of the content display system according to this embodiment. Figs. 2 and 3 are block diagrams showing the hardware configurations of a content display device and a determination processing device, respectively.

[0014] As shown in FIG. 1 , a content display system 1 according to this embodiment includes a content display device 10, a content management device 20, and a determination processing device 30. In this embodiment, the content display device 10 is a terminal device used by a user who views content related to real objects, and is configured as a portable terminal device, specifically a tablet terminal device. The content management device 20 is a device that manages content viewed using the content display device 10, and in this embodiment, is configured as a PC (personal computer). The determination processing device 30 is a device that performs determination and evaluation on a determination target in a real object, and is a device with higher data processing capabilities than the content display device 10. In this embodiment, the determination processing device 30 is configured as a PC, just like the content management device 20.

[0015] The content display device 10 may be configured as a smartphone or a head-mounted display, or as a device capable of displaying on a display or projecting using a projector. The various functions executed by the content management device 20 and the determination processing device 30, which will be described later, may be executed by the content display device 10, or may be executed by a server device connected to the content display device 10 via the Internet.

[0016] A real object is a target on which a user performs an arbitrary operation, and is a real object to which a marker, which will be described in detail later, is attached. The real object according to this embodiment is a component having at least one or more determination targets, the state of which needs to be determined and the position of which needs to be evaluated. In this embodiment, the determination targets are welding points, which are spot-welded portions, and it is determined whether the position of the welding point is abnormal. Here, a welding point is determined to be abnormal if it is not in the original position determined in the design of the real object, and if it is determined to be abnormal, the extent to which it is separated from the original position is evaluated.

[0017] The content display device 10 is connected to a network via a wireless LAN (Local Area Network) installed at a work site where real objects are arranged, and communicates via the network with the content management device 20 and the determination processing device 30. Note that communication between the content display device 10 and the content management device 20 and the determination processing device 30 may also be performed via a wired connection using a USB (Universal Serial Bus) cable or the like.

[0018] As shown in FIG. 2 , the content display device 10 includes, as hardware, a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage device 13, a touch panel 14, a network I / F (Interface) 15, a camera 16, and a depth sensor 17. The CPU 11 and RAM 12 work together to execute various functions, and the storage device 13 stores various data used in processes executed by the various functions. The touch panel 14 has a display and a touch sensor, and is an input / output device provided on the front side of the content display device 10. The network I / F 15 communicates wirelessly with the content management device 20 and the determination processing device 30.

[0019] The camera 16 is provided on the rear side of the content display device 10 and is an imaging device that captures two-dimensional images and videos using visible light. The depth sensor 17 is provided on the rear side of the content display device 10 and is a sensor that measures the distance from the content display device 10 to surrounding objects by projecting a laser and receiving the reflected light with a light-receiving element. In this embodiment, the depth sensor 17 is a sensor that measures the distance from the content display device 10 to surrounding objects and converts the distance into a three-dimensional point cloud using a ToF (Time of Flight) method that converts the delay time between a projected light pulse and a received light pulse into distance. It is assumed that the imaging direction of the camera 16 and the measurement direction of the depth sensor 17 are the same.

[0020] As shown in Fig. 3, the determination processing device 30 includes, as hardware, a CPU 21, a RAM 22, a storage device 23, an input / output I / F 24, and a network I / F 25. The CPU 21 and RAM 22 cooperate to execute various functions, and the storage device 23 stores various data used in processes executed by the various functions. The input / output I / F 24 inputs and outputs data to and from input devices such as a keyboard and output devices such as a display connected to the determination processing device 30. The network I / F 25 performs wired and wireless communication with other devices including the content display device 10. The content management device 20 has the same hardware configuration as the determination processing device 30. Therefore, a description of the hardware configuration of the content management device 20 will be omitted.

[0021] (Functional Configuration of Content Display Device) A description will now be given of the functional configuration of the content display device according to the first embodiment. Fig. 4 is a block diagram showing the functional configuration of the content display device according to this embodiment.

[0022] As shown in Figure 4, the content display device 10 has, as its functions, a parameter calculation unit 101, a parameter update unit 102, a content receiving unit 103, a display processing unit 104, an imaging processing unit 105, a judgment information transmission unit 106, a judgment result receiving unit 107, an evaluation information transmission unit 108, an evaluation result receiving unit 109, and a memory processing unit 110.

[0023] The imaging processing unit 105 acquires time-series data of captured images captured by the camera 16 and time-series data of depth information including a three-dimensional point cloud measured by the depth sensor 17. The parameter calculation unit 101 recognizes markers in the captured images acquired by the imaging processing unit 105 in which real objects are captured, and calculates, based on the recognized markers, the initial position and initial orientation of the content display device 10 in marker coordinates with the markers as the origin, as parameters.

[0024] The parameter update unit 102 calculates the displacement of the position and attitude of the content display device 10 for each frame based on the displacement between previous and next frames of the depth information acquired by the imaging processing unit 105, and updates the position and attitude of the content display device 10 based on this displacement, starting from the initial position and initial attitude calculated by the parameter calculation unit 101.

[0025] The content receiving unit 103 receives content transmitted from the content management device 20. The display processing unit 104 displays the content received by the content receiving unit 103 on the touch panel 14 based on the position and orientation of the content display device 10.

[0026] The determination information transmitting unit 106 transmits determination information including the captured image acquired by the imaging processing unit 105 and the virtual object corresponding to the determination object to the determination processing device 30. The determination result receiving unit 107 receives from the determination processing device 30 a determination result indicating whether the position of the determination object is normal or abnormal.

[0027] The evaluation information transmitting unit 108 transmits evaluation information including the captured image and depth information acquired by the imaging processing unit 105, and the virtual object corresponding to the object to be determined, to the determination processing device 30. The evaluation result receiving unit 109 receives evaluation result information indicating the position error of the object to be determined from the determination processing device 30. The storage processing unit 110 stores the evaluation result information in the storage device 13.

[0028] (Functional Configuration of Content Management Apparatus) A functional configuration of the content management apparatus according to the first embodiment will be described below. Fig. 5 is a block diagram showing the functional configuration of the content management apparatus according to this embodiment.

[0029] As shown in Fig. 5, content management device 20 has, as its functions, a conversion unit 201, a management unit 202, and a content transmission unit 203. Conversion unit 201 converts three-dimensional data related to real objects into content. The three-dimensional data converted into content includes CAD (Computer Aided Design) data, which is design information for real objects. Management unit 202 manages the content converted by conversion unit 201. Content transmission unit 203 transmits the content managed by management unit 202 to content display device 10.

[0030] The content includes at least one virtual object. All virtual objects included in the content are assigned an object ID, which is a unique identifier. At least one of the virtual objects included in the content has a determination flag set to indicate that it is a target for determination. All virtual objects are three-dimensional data, and reference coordinates are set for virtual objects for which a determination flag is set. The reference coordinates are three-axis coordinates with a point representing the virtual object as the origin, and are used to evaluate the position error of the target for determination relative to the virtual object, as will be described in detail later. In this embodiment, the target virtual object is three-dimensional data of a welding point, and the reference coordinates are set so that the origin is the center of a circular object exposed to the outside, two mutually orthogonal axes (x-axis, y-axis) are oriented in radial directions, and the remaining axis (z-axis) is orthogonal to the two radial axes (see FIG. 16 ).

[0031] (Functional Configuration of the Determination Processing Device) A functional configuration of the determination processing device according to the first embodiment will be described below. Fig. 6 is a block diagram showing the functional configuration of the determination processing device according to this embodiment.

[0032] As shown in Figure 6, the judgment processing device 30 has the following functions: a judgment information receiving unit 301, a judgment processing unit 302, a judgment result transmitting unit 303, an evaluation information receiving unit 304, an evaluation processing unit 305, and an evaluation result transmitting unit 306.

[0033] The determination information receiving unit 301 receives the determination information transmitted by the content display device 10. The determination processing unit 302 determines whether or not the position of the object to be determined is normal based on the determination information received by the determination information receiving unit 301. The determination result transmitting unit 303 transmits, to the content display device 10, determination result information indicating the determination result made by the determination processing unit 302 on the object to be determined.

[0034] The evaluation information receiving unit 304 receives the evaluation information transmitted by the content display device 10. The evaluation processing unit 305 evaluates the position error of the object to be determined based on the evaluation information received by the evaluation information receiving unit 304. The evaluation result transmitting unit 306 transmits evaluation result information indicating the evaluation result of the object to be determined by the evaluation processing unit 305 to the content display device 10.

[0035] (Operation of content display device) The operation of the content display device according to the first embodiment will be described. Fig. 7 is a flowchart showing the operation of the content display device according to this embodiment. Fig. 8 is a schematic diagram showing a real object according to this embodiment. Fig. 9 is a schematic diagram for explaining calculation of the initial position and initial orientation according to this embodiment. Fig. 10 is a schematic diagram showing content superimposed on a real object according to this embodiment. It should be noted that in the flowchart shown in Fig. 7, it is assumed that content has already been transmitted from the content management device to the content display device.

[0036] As shown in FIG. 7, first, the parameter calculation unit 101 determines whether or not the marker M is recognized in the image captured by the camera 16 (S101).

[0037] 8, the marker M is a two-dimensional image attached to the real object RO, and defines the origin of the virtual space in which the content is placed in real space. Examples of such a marker M include a two-dimensional image attached to the real object RO, and two-dimensional image-like projections and depressions imprinted on the real object RO. Note that the marker M does not necessarily have to be attached to the real object RO, and may be attached to the floor, wall, ceiling, or other surfaces around the real object RO.

[0038] Furthermore, the origin of the virtual space may be determined in real space by a triangulation method or a position fixing method without using the marker M. The triangulation method is a method in which the shape of at least three real objects is selected and a plane connecting the selected points is used instead of the marker M. The position fixing method is a method in which the placement position of the real object and the use position of the content display system 1, specifically the initial position of the content display device 10, are fixed in advance, and the content display system 1 is always used from the same position, thereby determining the origin of the virtual space in real space. In this way, the origin of the virtual space may be determined in real space by any method.

[0039] If the marker M is not recognized (S101, NO), the parameter calculation unit 101 again determines whether or not the marker M is recognized in the image captured by the camera 16 (S101). On the other hand, if the marker M is recognized (S101, YES), the parameter calculation unit 101 calculates, as parameters, the initial position and initial orientation of the content display device 10 with the marker M as the origin (S102).

[0040] Here, the calculation of the initial position and initial orientation will be described. As shown in FIG. 9 , when an image capture range S including a marker M added to a real object RO is captured by the camera 16, the marker m in the captured image is enlarged, reduced, and deformed depending on the positional relationship between the marker M and the content display device 10. The parameter calculation unit 101 calculates the relative position and orientation of the content display device 10 with respect to the marker M from the size and shape of the marker m, based on marker information indicating the shape and size of the reference marker M. Furthermore, the parameter calculation unit 101 calculates the position of the content display device 10 on three axes (X, Y, and Z in the figure) with the marker M as the origin, and the rotation of the content display device 10 around rotation axes (X1, Y1, and Z1) parallel to these three axes, thereby calculating the initial position and initial orientation of the content display device 10. Note that in this embodiment, the center point of the marker M is defined as the origin O, and the center of the camera 16, specifically the center point of the image sensor, is defined as the origin I of the rotation axes.

[0041] After the initial position and initial orientation are calculated, the current position and orientation of the content display device 10 are updated at predetermined intervals by the parameter update unit 102 based on the displacement of the three-dimensional feature point group detected by the depth sensor 17. The parameter update unit 102 calculates the displacement of the position and orientation of the content display device 10 based on the displacement of the three-dimensional feature point group that occurs back and forth in time series, and adds this displacement to the initial position and orientation or the previously updated position and orientation, thereby sequentially updating the current position and orientation of the content display device 10. Note that the calculation of the displacement of the content display device's own position and orientation based on the displacement of the three-dimensional feature point group uses a known method such as the SLAM (Simultaneous Localization and Mapping) method or the SfM (Structure from Motion) method.

[0042] After the initial position and initial attitude are calculated, one of the at least one piece of content transmitted by the content management device 20 is selected by the user of the content display device 10 (S103). The selection of content may be made based on a captured image of the real object RO. For example, content corresponding to the real object RO may be selected based on a captured image of the real object RO when the position of the content display device 10 relative to the real object RO is on the front side or side side of the real object RO.

[0043] After the content is selected, the display processing unit 104 superimposes the content CN on the real object RO in the captured image and displays it on the touch panel 14 (S104), as shown in Fig. 10 . Here, the relative position of the content CN with respect to the origin O in the virtual space is set in advance so that the content CN is displayed on the touch panel 14 with substantially the same posture and size as the real object RO. Furthermore, the virtual object JVO set as the determination target may be displayed in a color different from that of other virtual objects so that the user can more easily visually recognize the determination target JRO. Note that Fig. 10 shows determination targets JRO_1 and JRO_2 and their corresponding virtual objects JVO_1 and JVO_2.

[0044] After the content CN is superimposed, the determination information transmitting unit 106 transmits the determination information to the determination processing device 30 (S105). The determination information includes the captured image acquired by the imaging processing unit 105 and superimposition information regarding the virtual object JVO included in the content CN to be superimposed on the captured image. The superimposition information indicates the display area of ​​the virtual object JVO to be superimposed on the captured image.

[0045] Next, the determination result receiving unit 107 determines whether or not determination result information has been received from the determination processing device 30 (S106). The determination result information indicates the determination results for all determination objects JRO included in the captured image transmitted by the determination information transmitting unit 106. The determination result indicates the object ID of the virtual object JVO corresponding to the determination object JRO, and whether the position of the determination object JRO compared to this virtual object JVO is normal or abnormal.

[0046] When the determination result receiving unit 107 receives the determination result information (S106, YES), the evaluation information transmitting unit 108 determines whether or not a virtual object in the captured image has been selected as an evaluation target by the user via the touch panel 14 (S107). Here, the evaluation information transmitting unit 108 determines whether or not a virtual object JVO corresponding to the determination target JRO determined to be abnormal has been selected from among the virtual objects JVO superimposed on the captured image.

[0047] When an evaluation object is selected (S107, YES), the evaluation information transmitting unit 108 transmits evaluation information related to the selected evaluation object to the determination processing device 30 (S108). The evaluation information includes the object ID of the selected virtual object JVO, the reference coordinates of the selected virtual object JVO on the captured image, the captured image, and depth information.

[0048] Next, the evaluation result receiving unit 109 determines whether or not the determination result information transmitted by the determination processing device 30 has been received (S109). The determination result information indicates the relative position of the determination target object JRO with respect to the selected virtual object JVO as a position error of the determination target object JRO.

[0049] If the evaluation result receiving unit 109 receives the evaluation result information (S109, YES), the storage processing unit 110 stores the evaluation result information in the storage device 13 in association with the object ID of the selected virtual object JVO (S110). Next, the evaluation information sending unit 108 again determines whether or not the virtual object has been selected by the user (S107).

[0050] On the other hand, if the evaluation result receiving unit 109 has not received the evaluation result information (S109, NO), the evaluation result receiving unit 109 again determines whether or not it has received the judgment result information sent by the judgment processing device 30 (S109).

[0051] Also, in step S107, if an evaluation object is not selected (S107, NO), the evaluation information sending unit 108 again determines whether or not a virtual object in the captured image has been selected as an evaluation object by the user via the touch panel 14 (S107).

[0052] (Operation of the Determination Processing Device) The operation of the determination processing device according to the first embodiment will be described below. Fig. 11 is a flowchart showing the operation of the determination processing device according to this embodiment.

[0053] As shown in FIG. 11, first, the determination information receiving unit 301 determines whether or not determination information transmitted from the content display device 10 has been received (S201).

[0054] If the determination information receiving unit 301 has received the determination information (S201, YES), the determination processing unit 302 executes a determination process (S202) to be described later, and determines whether the number of times the determination process has been performed is equal to or greater than a predetermined number of times (S203). On the other hand, if the determination information receiving unit 301 has not received the determination information (S201, NO), the determination information receiving unit 301 again determines whether or not it has received the determination information transmitted from the content display device 10 (S201).

[0055] If the number of times the judgment process has been performed is equal to or greater than the designated number (S203, YES), the judgment result transmitting unit 303 transmits judgment result information indicating the judgment result of the judgment process to the content display device 10 (S204), and the evaluation information receiving unit 304 determines whether or not it has received the evaluation information transmitted from the content display device 10 (S205). On the other hand, if the number of times the judgment process has been performed is not equal to or greater than the designated number (S203, NO), the judgment information receiving unit 301 again determines whether or not it has received the judgment information transmitted from the content display device 10 (S201).

[0056] If the evaluation information receiving unit 304 has received the evaluation information (S205, YES), the evaluation processing unit 305 executes the evaluation process described below (S206), and the evaluation result sending unit 306 sends evaluation result information indicating the evaluation result of the evaluation process to the content display device 10 (S207). On the other hand, if the evaluation information receiving unit 304 has not received the evaluation information (S205, NO), the evaluation information receiving unit 304 determines again whether or not the evaluation information has been received (S205).

[0057] (Determination Process) The operation of the determination process according to the first embodiment will be described. Fig. 12 is a flowchart showing the determination process according to this embodiment. Fig. 13 is a schematic diagram showing a determination target object and a virtual object corresponding to this determination target object according to this embodiment.

[0058] 12, first, the determination processing unit 302 recognizes the area of ​​the welding point as the determination object JRO from the captured image included in the determination information (S301). Here, the recognition of the welding point in the captured image is performed by a machine learning model that has been trained in advance based on the captured image including the welding point.

[0059] Next, the determination processing unit 302 selects one unselected virtual object JVO from among the virtual objects JVO corresponding to the recognized determination object JRO (S302). Here, the virtual object JVO superimposed so as to be closest to the determination object JRO is selected as the virtual object JVO corresponding to the determination object JRO.

[0060] After selecting the virtual object JVO, the determination processing unit 302 determines whether the overlap between the selected virtual object JVO and the determination target object JRO corresponding to this virtual object JVO is equal to or greater than a preset overlap threshold (S303). Here, the overlap indicates the degree of overlap between the area occupied by the determination target object JRO in the captured image and the area occupied by the virtual object JVO superimposed on the captured image, as shown in Fig. 13. Indicators used as the overlap include the area of ​​the area where the determination target object JRO and the virtual object JVO overlap (overlapping area), the ratio of the overlapping area to the total area of ​​the area occupied by the virtual object JVO, and the ratio of the overlapping area to the total area of ​​the area occupied by the determination target object JRO.

[0061] If the degree of overlap is equal to or greater than the overlap threshold (S303, YES), the determination processing unit 302 determines that the position of the determination object JRO corresponding to the selected virtual object JVO is normal (S305), and determines whether or not there is an unselected virtual object JVO that corresponds to the recognized determination object JRO (S305). On the other hand, if the degree of overlap is less than the overlap threshold (S303, NO), the determination processing unit 302 determines that the position of the determination object JRO corresponding to the selected virtual object JVO is abnormal (S306), and determines whether or not there is an unselected virtual object JVO (S305).

[0062] If an unselected virtual object JVO exists (S305, YES), the determination processing unit 302 again recognizes the area of ​​the welding point as the determination target JRO from the captured image included in the determination information (S301). On the other hand, if an unselected virtual object JVO does not exist (S305, NO), the determination processing unit 302 ends the determination process.

[0063] In this way, according to the determination process, the determination of the determination object JRO is performed based on the degree of overlap between the determination object JRO and the virtual object JVO. Note that if the determination object JRO is not recognized, more specifically, if the recognized determination object JRO does not exist within a predetermined distance range from the selected virtual object JVO, the determination object JRO that should exist corresponding to the virtual object JVO may be determined to be abnormal.

[0064] (Evaluation Processing) The operation of the evaluation processing according to the first embodiment will be described. Fig. 14 is a flowchart showing the evaluation processing according to this embodiment. Fig. 15 is a schematic diagram showing a reference point of an object to be determined according to this embodiment. Fig. 16 is a schematic diagram showing an evaluation method for an object to be determined according to this embodiment.

[0065] 14, first, the evaluation processing unit 305 calculates a reference point of the welding point, which is the object to be determined JRO, corresponding to the virtual object JVO indicated by the object ID included in the evaluation information (S401). Here, since the welding point, which is the object to be determined JRO, is approximately circular, the evaluation processing unit 305 calculates, as a reference point P1, a position that is the midpoint in the vertical direction of the bounding box that encloses the area of ​​the object to be determined JRO in the captured image, and the midpoint in the horizontal direction of the bounding box, as shown in FIG. 15. Note that the center of gravity of the area of ​​the object to be determined JRO may also be calculated as the reference point.

[0066] Next, the evaluation processing unit 305 calculates the relative position of the determination object JRO with respect to the virtual object JVO (S402). Here, as shown in Fig. 16, the evaluation processing unit 305 calculates the position of the reference point P1 with respect to the origin P0 of the reference coordinates based on the reference point P1 of the determination object JRO, the depth information included in the evaluation information, and the reference coordinates of the virtual object JVO. Note that by having three axes including the depth direction (z-axis), it is possible to more accurately evaluate the positional deviation of the determination object JRO, which is provided in a recessed position with respect to the virtual object JVO_2 shown in Fig. 10.

[0067] Second Embodiment (Overall Configuration and Hardware Configuration) The overall configuration of a content display system and the hardware of a content display device according to a second embodiment will be described. Fig. 17 is a schematic diagram showing the configuration of a content display system according to this embodiment. Fig. 18 is a block diagram showing the hardware configuration of a content display device according to this embodiment.

[0068] As shown in Fig. 17 , the content display system 1A according to this embodiment differs from the content display system 1 according to the first embodiment in that it does not include a determination processing device 30 and that it includes a content display device 10A instead of the content display device 10. As shown in Fig. 18 , the content display device 10A differs from the content display device 10 according to the first embodiment in that it does not include a depth sensor 17.

[0069] (Functional Configuration of Content Display Device) A description will now be given of the functional configuration of a content display device according to the second embodiment. Fig. 19 is a block diagram showing the functional configuration of a content display device according to this embodiment.

[0070] 19 , content display device 10A according to this embodiment differs from content display device 10 in that it does not include, as functions, determination information transmission unit 106, determination result reception unit 107, evaluation information transmission unit 108, and evaluation result reception unit 109. Content display device 10A also differs from content display device 10 in that it includes parameter update unit 102A and imaging processing unit 105A instead of parameter update unit 102 and imaging processing unit 105. Content display device 10A also differs from content display device 10 in that it further includes determination processing unit 111 and evaluation processing unit 112.

[0071] Imaging processing unit 105A differs from imaging processing unit 105 in that it acquires only time-series data of captured images captured by camera 16. Parameter update unit 102A differs from parameter update unit 102 in that it uses a Visual SLAM (VSLAM) method to calculate the displacement of the position and orientation of content display device 10A for each frame, based on the displacement between previous and next frames of captured images acquired by imaging processing unit 105A.

[0072] The judgment processing unit 111 corresponds to the judgment processing unit 302 of the judgment processing device 30 according to the first embodiment, and executes judgment processing. The evaluation processing unit 112 corresponds to the evaluation processing unit 305 of the judgment processing device 30, and executes evaluation processing.

[0073] (Operation of content display device) The operation of the content display device according to the second embodiment will be described. Fig. 20 is a flowchart showing the operation of the content display device according to this embodiment. Fig. 21 is a schematic diagram showing a facing determination method for an object to be determined according to this embodiment. Fig. 22 is a schematic diagram showing an evaluation method for an object to be determined according to this embodiment.

[0074] 20, content display device 10A according to this embodiment differs from content display device 10 in that it executes steps S501 to S505 instead of steps S105 to S110. Therefore, only the operations of steps S501 to S505 will be explained below.

[0075] After step S104 is executed, the determination processing unit 111 executes a determination process (S501). This determination process is similar to the determination process executed by the determination processing unit 302 of the determination processing device 30 according to the first embodiment, and therefore a description thereof will be omitted.

[0076] After the determination process is executed, the evaluation processing unit 112 determines whether or not the virtual object JVO corresponding to the determination target object JRO determined to be abnormal has been selected by the user (S502). If the virtual object JVO has not been selected (S502, NO), the evaluation processing unit 112 determines again whether or not the virtual object JVO has been selected (S502).

[0077] On the other hand, if the virtual object JVO is selected (S502, YES), the evaluation processing unit 112 determines whether or not the content display device 10A is facing the determination object JRO (S503). Here, the evaluation processing unit 112 determines that the content display device 10A is facing the determination object JRO if the difference between the width LY and the height LX of the bounding box surrounding the determination object JRO corresponding to the selected virtual object JVO is equal to or less than a predetermined threshold value, as shown in Figure 21. In other words, it determines whether or not the welding point, which is the determination object JRO, is substantially circular in the captured image.

[0078] If the content display device 10A is not facing the object to be judged JRO (S503, NO), the evaluation processing unit 112 judges whether the content display device 10A is facing the object to be judged JRO based on the newly acquired current captured image (S503).

[0079] On the other hand, if the content display device 10A is facing the determination object JRO (S503, YES), the evaluation processing unit 112 executes evaluation processing (S504). This evaluation processing differs from the evaluation processing according to the first embodiment in that, as shown in Fig. 22 , of the three axes in the reference coordinates set for the selected virtual object JVO, the position of a reference point P1 relative to the origin P0 is calculated for only two axes (x and y axes) excluding the depth direction (z axis).

[0080] After the evaluation process is executed, the storage processing unit 110 stores the relative position of the object to be judged JRO with respect to the virtual object JVO calculated by the evaluation process in the storage device 13 in association with the object ID of the virtual object JVO (S505), and the evaluation processing unit 112 again determines whether the virtual object JVO has been selected (S502).

[0081] In this way, according to the content display device 10A of this embodiment, particularly when the content display device 10A has sufficient information processing capabilities, it is possible to judge and evaluate the judgment object JRO without linking with the judgment processing device 30.

[0082] Furthermore, by performing the evaluation process when the determination object JRO is a perfect circle, it is possible to evaluate the deviation of the position of the determination object JRO_2 from the position of the virtual object JVO_2 as shown in FIG. 10 using a content display device 10A that does not have a depth sensor 17. Note that the determination in step S503 may be performed by the determination processing device 30.

[0083] Although embodiments of the invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as set forth in the claims.

[0084] 10, 10A Content display device 101 Parameter calculation unit 102 Parameter update unit 104 Display processing unit 106 Judgment information transmission unit 107 Judgment result reception unit 108 Evaluation information transmission unit 109 Evaluation result reception unit 111 Judgment processing unit 112 Evaluation processing unit 301 Judgment information reception unit 302 Judgment processing unit 303 Judgment result transmission unit 304 Evaluation information reception unit 305 Evaluation processing unit 306 Evaluation result transmission unit

Claims

1. A content display system comprising: a content display device that displays content including at least one virtual object; and a determination processing device, wherein the content display device comprises: a parameter calculation unit that calculates, as initial parameters, an initial position and initial orientation of the content display device based on a captured image of the real object, with the position of the real object as the origin; a parameter update unit that calculates displacement of the position and orientation of the content display device and updates the position and orientation of the content display device based on the initial position and initial orientation; a display processing unit that displays the content arranged in a virtual space with the position of the real object as the origin, superimposed on the real object; a determination information transmission unit that transmits to the determination processing device determination information including the captured image and a display area of ​​the virtual object superimposed on the captured image; and a determination result information receiving unit that receives from the determination processing device determination result information indicating a determination result based on the determination information, wherein the determination processing device comprises: a determination information receiving unit that receives the determination information from the content display device; a judgment processing unit that recognizes a judgment target in the real object based on the received judgment information, and judges whether or not the position of the judgment target is abnormal based on the degree of overlap between the area of ​​the judgment target and the area of ​​a virtual object corresponding to the judgment target; and a judgment result information transmission unit that transmits the judgment result information indicating the judgment result to the content display device.

2. The content display system according to claim 1, further comprising: an evaluation information transmitting unit that transmits evaluation information including the captured image and reference coordinates having an origin at a predetermined position in a virtual object corresponding to the object to be determined to be abnormal to the evaluation processing device; and an evaluation result information receiving unit that receives evaluation result information from the evaluation processing device, indicating an evaluation result based on the evaluation information; and the evaluation processing device further comprising: an evaluation information receiving unit that receives the evaluation information from the content display device; an evaluation processing unit that calculates, based on the received evaluation information, the relative position of the object to be determined to be abnormal with respect to a virtual object corresponding to the object to be determined to be abnormal, which is the object to be determined to be abnormal, as the evaluation result; and an evaluation result transmitting unit that transmits the evaluation result information indicating the calculated evaluation result to the content display device.

3. The content display system according to claim 2, characterized in that the evaluation processing unit calculates a reference point within the area of ​​the object to be determined as abnormal, and calculates the position of the calculated reference point in the reference coordinates as the evaluation result.

4. The content display system according to claim 3, wherein the reference point is the center position of a bounding box that encloses the area of ​​the object to be determined as an abnormality, or the center of gravity of the area of ​​the object to be determined as an abnormality.

5. A content display system as described in any one of claims 2 to 4, characterized in that the object to be judged is a welding point, and the evaluation processing unit calculates the evaluation result based on a captured image in which the difference between the vertical width and horizontal width of a bounding box surrounding the area of ​​the object to be judged as abnormal is less than a predetermined threshold.

6. A content display method executed by a content display device that displays content including at least one or more virtual objects and a determination processing device, wherein the content display device: calculates, based on a captured image of a real object, an initial position and initial orientation of the content display device as initial parameters, with the position of the real object as the origin; calculates a displacement of the position and orientation of the content display device, and updates the position and orientation of the content display device with the initial position and initial orientation as origins; displays the content arranged in a virtual space with the position of the real object as the origin, superimposed on the real object; transmits determination information including the captured image and a display area of ​​the virtual object superimposed on the captured image to the determination processing device; the determination processing device: receives the determination information from the content display device; recognizes a determination target in the real object based on the received determination information, and determines whether the position of the determination target is abnormal based on the degree of overlap between the area of ​​the determination target and the area of ​​the virtual object corresponding to the determination target; and transmits the determination result information indicating the determination result to the content display device; and the content display device: A content display method, comprising receiving, from the determination processing device, determination result information indicating a determination result based on the determination information.

7. A content display device that displays content including at least one virtual object, comprising: a parameter calculation unit that calculates, based on a captured image of a real object, an initial position and initial orientation of the content display device as initial parameters, with the position of the real object as the origin; a parameter update unit that calculates displacement of the position and orientation of the content display device and updates the position and orientation of the content display device with the initial position and initial orientation as the origin; a display processing unit that displays the content arranged in a virtual space with the position of the real object as the origin, superimposed on the real object; and a judgment processing unit that recognizes a judgment target in the real object based on the captured image, and judges whether the position of the judgment target is abnormal based on the degree of overlap between the area of ​​the judgment target and the area of ​​a virtual object corresponding to the judgment target.

Images