Image processing device, image processing method and program

Abstract

To enable a position of an image processing device to be identified even when a person in a mobile body views an internal space and an external space of the mobile body simultaneously.SOLUTION: An image processing device 1 includes: an image data acquisition unit 161 that acquires captured image data generated by imaging an internal space of a mobile body P and an external space of the mobile body P by the image processing device 1 worn by a person riding the mobile body P; a boundary identification unit 162 that identifies a boundary between internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; and a position identification unit 163 that, on the basis of positions of a plurality of feature points of a subject contained in the internal space image data, identifies a position of the image processing device 1 in the internal space and, on the basis of positions of a plurality of feature points of a subject contained in the external space image data, identifies a position of the image processing device 1 in the external space.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program for processing image data. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for displaying an augmented reality image corresponding to a position identified based on image data captured by a camera provided in electronic goggles on a display of the electronic goggles (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-166405 Summary of the Invention [Problem to be solved by the invention]

[0004] In an image processing device such as the electronic goggles described in Patent Document 1, the position of the image processing device can be identified by comparing a pattern of feature points of a subject in captured image data generated in advance by a camera provided in the image processing device with a pattern of feature points previously associated with a position in space. As an example, the image processing device can display an augmented reality image corresponding to the identified position.

[0005] However, when a person inside a moving vehicle uses such an image processing device to simultaneously view the internal space of the moving vehicle and the external space, there is a problem in that even when the pattern of feature points of the subject in the captured image data is compared with the pattern of feature points previously associated with a position in the external space, these patterns do not match, and the position of the image processing device cannot be identified.

[0006] Therefore, the present invention has been made in consideration of these points, and aims to enable a person inside a moving body to identify the position of an image processing device even when the person inside the moving body is looking at the internal space and the external space at the same time. [Means for solving the problem]

[0007] An image processing device of a first aspect of the present invention is an image processing device that can be worn by a person, and includes an image data acquisition unit that acquires captured image data generated by the image processing device worn by a person riding on a moving body by photographing the internal space of the moving body and the external space of the moving body, a boundary identification unit that identifies the boundary between internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data, and a position identification unit that identifies the position of the image processing device in the internal space based on the positions of multiple feature points of the subject included in the internal space image data, and identifies the position of the image processing device in the external space based on the positions of multiple feature points of the subject included in the external space image data.

[0008] The position identification unit may identify the position of the image processing device in the internal space based on the relationship between the positions of the multiple feature points included in the latest internal space image data and the positions of the multiple feature points included in the immediately previous internal space image data, and may identify the position of the image processing device in the external space based on the relationship between the positions of the multiple feature points included in the latest external space image data and the positions of the multiple feature points included in the immediately previous external space image data.

[0009] The position specifying unit may specify the position of the image processing device in the internal space at a cycle slower than a cycle for executing a process for specifying the position of the image processing device in the external space.

[0010] The image processing device may further include an internal image display processing unit that displays an internal augmented reality image corresponding to the position of the image processing device in the internal space identified by the position identification unit on a display unit possessed by the image processing device, and an external image display processing unit that displays an external augmented reality image corresponding to the position of the image processing device in the external space identified by the position identification unit on the display unit.

[0011] The internal image display processing unit may display the internal augmented reality image on the display unit at a position based on a predetermined subject in the internal space in the captured image data.

[0012] The internal image display processing unit may cause the display unit to display the internal augmented reality image of a size determined based on a pattern of a plurality of feature points that indicate features of one or more subjects in the captured image data.

[0013] The external image display processing unit may display the external augmented reality image on the display unit at a position that is stored in advance in a storage unit in association with a position of the image processing device in the external space.

[0014] The image data acquisition unit may acquire the captured image data at predetermined time intervals, and the boundary identification unit may identify the boundary based on the rate of change of the position of each of multiple subjects included in the captured image data acquired at different times.

[0015] An image processing method of a second aspect of the present invention includes the steps of: acquiring captured image data generated by an image processing device worn by a person riding on a moving body by photographing the internal space of the moving body and the external space of the moving body, executed by a processor; identifying internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; identifying the position of the image processing device in the internal space based on the positions of multiple feature points of the subject included in the internal space image data; and identifying the position of the image processing device in the external space based on the positions of multiple feature points of the subject included in the external space image data.

[0016] A third aspect of the program of the present invention causes a processor to execute the steps of acquiring captured image data generated by an image processing device worn by a person riding on a moving body by photographing the internal space of the moving body and the external space of the moving body; identifying internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; identifying the position of the image processing device in the internal space based on the positions of multiple feature points of the subject included in the internal space image data; and identifying the position of the image processing device in the external space based on the positions of multiple feature points of the subject included in the external space image data. [Effects of the Invention]

[0017] According to the present invention, it is possible to determine the position of the image processing device even when a person inside the moving body is looking at the internal space of the moving body and the external space at the same time. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating an overview of an image processing device. [Figure 2]FIG. 10 is a diagram schematically illustrating an example of an image viewed by a user wearing the image processing device. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus. [Figure 5] 10 is an example of point cloud data created by a boundary specifying unit. [Figure 6] 10 is a flowchart showing a processing flow in the image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0019] [Overview of image processing device 1] FIG. 1 is a diagram for explaining an overview of an image processing device 1. The image processing device 1 is a terminal that can be worn by a user U (e.g., a pilot) riding on a moving object P, and has the shape of, for example, glasses or goggles. The moving object P is any object that can move with a person on board, such as an airplane, ship, train, or vehicle. FIG. 1 schematically shows a user U in the cockpit of an airplane operating a control stick H.

[0020] Through the lens of the image processing device 1, the user U can see the space inside the cockpit (hereinafter referred to as the "internal space") and the space outside the window W (hereinafter referred to as the "external space"). The user U can see objects such as the control stick H, meters M1, M2, etc. in the internal space, and can also see the runway in the external space. In addition, while viewing the internal space and the external space, the user U can also see an augmented reality image (hereinafter referred to as an "AR image") displayed on the lens by the image processing device 1.

[0021] Fig. 2 is a diagram schematically illustrating an example of an image viewed by a user U wearing the image processing device 1. In Fig. 2, area R1 indicates a runway in the external space, and area R2 indicates a cockpit in the internal space. Fig. 2(a) shows an AR image displayed superimposed on the internal and external spaces viewed by the user U while sitting upright, and Fig. 2(b) shows an AR image displayed superimposed on the internal and external spaces viewed by the user U while tilted relative to the vertical direction.

[0022] The image shown in Figure 2(a) displays an internal AR image G1 and an external AR image G2. The internal AR image G1 indicates the speed of the airplane and is displayed above the control stick H. The external AR image G2 indicates the direction the airplane should go and is displayed at a predetermined position on the runway.

[0023] Similarly, the image shown in FIG. 2(b) also displays an internal AR image G1' and an external AR image G2'. The internal AR image G1 and the internal AR image G1' are displayed in the same orientation above the control stick H. In contrast, the external AR image G2 and the external AR image G2' are displayed in different orientations. Specifically, in FIG. 2(a), the pilot is not leaning, so the runway appears vertical, and the external AR image G2 is composed of straight lines in the vertical and horizontal directions. In contrast, in FIG. 2(b), the pilot is leaning, so the runway appears diagonal, and the external AR image G2' is composed of multiple straight lines in the diagonal direction.

[0024] The image processing device 1 displays an AR image including information related to the internal space at a position determined based on the position of the image processing device 1 in the internal space, and displays an AR image including information related to the external space at a position determined based on the position of the image processing device 1 in the external space. In order to display the AR image corresponding to the internal space and the AR image corresponding to the external space at appropriate positions, the image processing device 1 identifies the position of the image processing device 1 in the internal space and the position of the image processing device 1 in the external space based on captured image data captured and generated by the image processing device 1.

[0025] Specifically, the image processing device 1 identifies the position of the image processing device 1 in the internal space based on internal space image data corresponding to the internal space in the captured image data, and identifies the position of the image processing device 1 in the external space based on external space image data corresponding to the external space in the captured image data. In this way, the image processing device 1 can identify the position of the image processing device 1 even when a person inside the moving body P simultaneously views the internal space and the external space of the moving body P, and can display an AR image suited to the state of the internal space and an AR image suited to the state of the external space based on the identified position.

[0026] [Configuration of image processing device 1] 3 is a diagram showing the hardware configuration of the image processing device 1. FIG. 4 is a diagram showing the functional configuration of the image processing device 1.

[0027] As shown in Fig. 3, the image processing device 1 includes an imaging unit 11, a right lens 121, a left lens 122, a right projection unit 123, and a left projection unit 124. The right lens 121, the left lens 122, the right projection unit 123, and the left projection unit 124 configure the display unit 12 shown in Fig. 4. The display unit 12 displays an AR image so that the user U can view it while viewing subjects in the internal space and the external space. In the display unit 12, the right projection unit 123 projects a projection image L1 toward the right lens 121, and the left projection unit 124 projects a projection image L2 toward the left lens 122, thereby displaying the AR image on the right lens 121 and the left lens 122.

[0028] The imaging unit 11 has an imaging element that creates captured image data by capturing an image of a subject. In the example shown in Fig. 3, the imaging unit 11 is provided near the center of the image processing device 1, but the location of the imaging unit 11 is arbitrary as long as it can create captured image data corresponding to the area viewed by the user U. The imaging unit 11 has multiple imaging elements provided at different positions in the left-right direction, and may function as a stereo camera that can generate data in the depth direction.

[0029] The right lens 121 corresponds to the right eye of the user U, and is located in front of the right eye of the user U when the user U is wearing the image processing device 1. The right lens 121 is optically transparent, allowing the user U to see surrounding subjects through the right lens 121. The right lens 121 also displays the AR image projected by the right projection unit 123 in a manner that allows the user U to see it. The right lens 121 has, for example, a light guide plate that propagates light projected onto a partial area of ​​the right lens 121 by the right projection unit 123 and then refracts the light toward the right eye of the user U.

[0030] The left lens 122 has a configuration similar to that of the right lens 121. The left lens 122 corresponds to the left eye of the user U and is located in front of the left eye of the user U when the user U is wearing the image processing device 1. The left lens 122 is optically transparent, allowing the user U to see surrounding subjects through the left lens 122. The left lens 122 also displays the AR image projected by the left projection unit 124 in a manner that allows the user U to see it. The left lens 122 has, for example, a light guide plate that propagates light projected onto a partial area of ​​the left lens 122 by the left projection unit 124 and then refracts the light toward the right eye of the user U.

[0031] The right projection unit 123 projects an AR image onto the right lens 121. The right projection unit 123 projects an AR image for the right eye input from the display processing unit 166 onto the right lens 121. The right projection unit 123 and the left projection unit 124 project the AR image onto the right lens 121 and the left lens 122 while the user is viewing a subject through the right lens 121 and the left lens 122.

[0032] The left projection unit 124 projects the AR image onto the left lens 122. The left projection unit 124 projects the AR image for the left eye input from the display processing unit 166 onto the left lens 122.

[0033] Next, the internal configuration and functions of the image processing device 1 will be described with reference to Fig. 4. As shown in Fig. 4, the image processing device 1 has an operation unit 13, a communication unit 14, a storage unit 15, and a control unit 16, in addition to an imaging unit 11 and a display unit 12. The control unit 16 has an image data acquisition unit 161, a boundary identification unit 162, a position identification unit 163, an internal image display processing unit 164, an external image display processing unit 165, and a display processing unit 166.

[0034] The operation unit 13 is a device for receiving an operation of the image processing device 1 by the user U. The operation unit 13 may be, for example, a touch switch or a push switch, or may be a short-range wireless communication device that receives operation data from an external terminal such as a smartphone or a tablet. The operation data is, for example, data indicating the operation content for starting or stopping the display of an AR image, data indicating the model name of the mobile object P, or data indicating the location to which the mobile object P is moving (data indicating the name of an airport, a port, a railway line, or a road). The operation unit 13 notifies the control unit 16 of the operation data indicating the operation content of the user U.

[0035] The communication unit 14 has a communication interface for transmitting and receiving data to and from an external server that provides data used in processing by the control unit 16. The communication unit 14 has, for example, a wireless communication controller for communicating using a wireless communication line. The communication unit 14, for example, acquires, from the external server, reference point cloud data indicating feature points of various objects in the space near where the mobile body P on which the user U is riding is moving, and AR image data to be displayed on the display unit 12, and stores the acquired reference point cloud data and AR image data in the storage unit 15.

[0036] The communication unit 14 acquires a plurality of internal reference point cloud data consisting of feature points of a subject included in images captured at a plurality of positions in the internal space, and also acquires a plurality of external reference point cloud data consisting of feature points of a subject included in images captured at a plurality of positions in the external space.

[0037] The AR image data is associated with one or more internal reference point cloud data or external reference point cloud data. When point cloud data based on feature points of a subject included in an area corresponding to an internal space in the captured image data matches any of the internal reference point cloud data and the AR image data is associated with the internal reference point cloud data, the image processing device 1 displays an AR image based on the AR image data. Similarly, when point cloud data based on feature points of a subject included in an area corresponding to an external space in the captured image data matches any of the external reference point cloud data and the AR image data is associated with the external reference point cloud data, the image processing device 1 displays an AR image based on the AR image data.

[0038] The internal reference point cloud data and the internal AR image data corresponding to the internal reference point cloud data may be associated with the model name of the mobile object P. Furthermore, the external reference point cloud data and the external AR image data corresponding to the external reference point cloud data may be associated with location information indicating latitude, longitude, and altitude, or information indicating an area. As an example, the external reference point cloud data and the external AR image data may be associated with the name of an airport, a port, a railway line, or a road.

[0039] The image processing device 1 may not acquire various data from an external server, but may use reference point cloud data and AR image data stored in advance in the storage unit 15. In this case, the image processing device 1 may not have the communication unit 14.

[0040] The storage unit 15 has storage media such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 15 stores a program executed by the control unit 16. The storage unit 15 also stores reference point cloud data and AR image data received from an external server by the communication unit 14 in association with each other. The storage unit 15 may store multiple pieces of reference point cloud data and AR image data in association with location information or information indicating an area.

[0041] The storage unit 15 stores, as reference point cloud data, a plurality of internal reference point cloud data configured from feature points of a subject included in images captured at a plurality of positions in the internal space. The storage unit 15 may store the internal reference point cloud data in association with a position and orientation in the internal space. A position in the internal space is indicated by a distance and orientation relative to a reference position in the internal space. An orientation in the internal space is indicated by an orientation relative to a reference orientation (e.g., forward) in the internal space.

[0042] The storage unit 15 stores, as reference point cloud data, a plurality of external reference point cloud data configured by feature points of a subject included in images captured at a plurality of positions in external space. The storage unit 15 may store the external reference point cloud data in association with a position and orientation in external space. The position in external space is indicated by a position (coordinates) relative to a reference position. The reference position is, for example, the position of an airport, port, railway line, or road where the image processing device 1 is located. The reference position may be expressed by latitude and longitude, with a position where latitude and longitude are 0 degrees as the reference position, and altitude relative to the position of sea level. The orientation in external space is indicated by an orientation relative to a reference orientation of external space (for example, facing north).

[0043] Furthermore, the storage unit 15 stores internal AR image data in association with each of the multiple internal reference point cloud data. For example, the storage unit 15 stores internal AR image data showing different content in association with each of the multiple internal reference point cloud data, or stores internal AR image data having the same content but different sizes in association with each of the multiple internal reference point cloud data. Furthermore, the storage unit 15 may store internal AR image data in association with the state of the moving object P (for example, speed, altitude, type of problem that has occurred).

[0044] The storage unit 15 also stores external AR image data in association with each of the plurality of external reference point cloud data. For example, the storage unit 15 stores external AR image data showing different content in association with each of the plurality of external reference point cloud data, or stores external AR image data having the same content but different sizes in association with each of the plurality of external reference point cloud data. The storage unit 15 may store external AR image data in association with a position and orientation in external space.

[0045] The storage unit 15 may store the external control point cloud data in association with the name of an airport, a port, a railway line, or a road. For example, the storage unit 15 stores a large number of external control point cloud data created based on captured image data generated by photographing a large number of points in Airport A in a plurality of directions as external control point cloud data for Airport A.

[0046] The storage unit 15 may also store the internal reference point cloud data in association with moving body identification information (e.g., model name) for identifying the moving body P. The storage unit 15 stores, as internal reference point cloud data for model B, a large number of internal reference point cloud data created based on captured image data generated by capturing images at multiple positions in multiple orientations within the cockpit of model B, for example.

[0047] The control unit 16 has a CPU (Central Processing Unit) as a processor that processes various types of data by executing programs stored in the storage unit 15. By executing the programs stored in the storage unit 15, the control unit 16 functions as an image data acquisition unit 161, a boundary identification unit 162, a position identification unit 163, an internal image display processing unit 164, an external image display processing unit 165, and a display processing unit 166.

[0048] The image data acquisition unit 161 acquires captured image data generated by the image processing device 1 worn by the user U riding on the moving body P, capturing an image of the interior space of the moving body P and the exterior space of the moving body P. That is, the image data acquisition unit 161 acquires captured image data generated by the imaging unit 11 while the user U is wearing the image processing device 1. The image data acquisition unit 161 acquires captured image data at predetermined time intervals. The predetermined time interval is arbitrary, but is, for example, 33.3 milliseconds (corresponding to a frame rate of 30 fps). The image data acquisition unit 161 inputs the acquired captured image data to the boundary identification unit 162 and the display processing unit 166.

[0049] The boundary identification unit 162 identifies the boundary between the area corresponding to the internal space and the area corresponding to the external space in the captured image data input from the image data acquisition unit 161. Based on the position of the boundary in the captured image data, the boundary identification unit 162 creates internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data. The boundary is located at the position indicated by the dashed line in the example shown in FIG. 2.

[0050] The method by which the boundary identification unit 162 identifies the boundary is arbitrary, but for example, the boundary may be identified by machine learning (e.g., deep learning), the boundary may be identified by the difference in the rate of change of feature points, or the boundary may be identified by the difference in distance to feature points.

[0051] When identifying a boundary using machine learning, the boundary identification unit 162 inputs the captured image data acquired by the image data acquisition unit 161 into a machine learning model that learns, for example, using captured image data including data indicating the boundary position as training data and outputs boundary position data indicating the boundary position when the captured image data is input. The boundary identification unit 162 identifies a boundary in the captured image data based on the boundary position data output from the machine learning model.

[0052] When identifying a boundary based on the difference in the change speed of feature points, the boundary identifying unit 162 identifies the boundary based on, for example, the change speed of the position of each of a plurality of subjects included in a plurality of pieces of captured image data acquired at different times. Specifically, the boundary identifying unit 162 creates first point cloud data consisting of feature points of a plurality of subjects included in the captured image data generated at a first time, and second point cloud data consisting of feature points of a plurality of subjects included in the captured image data generated at a second time next to the first time. The feature points are points located on the contour line of the subject, on the boundary line of an uneven area, or on the boundary line of an area of ​​different color.

[0053] The boundary identification unit 162 identifies the difference in position between the multiple feature points included in the first point cloud data and the multiple feature points corresponding to these feature points in the second point cloud data. The greater this difference, the greater the rate of change of the feature points. Therefore, the boundary identification unit 162 classifies the multiple feature points into two groups based on the rate of change, and identifies the area containing feature points with a relatively slow rate of change as the internal space, and the area containing feature points with a relatively fast rate of change as the external space.

[0054] Fig. 5 is an example of point cloud data created by boundary identification unit 162. The point cloud data shown in Fig. 5 corresponds to the image shown in Fig. 2. In Fig. 5, feature points corresponding to the internal space are indicated by white squares, and feature points corresponding to the external space are indicated by black circles.

[0055] The feature points of the internal space indicated by white squares have a slower rate of change than the feature points of the external space indicated by black circles, and so the boundary identifying unit 162 can identify the boundary between the internal space containing the feature points indicated by white squares and the external space containing the feature points indicated by black circles, based on the rate of change. For example, the boundary identifying unit 162 identifies, as the boundary, a line connecting multiple feature points that are closest to feature points with a relatively fast rate of change in the up-down and left-right directions, among the feature points with a relatively slow rate of change.

[0056] Similarly, when identifying a boundary based on the difference in distance to a feature point, the boundary identifying unit 162 identifies a feature point corresponding to a distance within a predetermined distance (e.g., 1 m) corresponding to the internal space as a feature point corresponding to the internal space. Furthermore, the boundary identifying unit 162 identifies a feature point corresponding to a distance greater than the predetermined distance as a feature point corresponding to the external space. For example, since the distance to the feature point indicated by the white square in FIG. 5 is shorter than the distance to the feature point indicated by the black circle, it is possible to identify the boundary between the internal space including the feature point indicated by the white square and the external space including the feature point indicated by the black circle.

[0057] The boundary identification unit 162 creates internal space point cloud data corresponding to the internal space based on the internal space image data created based on the identified boundary, and creates external space point cloud data corresponding to the external space based on the external space image data. The boundary identification unit 162 inputs the created internal space point cloud data and external space point cloud data to the position identification unit 163. The boundary identification unit 162 may divide the captured image data into internal space image data and external space image data, and input the internal space image data and external space image data to the position identification unit 163.

[0058] It is to be noted that if the user U does not move frequently within the internal space, it is considered that the position of the boundary will hardly change. Therefore, the boundary identification unit 162 may identify the boundary within a predetermined time period after the moving object P starts moving, and store the position of the identified boundary in the storage unit 15. Thereafter, the boundary identification unit 162 may create internal space point cloud data and external space point cloud data based on the position of the boundary stored in the storage unit 15.

[0059] Alternatively, the boundary specifying unit 162 may specify the boundary using another method. As an example, the boundary specifying unit 162 may specify the boundary by displaying an image based on the captured image data on an information terminal (e.g., a smartphone) used by the user U and acquiring data indicating the boundary that the user U inputs on the information terminal via the operation unit 13.

[0060] The position identifying unit 163 identifies the position of the image processing device 1 in the internal space based on the internal space point cloud data input from the boundary identifying unit 162, and identifies the position of the moving object P in the external space based on the external space point cloud data. That is, the position identifying unit 163 identifies the position of the image processing device 1 in the internal space based on the positions of multiple feature points of the subject included in the internal space image data, and identifies the position of the image processing device 1 in the external space based on the positions of multiple feature points of the subject included in the external space image data. The position identifying unit 163 may further identify the orientation of the image processing device 1 in the internal space and the orientation of the image processing device 1 in the external space.

[0061] As an example, the position specifying unit 163 specifies the internal reference point group data that is closest to the internal space point group data by comparing a plurality of internal reference point group data stored in the storage unit 15 with the internal space point group data input from the boundary specifying unit 162. The position specifying unit 163 specifies the position or orientation stored in the storage unit 15 in association with the internal reference point group data as the position or orientation of the image processing device 1 in the internal space. The position specifying unit 163 notifies the internal image display processing unit 164 of the specified position or orientation of the image processing device 1.

[0062] The position specifying unit 163 may specify the position and orientation of the image processing device 1 based on the relationship between the positions of a plurality of first feature points included in the latest first captured image data and the positions of a plurality of second feature points included in the immediately preceding second captured image data. Specifically, the position specifying unit 163 specifies the amount of change between the positions of the plurality of first feature points and the positions of second feature points corresponding to each of the plurality of first feature points, and specifies, as the latest position and orientation of the image processing device 1, the position and orientation changed from the position and orientation of the image processing device 1 specified immediately before by a distance and orientation corresponding to the specified amount of change.

[0063] In this case, if the feature points used by the position specifying unit 163 to specify the position of the image processing device 1 include a mixture of feature points of the internal space and feature points of the external space, each of which has a different rate of change, the amount of change differs depending on the feature point, and therefore the position specifying unit 163 cannot specify the space in which the image processing device 1 exists. As a result, a problem occurs in that the position specifying unit 163 cannot specify the latest position and orientation of the image processing device 1 based on the position and orientation of the image processing device 1 specified immediately before.

[0064] In response to this, the position specifying unit 163 specifies the position of the image processing device 1 in the internal space based on the relationship (i.e., the amount of change in position) between the positions of a plurality of first feature points included in the latest internal space image data and the positions of a plurality of second feature points included in the immediately preceding internal space image data. Furthermore, the position specifying unit 163 specifies the position of the image processing device 1 in the external space based on the relationship between the positions of a plurality of first feature points included in the latest external space image data and the positions of a plurality of second feature points included in the immediately preceding external space image data. By operating the position specifying unit 163 in this manner, the image processing device 1 can correctly specify the position in the internal space and the position in the external space even when the captured image data includes internal space image data and external space image data.

[0065] The position identification unit 163 may identify the position of the image processing device 1 in the internal space using internal reference point cloud data corresponding to the model name of the moving object P indicated by the operation data input via the operation unit 13. By operating the position identification unit 163 in this manner, the position identification unit 163 does not need to refer to a large number of internal reference point cloud data corresponding to a large number of models, and therefore the time required to identify the position of the image processing device 1 in the internal space can be shortened.

[0066] Similarly, the position specifying unit 163 specifies the external reference point cloud data that is closest to the external space point cloud data by comparing the plurality of external reference point cloud data stored in the storage unit 15 with the external space point cloud data input from the boundary specifying unit 162. The position specifying unit 163 specifies the position or orientation stored in the storage unit 15 in association with the external reference point cloud data as the position or orientation of the image processing device 1 in external space. The position specifying unit 163 notifies the external image display processing unit 165 of the specified position or orientation of the image processing device 1.

[0067] The position identification unit 163 may identify the position of the image processing device 1 in the external space using external reference point cloud data corresponding to the location of the moving object P indicated by the operation data input via the operation unit 13. By operating the position identification unit 163 in this manner, the position identification unit 163 does not need to refer to a large number of external reference point cloud data corresponding to a large number of locations, and therefore the time required to identify the position of the image processing device 1 in the external space can be shortened.

[0068] Incidentally, internal space image data changes less than external space image data. Therefore, the position specifying unit 163 may specify the position of the image processing device 1 in the internal space at a cycle slower than the cycle for executing the process of specifying the position of the image processing device 1 in the external space. By operating the position specifying unit 163 in this manner, the processing load can be reduced, and therefore the time required for the position specifying unit 163 to specify the position of the image processing device 1 in the external space can be shortened.

[0069] The internal image display processing unit 164 refers to the multiple internal AR image data stored in the storage unit 15, and instructs the display processing unit 166 to display, on the display unit 12, an internal AR image that corresponds to the position of the image processing device 1 in the internal space identified by the position identification unit 163. The internal image display processing unit 164 may display an internal AR image that corresponds to the orientation of the image processing device 1 in the internal space. The internal image display processing unit 164 may identify internal reference point cloud data that is closest to the internal point cloud data among the multiple internal reference point cloud data stored in the storage unit 15, and display, on the display unit 12, the internal AR image data that is stored in association with the identified internal reference point cloud data.

[0070] The internal image display processing unit 164 instructs the display processing unit 166 to display an internal AR image based on the internal AR image data at a position on the display unit 12 that is stored in the storage unit 15 in association with the identified internal AR image data. The internal image display processing unit 164 displays the internal AR image at a position determined based on the position of a predetermined subject (e.g., the joystick H) in the internal space in the captured image, such as the internal AR image G1 shown in FIG. 2(a) and the internal AR image G1' shown in FIG. 2(b). If the meter M2 is a speedometer, the internal image display processing unit 164 may display an internal AR image including information about speed at a position determined based on the position of the speedometer.

[0071] At this time, the internal image display processing unit 164 may display the internal AR image in a direction determined based on the direction of the display unit 12, regardless of the tilt of the image processing device 1 (i.e., the tilt of the user U). If the internal AR image includes content that does not need to be linked to the tilt of the subjects of the internal image and the external image, the internal image display processing unit 164 operating in this manner makes it easier for the user U to grasp the content of the internal AR image even when the user U is tilted. Note that if the internal AR image includes information indicating the direction in which the subject in the internal space is operated, the internal image display processing unit 164 may display the internal AR image in a state in which the internal AR image is tilted with respect to the direction of the display unit 12 in conjunction with the tilt of the image processing device 1.

[0072] Furthermore, the internal image display processing unit 164 may display the internal AR image on the condition that the state of the internal space has reached a predetermined state. As an example, when the internal image display processing unit 164 detects a change in the state of an indicator light (e.g., an alarm light) in the cockpit based on the internal space image data, the internal image display processing unit 164 may display an internal AR image including information indicating what action should be taken at a predetermined position relative to the indicator light.

[0073] Incidentally, when the user U leans back, objects in the internal space (for example, the control stick H and meters M1 in the cockpit) appear smaller. Therefore, the internal image display processing unit 164 may change the size of the internal AR image in accordance with the size of these objects. Specifically, the internal image display processing unit 164 may cause the display unit 12 to display an internal augmented reality image of a size determined based on a pattern of multiple feature points that indicate the characteristics of one or more objects in the captured image data.

[0074] More specifically, the internal image display processing unit 164 identifies internal AR image data of a size stored in the storage unit 15 in association with the internal reference point cloud data that is closest to the internal space point cloud data created by the position identification unit 163, and displays the identified internal AR image on the display unit 12. By operating the internal image display processing unit 164 in this manner, the size of the internal AR image changes depending on the posture of the user U, and therefore the internal AR image appears as if it exists in the internal space, enhancing the sense of realism.

[0075] The external image display processing unit 165 refers to the multiple external AR image data stored in the storage unit 15 and instructs the display processing unit 166 to display, on the display unit 12, an external AR image corresponding to the position of the image processing device 1 in the external space identified by the position identification unit 163. The external image display processing unit 165 displays the external AR image on the display unit 12 at a position that is stored in advance in the storage unit 15 in association with the position of the image processing device 1 in the external space. The external image display processing unit 165 may also display an internal AR image that corresponds to the orientation of the image processing device 1 in the external space. The external image display processing unit 165 may identify external AR image data stored in the storage unit 15 in association with external reference point cloud data that is closest to the external space point cloud data created by the position identification unit 163, and display, on the display unit 12, an external AR image based on the identified external AR image data.

[0076] The display processing unit 166 displays the internal AR image data input from the internal image display processing unit 164 and the external AR image data input from the external image display processing unit 165 on the display unit 12. The display processing unit 166 may acquire captured image data from the image data acquisition unit 161, and display on the display unit 12 an image based on image data obtained by combining the acquired captured image data with the internal AR image data and the external AR image data.

[0077] [Processing flow in image processing device 1] Fig. 6 is a flowchart showing the flow of processing in the image processing device 1. The flowchart shown in Fig. 6 starts when the image processing device 1 is powered on.

[0078] The image data acquisition unit 161 acquires captured image data from the imaging unit 11 at predetermined time intervals (S11). The boundary identification unit 162 creates point cloud data by extracting feature points of the subject included in the acquired captured image data, and stores the created point cloud data in the storage unit 15 in association with time (S12).

[0079] The boundary identification unit 162 identifies the speed of change of the position of each feature point by comparing the created point cloud data with point cloud data corresponding to the immediately preceding captured image data (S13). The boundary identification unit 162 classifies the identified speed of change to identify the boundary between the area corresponding to the internal space and the area corresponding to the external space in the captured image data (S14). The boundary identification unit 162 creates internal space point cloud data corresponding to the internal space and external space point cloud data corresponding to the external space based on the identified boundary (S15). The boundary identification unit 162 stores the created internal space point cloud data and external space point cloud data in the storage unit 15 in association with time.

[0080] Next, the position specifying unit 163 specifies the internal reference point cloud data that is most similar to the internal space point cloud data created by the boundary specifying unit 162 from the multiple internal reference point cloud data stored in the storage unit 15, and specifies the position stored in the storage unit 15 in association with the specified internal reference point cloud data as a position in the internal space of the image processing device 1 (S16). The internal image display processing unit 164 displays, on the display unit 12, an internal AR image that corresponds to the position in the internal space specified by the position specifying unit 163 (S17).

[0081] Similarly, the position specifying unit 163 specifies external reference point cloud data that is most similar to the external space point cloud data created by the boundary specifying unit 162 from among the multiple external reference point cloud data stored in the storage unit 15, and specifies a position stored in the storage unit 15 in association with the specified external reference point cloud data as a position in the external space of the image processing device 1 (S18). The external image display processing unit 165 displays, on the display unit 12, an external AR image corresponding to the position in the external space specified by the position specifying unit 163 (S19). Note that the order of executing S16 and S17 and S18 and S19 is arbitrary.

[0082] The image processing device 1 repeats the processes from S11 to S19 until an operation to stop displaying the AR image is performed on the operation unit 13 (NO in S20).

[0083] [Effects of image processing device 1] As described above, the position specifying unit 163 specifies the position of the image processing device 1 in the internal space based on the positions of multiple feature points of the subject included in the internal space image data, and specifies the position of the image processing device in the external space based on the positions of multiple feature points of the subject included in the external space image data. By configuring the image processing device 1 in this way, it is possible to correctly specify the position of the image processing device 1 even when a person riding on the moving body P looks at the internal space and the external space of the moving body P at the same time.

[0084] Furthermore, by the position specifying unit 163 specifying the position of the image processing device 1 in the internal space, the internal image display processing unit 164 can display, on the display unit 12, an internal AR image corresponding to the position of the image processing device 1 in the internal space at a predetermined position in the internal space or external space viewed by the user. Furthermore, the external image display processing unit 165 can display, on the display unit 12, an external AR image corresponding to the position of the image processing device 1 in the external space at a predetermined position in the internal space or external space viewed by the user U.

[0085] With the internal image display processing unit 164 and the external image display processing unit 165 configured in this manner, when the user U is viewing an internal space and an external space that change at different speeds, it is possible to present to the user U internal AR images and external AR images that are appropriate for the state of the internal space and the state of the external space, respectively. Therefore, the image processing device 1 is suitable for applications in which AR images need to be displayed while the user U is wearing and using the device while riding on a moving object P.

[0086] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0087] 1. Image processing device 11 Imaging unit 12 Display section 13 Control section 14 Communications Department 15 Storage section 16 Control Unit 121 Right lens 122 left lens 123 Right side projection section 124 Left side projection section 161 Image data acquisition unit 162 Boundary identification part 163 Location identification part 164 Internal image display processing unit 165 External image display processing unit 166 Display processing section

Claims

1. An image processing device in the form of glasses that can be worn by a person, an image data acquisition unit that acquires captured image data generated by the image processing device worn by a person riding on the moving body by capturing an image of the interior space of the moving body and the exterior space of the moving body; a boundary specifying unit that specifies a boundary between internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; a position specifying unit that specifies a position of the image processing device in the internal space based on positions of a plurality of feature points of a subject included in the internal space image data, and that specifies a position of the image processing device in the external space based on positions of a plurality of feature points of a subject included in the external space image data; an internal image display processing unit that displays, on a display unit of the image processing device, an internal augmented reality image corresponding to the position of the image processing device in the internal space identified by the position identifying unit; and When the internal augmented reality image includes content that does not need to be linked to the tilt of the subject within the internal space, the internal image display processing unit displays the internal augmented reality image in a direction determined based on the direction of the display unit, regardless of the tilt of the image processing device; when the internal augmented reality image includes information indicating the direction in which the subject within the internal space is to be operated, the internal image display processing unit displays the internal augmented reality image in a state in which the internal augmented reality image is tilted relative to the direction of the display unit in conjunction with the tilt of the image processing device. Image processing device.

2. the position identification unit identifies a position of the image processing device in the internal space based on a relationship between positions of the plurality of feature points included in the latest internal space image data and positions of the plurality of feature points included in the immediately preceding internal space image data, and identifies a position of the image processing device in the external space based on a relationship between positions of the plurality of feature points included in the latest external space image data and positions of the plurality of feature points included in the immediately preceding external space image data; The image processing device according to claim 1 .

3. the position specifying unit specifies the position of the image processing device in the internal space at a cycle slower than a cycle for executing a process for specifying the position of the image processing device in the external space; 3. The image processing device according to claim 1 or 2.

4. The present invention further comprises an external image display processing unit that displays on the display unit an external augmented reality image corresponding to the position of the image processing device in the external space identified by the position identification unit. The image processing device according to claim 1 .

5. the internal image display processing unit displays the internal augmented reality image on the display unit, the internal augmented reality image having a size determined based on a pattern of a plurality of feature points that indicate features of one or more objects in the captured image data. The image processing device according to claim 1 .

6. the external image display processing unit displays the external augmented reality image on the display unit at a position that is stored in advance in a storage unit in association with a position of the image processing device in the external space; The image processing device according to claim 4 .

7. the image data acquisition unit acquires the captured image data at predetermined time intervals; the boundary specifying unit specifies the boundary based on a speed of change in the position of each of a plurality of subjects included in the plurality of pieces of captured image data acquired at different times; The image processing device according to claim 1 .

8. The processor executes acquiring captured image data generated by an eyeglass-shaped image processing device worn by a person riding on a moving body by capturing an image of an interior space of the moving body and an exterior space of the moving body; Identifying internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; Identifying a position of the image processing device in the internal space based on positions of a plurality of feature points of a subject included in the internal space image data; specifying a position of the image processing device in the external space based on positions of a plurality of feature points of a subject included in the external space image data; displaying an internal augmented reality image corresponding to the specified position of the image processing device in the internal space on a display unit of the image processing device; and In the displaying step, if the internal augmented reality image contains content that does not need to be linked to the tilt of the subject within the internal space, the internal augmented reality image is displayed in a direction determined based on the direction of the display unit, regardless of the tilt of the image processing device, and if the internal augmented reality image contains information indicating the direction in which the subject within the internal space is to be operated, the internal augmented reality image is displayed in a state in which it is tilted relative to the direction of the display unit in accordance with the tilt of the image processing device.

9. The processor acquiring captured image data generated by an eyeglass-shaped image processing device worn by a person riding on a moving body by capturing an image of an interior space of the moving body and an exterior space of the moving body; Identifying internal space image data corresponding to the internal space in the captured image data and external space image data corresponding to the external space in the captured image data; Identifying a position of the image processing device in the internal space based on positions of a plurality of feature points of a subject included in the internal space image data; specifying a position of the image processing device in the external space based on positions of a plurality of feature points of a subject included in the external space image data; displaying an internal augmented reality image corresponding to the specified position of the image processing device in the internal space on a display unit of the image processing device; Execute In the displaying step, if the internal augmented reality image contains content that does not need to be linked to the tilt of the subject within the internal space, the internal augmented reality image is displayed in a direction determined based on the direction of the display unit, regardless of the tilt of the image processing device, and if the internal augmented reality image contains information indicating the direction in which the subject within the internal space is to be operated, the internal augmented reality image is displayed in a state in which the internal augmented reality image is tilted relative to the direction of the display unit in conjunction with the tilt of the image processing device.

Images