Information providing method, program, and information providing device

Abstract

To provide an information provision method, program, and information provision device, which allow for nicely providing information.SOLUTION: An information provision method implemented by an information provision system, comprising terminal devices (user terminal, mobile terminal, information provision device) and a server device, comprises: an image acquisition step S1 of acquiring a captured image; a determination step S3 of determining the presence of out-of-imaging-range parts, or parts located outside an imaging range of an image capturing unit of a terminal device among a plurality of parts of a vehicle; and a display step S7, S8 of displaying the captured image on a display screen of the terminal device, together with parts-related information if at least one of the plurality of parts is included in the captured image. In the display step, an indication showing the presence of out-of-imaging-range parts is also displayed if the out-of-imaging-range parts are determined to be present in the determination step.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to an information providing method, a program, and an information providing device. [Background technology]

[0002] Patent Document 1 discloses a vehicle information providing system that provides vehicle information. In Patent Document 1, a photograph of an inquired location of the vehicle's devices and equipment is taken by a mobile terminal, and the photograph is transmitted from the mobile terminal to a processing server. The processing server is equipped with a recognition server that recognizes the inquired location indicated by the photograph transmitted from the mobile terminal, and an FAQ database that has a plurality of explanatory information that provides explanations about the devices and equipment. The processing server selects explanatory information corresponding to the inquired location recognized by the recognition server from the FAQ database and transmits it. The explanatory information transmitted from the processing server is displayed by a display device provided in the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-36811 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 merely discloses displaying explanatory information corresponding to the inquired portion shown in the photograph by a display device.

[0005] An object of the present invention is to provide an information providing method, a program, and an information providing device that are capable of providing information in an excellent manner. [Means for solving the problem]

[0006] An information providing method according to one embodiment of the present invention includes an image acquisition step of acquiring an image obtained by imaging at least a part of a vehicle with an imaging unit provided in a terminal device, a determination step of determining the presence of an outside-imaging-range part, which is a part among a plurality of parts provided on the vehicle that is located outside the imaging range of the imaging unit, and a display step of displaying the image on a display screen provided in the terminal device and, if at least one of the plurality of parts is included in the image, also displaying information regarding the part, wherein, in the display step, if the presence of the outside-imaging-range part is determined in the determination step, an indication of the presence of the outside-imaging-range part is also displayed.

[0007] A program according to another aspect of the present invention causes a computer to execute the information providing method as described above.

[0008] An information providing device according to yet another aspect of the present invention comprises an imaging unit, a display unit having a display screen, an image acquisition unit that acquires an imaged image obtained by imaging at least a part of a vehicle with the imaging unit, a determination unit that determines the presence of an outside-imaging-range part which is one of a plurality of parts provided on the vehicle and is located outside the imaging range of the imaging unit, and a display control unit that displays the imaged image on the display screen and, if at least one of the plurality of parts is included in the imaged image, also displays information regarding the part, and when the presence of the outside-imaging-range part is determined by the determination unit, the display control unit displays the imaged image on the display screen and also displays an indication of the presence of the outside-imaging-range part. Effect of the Invention

[0009] According to the present invention, it is possible to provide an information providing method, a program, and an information providing device that can provide information satisfactorily. [Brief description of the drawings]

[0010] [Figure 1]1 is a block diagram showing an information providing system according to an embodiment; [Diagram 2] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. [Diagram 3] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. [Figure 4] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. [Diagram 5] 5A to 5C are diagrams showing examples of images displayed on the display screen of the display unit. [Figure 6] 6A and 6B are diagrams showing examples of images displayed on the display screen of the display unit. [Figure 7] 7A and 7B are diagrams showing examples of images displayed on the display screen of the display unit. [Figure 8] 8A and 8B are diagrams showing examples of images displayed on the display screen of the display unit. [Figure 9] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. [Figure 10] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. [Figure 11] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. [Figure 12] 1 is a flow chart illustrating an example of a method for providing information according to one embodiment. [Figure 13] 5A and 5B are diagrams illustrating examples of images displayed on a display screen of a display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information providing method, a program, and an information providing device according to the present invention will be described in detail below with reference to preferred embodiments and the accompanying drawings.

[0012] [One embodiment] An information providing method, a program, and an information providing device according to an embodiment will be described with reference to Figures 1 to 12. Figure 1 is a block diagram showing an information providing system according to the embodiment.

[0013] The information providing system 10 according to the present embodiment can be configured by a terminal device (user terminal, mobile terminal, information providing device) 12 and a server device (server) 14, but is not limited to this.

[0014] The terminal device 12 includes an imaging unit 16, a sensor group 18, a calculation unit 20, a storage unit 22, a communication unit 24, an operation unit 26, and a display unit 28. The terminal device 12 may include components other than these components, but the description of these components will be omitted here. The server device 14 includes a calculation unit 48, a storage unit 50, and a communication unit 52. The server device 14 may include components other than these components, but the description of these components will be omitted here.

[0015] An application program for causing a computer to execute the information providing method according to the present embodiment is pre-installed in the terminal device 12. The application program can be started based on, for example, an operation by a user.

[0016] The imaging unit 16, i.e., a camera, may be provided at least on the back side of the terminal device 12. A captured image 110 (see FIG. 2), which is an image obtained by imaging using the imaging unit 16, may be displayed on the front side of the terminal device 12, i.e., on a display screen 28A (see FIG. 2) of a display unit 28 provided on the front side. The imaging unit 16 may acquire the captured image 110 by imaging at least a part of the vehicle 11 (see FIG. 2).

[0017] The sensor group 18 may include, for example, an acceleration sensor 30 and a gyro sensor 32. The acceleration sensor 30 may measure, for example, acceleration in three directions simultaneously. The gyro sensor 32, i.e., an angular velocity sensor, may measure, for example, angular velocity in three directions simultaneously. Based on information acquired by the acceleration sensor 30 and the gyro sensor 32, the three-dimensional behavior of the terminal device 12 may be grasped.

[0018] The calculation unit 20 may be configured with, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), that is, a processing circuit. The calculation unit 20 may include a control unit 34, an image acquisition unit 36, a calibration unit 38, a position and orientation determination unit 40, a determination unit 42, a display control unit 44, and an information acquisition unit 46. The control unit 34, the image acquisition unit 36, the calibration unit 38, the position and orientation determination unit 40, the determination unit 42, the display control unit 44, and the information acquisition unit 46 may be realized by the calculation unit 20 executing a program stored in the storage unit 22.

[0019] At least a part of the control unit 34, the image acquisition unit 36, the calibration unit 38, the position and orientation determination unit 40, the determination unit 42, the display control unit 44, and the information acquisition unit 46 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Also, at least a part of the control unit 34, the image acquisition unit 36, the calibration unit 38, the position and orientation determination unit 40, the determination unit 42, the display control unit 44, and the information acquisition unit 46 may be configured by an electronic circuit including a discrete device.

[0020] The storage unit 22 may include a volatile memory (not shown) and a non-volatile memory (not shown). The volatile memory may include, for example, a random access memory (RAM). The non-volatile memory may include, for example, a read only memory (ROM) and a flash memory. Data and the like may be stored in, for example, the volatile memory. Programs, tables, maps, and the like may be stored in, for example, the non-volatile memory. At least a part of the storage unit 22 may be included in a processor, an integrated circuit, and the like as described above.

[0021] The communication unit 24 includes, for example, a communication module (not shown) that is compatible with a mobile phone network. The communication unit 24 can communicate via the mobile phone network. The communication unit 24 can communicate with the communication unit 52 included in the server device 14 via a relay station 54 and a network 56.

[0022] The operation unit 26 can be used when a user operates the terminal device 12. The display unit 28 is provided with a display element (not shown). As the display element, for example, a liquid crystal display element, an organic electroluminescence display element, or the like can be used. The operation unit 26 and the display unit 28 can be configured by a touch panel (not shown) provided with such a display element.

[0023] The control unit 34 is responsible for the overall control of the terminal device 12 .

[0024] The image acquisition section 36 can acquire a captured image 110 obtained by capturing an image using the imaging section 16 .

[0025] The calibration unit 38 may execute a calibration process. Prior to starting to provide information to the user, calibration may be performed. This calibration may be performed, for example, by capturing an image of a home button (not shown) provided in the vehicle interior 100 (see FIG. 2) using the imaging unit 16 of the terminal device 12 while the application program according to this embodiment is running. More specifically, the user faces the imaging unit 16 of the terminal device 12 directly to the home button so that the home button is displayed in a predetermined size at the center of the display screen 28A of the display unit 28. The calibration unit 38 may determine the position and attitude of the terminal device 12 based on the captured image 110 including the home button and the signal supplied from the sensor group 18. When it is determined that the position and attitude of the terminal device 12 are within a predetermined range, the calibration unit 38 determines that the calibration has been successful and completes the calibration. When the calibration is completed, the display control unit 44 displays information indicating that the calibration has been completed on the display screen 28A of the display unit 28. An example of the information indicating that the calibration is complete is a check mark (not shown).

[0026] The position and orientation determination unit 40 can determine the position and orientation of the terminal device 12. The position and orientation determination unit 40 stores information related to the position and orientation of the terminal device 12, i.e., the position and orientation information, in the storage unit 22. The position and orientation determination unit 40 updates the position and orientation information as needed based on the captured image 110 acquired by the image acquisition unit 36 ​​and the signals supplied from the sensor group 18, while using the position and orientation information acquired during calibration as a reference.

[0027] Fig. 2 is a diagram showing an example of a display on the display screen of the display unit. The display control unit 44 can display a captured image 110 obtained by imaging using the imaging unit 16, i.e., a captured image 110 acquired by the image acquisition unit 36, on the display screen 28A of the display unit 28. Fig. 2 shows an example of a captured image 110 when the interior of the vehicle 11, i.e., the interior of the vehicle (car compartment) 100, is captured by the imaging unit 16, but is not limited to this. The outside of the car compartment may also be captured by the imaging unit 16.

[0028] As shown in FIG. 2, the vehicle 11 may be equipped with a plurality of devices 102A to 102O. When describing the devices in general, the reference numeral 102 is used, and when describing each device, the reference numerals 102A to 102O are used. The device 102A is, for example, an infotainment multi-display device. The device 102B is, for example, a driving system monitor. The devices 102C and 102D are, for example, driving system rear monitors. The device 102E is, for example, a steering wheel. The device 102F is, for example, a content multi-switch. The device 102G is, for example, a driving system switch. The device 102H is, for example, an infotainment switch. The device 102I is, for example, a simple parking switch. The device 102J is, for example, a hazard switch. The device 102K is, for example, an air conditioner, that is, an air conditioning device. The device 102L is, for example, a console driving system switch. The device 102M is, for example, an ETC vehicle-mounted device. The device 102N is, for example, a side camera mirror system adjustment switch, and the device 102O is, for example, a door knob.

[0029] The display control unit 44 controls the display of the display unit 28. The display control unit 44 displays the captured image 110 acquired by the image acquisition unit 36 ​​on the display screen 28A of the display unit 28, and when at least one of the multiple devices 102 is included in the captured image 110, the graphics 31A to 31O corresponding to the devices 102A to 102O may be superimposed on the devices 102A to 102O. That is, in this embodiment, an augmented reality (AR) display may be performed. Note that, when describing graphics in general, the reference numeral 31 is used, and when describing each of the graphics, the reference numerals 31A to 31O are used. The graphics 31 corresponding to the device 102 are, for example, semi-transparent graphics, but are not limited thereto. The graphics 31 may be given a color indicating the type to which the device 102 belongs. Since the graphics 31 with a color corresponding to the type are superimposed on the device 102, the function of the device 102 may be understood based on the color.

[0030] FIG. 3 is a diagram showing an example of an image displayed on the display screen of the display unit. FIG. 3 shows an example of an image displayed when the terminal device 12 is brought close to the device 102. As shown in FIG. 3, the device 102 provided in the vehicle 11 may be provided with components (constituent elements) 102a to 102c that constitute a part of the device 102. Examples of the components 102a to 102c include operation members such as an operation switch, an operation button, and an operation lever. In the example shown in FIG. 3, the component 102a is a selector wheel. In the example shown in FIG. 3, the component 102b is a home switch. In the example shown in FIG. 3, the component 102c is a speech switch. Note that the components 102a to 102c are not limited to these.

[0031] There is no doubt that the devices 102A-102O provided in the vehicle 11 and the components 102a-102c that constitute the devices 102A-102O are parts provided in the vehicle 11. Therefore, the devices 102A-102O provided in the vehicle 11 and the components 102a-102c that constitute the devices 102 may be referred to as parts 102.

[0032] The determination unit 42 may determine whether or not the target of the augmented reality display is included in the captured image 110 acquired by the image acquisition unit 36. For example, the target of the augmented reality display may be a part 102 provided in the vehicle 11. The part 102 provided in the vehicle 11 may be a part 102 provided in the interior 100 of the vehicle, or may be a part 102 provided outside the vehicle cabin. The determination of whether or not the target of the augmented reality display is included in the captured image 110 may be performed based on the position and orientation information obtained by the position and orientation determination unit 40 and the captured image 110 acquired by the image acquisition unit 36. The determination of whether or not the target of the augmented reality display is included in the captured image 110 is performed based on not only the captured image 110 acquired by the image acquisition unit 36 ​​but also the position and orientation information in order to perform a more accurate determination. Information indicating the part 102 provided in the vehicle 11 is stored in advance in the storage unit 22. More specifically, position information of each of the multiple parts 102 provided on the vehicle 11, image information of each of these parts 102, etc. are stored in advance in the storage unit 22. The determination unit 42 compares the information stored in the storage unit 22 with information obtained from the captured image 110 acquired by the image acquisition unit 36. In this way, the determination unit 42 can determine whether or not the captured image 110 includes the part 102 provided on the vehicle 11. The determination unit 42 can also determine which part 102 of the multiple parts 102 provided on the vehicle 11 the part 102 included in the captured image 110 is. In this way, the determination unit 42 can determine whether or not the target of the augmented reality display is included in the captured image 110 acquired by the image acquisition unit 36.

[0033] The display control unit 44 displays the captured image 110 on the display screen 28A of the display unit 28, and when at least one of the multiple parts 102 provided in the vehicle 11 is included in the captured image 110, the display control unit 44 can perform the following process. That is, the display control unit 44 displays the captured image 110 on the display screen 28A of the display unit 28, and when the captured image 110 includes the part 102 to be displayed in augmented reality, the display control unit 44 can also display information 64a to 64c related to the part 102, as shown in FIG. 3. That is, in this embodiment, the information 64a to 64c can be displayed in augmented reality. Note that when describing information in general, the reference symbol 64 is used, and when describing each piece of information, the reference symbols 64a to 64c are used.

[0034] There may be cases where the part 102 to be the subject of the augmented reality display is located outside the imaging range of the imaging unit 16. The part 102 located outside the imaging range of the imaging unit 16 may be referred to as an outside imaging range part. As described below, the display control unit 44 may also display indications 103A to 103C (see FIG. 4) indicating the presence of the outside imaging range part 101 (see FIG. 4). When describing the indications in general, the reference numeral 103 is used, and when describing each of the indications, the reference numerals 103A to 103C are used. The indication 103 may be displayed, for example, semi-transparently, but is not limited to this.

[0035] The determination unit 42 can determine the presence of an outside-imaging-range part 101, which is a part 102 located outside the imaging range of the imaging unit 16. As described above, position information of each of the multiple parts 102 provided on the vehicle 11, image information of each of these parts 102, and the like are stored in advance in the storage unit 22. In addition, as described above, the determination unit 42 can also determine which part 102 of the multiple parts 102 provided on the vehicle 11 the part 102 included in the captured image 110 is. The determination unit 42 can determine the presence of the outside-imaging-range part 101 based on the position information of each of the multiple parts 102 provided on the vehicle 11, information indicating the part 102 included in the captured image 110, and position and orientation information indicating the position and orientation of the terminal device 12.

[0036] However, if the indications 103 are displayed for all the components 102 located outside the imaging range of the imaging unit 16, the display of the indications 103 becomes very cumbersome. Therefore, it may be preferable not to display the indications 103 for the components 102 that are significantly outside the imaging range of the imaging unit 16. Thus, the determination unit 42 may determine, among the components 102 located outside the imaging range of the imaging unit 16, the components 102 that are not significantly outside the imaging range of the imaging unit 16 as the components 101 outside the imaging range. That is, the determination unit 42 may determine, among the components 102 located within a predetermined range, i.e., a specified range, the components 102 that are outside the imaging range as the components 101 outside the imaging range. The specified range is a range that is wider than the imaging range of the imaging unit 16 by a predetermined amount.

[0037] When the determination unit 42 determines that the devices 102A to 102O to be the target of the augmented reality display are included in the captured image 110, the information acquisition unit 46 may acquire information about the devices 102. The information acquisition unit 46 may acquire not only information about the devices 102A to 102O provided in the vehicle 11, but also information about the components 102a to 102c that configure the devices 102A to 102O. In this way, when the determination unit 42 determines that the part 102 to be the target of the augmented reality display is included in the captured image 110, the information acquisition unit 46 may acquire information about the part 102. For example, the information about the part 102 may include the name of the part 102, but is not limited thereto. The information about the part 102 may be information about an operation method of the part 102, etc. The information about the part 102 may be supplied from the server device 14, for example. The information acquisition unit 46 transmits a request signal requesting the provision of information about the part 102 to the server device 14 via the communication unit 24. The server device 14 reads out information about the part 102 from the storage unit 50 included in the server device 14, and transmits the information about the part 102 to the terminal device 12 via the communication unit 52 included in the server device 14. The information acquisition unit 46 receives the information about the part 102 from the server device 14. Note that if the information about the part 102 has already been stored in the storage unit 22 included in the terminal device 12, it is not necessary to acquire the information about the part 102 from the server device 14. In such a case, the information acquisition unit 46 can acquire the information about the part 102 from the storage unit 22 included in the terminal device 12.

[0038] FIG. 4 is a diagram showing an example of an image displayed on the display screen of the display unit. FIG. 4 shows an example of a display when the imaging range of the imaging unit 16 is changed. The imaging range of the imaging unit 16 in FIG. 4 is located to the right of the imaging range of the imaging unit 16 in FIG. 3. In the example shown in FIG. 4, the components 102a to 102c, that is, the parts 102a to 102c, are located outside the imaging range of the imaging unit 16. That is, in the example shown in FIG. 4, an outside imaging range part 101 exists. In such a case, the display control unit 44 may display indications 103A to 103C indicating the presence of the outside imaging range part 101 on the display screen 28A of the display unit 28, as shown in FIG. 4. The indication 103A indicates the presence of the part 102a, which is the outside imaging range part 101. The indication 103B indicates the presence of the part 102b, which is the outside imaging range part 101. The indication 103C indicates the presence of the part 102c, which is the outside imaging range part 101. The indication 103 may be displayed on, for example, an edge of the display screen 28A of the display unit 28. The indication 103 may indicate the direction of the part 101 outside the imaging range. In the example shown in Fig. 4, the direction of the smallest angle of the triangle constituting the indication 103 corresponds to the direction of the part 101 outside the imaging range. Since such an indication 103 is displayed on the display screen 28A of the display unit 28, the user can recognize the presence of the part 101 outside the imaging range based on the indication 103.

[0039] The display control unit 44 may further display a plurality of operation buttons 60A to 60C on a part of the display screen 28A. The operation button 60A is, for example, a useful information button. The operation button 60B is, for example, a warning light list button. The operation button 60C is, for example, a setting button. Note that the operation buttons 60A to 60C are not limited to these. The operation button 60A, i.e., the useful information button, is a button for transitioning to a screen on which information that is expected to be useful to the user is displayed. The operation button 60B, i.e., the warning light list button, is a button for transitioning to a screen on which a list of warning lights is displayed. The operation button 60C, i.e., the setting button, is a button for transitioning to a screen for performing various settings related to the application program. The display screen 28A may further display a recalibration button 60D, a voice search button 60E, etc., but is not limited thereto. The recalibration button 60D is a button for instructing the execution of recalibration. The voice search button 60E is a button for instructing execution of a voice search.

[0040] 5A to 5C are diagrams showing examples of images displayed on the display screen of the display unit. FIG. 5A shows an example in which parts 102P to 102R provided on the vehicle 11 are shown in the captured image 110. To simplify the drawing, FIG. 5A shows the parts 102P to 102R abstractly. As shown in FIG. 5A, the display control unit 44 may superimpose and display information 64P to 64R related to the parts 102P to 102R on the captured image 110. The part 102P belongs to a first type. Therefore, the information 64P related to the part 102P may be displayed in, for example, a first color. The first color may be, for example, red, but is not limited to this. The parts 102Q and 102R belong to, for example, a second type different from the first type. Therefore, the information 64Q and 64R related to the parts 102Q and 102R may be displayed in, for example, a second color different from the first color. The second color may be, for example, but is not limited to, green.

[0041] FIG. 5B shows a state in which the imaging range of the imaging unit 16 is changed. Specifically, FIG. 5B shows a state in which the imaging range is shifted to the right. FIG. 5C shows a state in which the imaging range of the imaging unit 16 is further changed. In the example shown in FIG. 5C, indications 103P to 103R indicating the presence of a part 101 outside the imaging range are displayed on the display screen 28A of the display unit 28. The part 101 outside the imaging range corresponding to the indication 103P is part 102P. The part 102P belongs to, for example, a first type. Since the part 102P belongs to the first type, the indication 103P may be displayed in, for example, a first color. The part 101 outside the imaging range corresponding to the indication 103Q is part 102Q. The part 102Q belongs to, for example, a second type different from the first type. Since the component 102Q belongs to the second type, the indication 103Q may be displayed, for example, in a second color different from the first color. The component outside the imaging range 101 corresponding to the indication 103R is the component 102R. The component 102R belongs, for example, to the second type. Since the component 102R belongs to the second type, the indication 103R may be displayed, for example, in the second color. In this manner, the indication 103 may be displayed in a color according to the type of the component 102.

[0042] 6A and 6B are diagrams showing examples of images displayed on the display screen of the display unit. FIG. 6A shows an example in which the number of parts 101 outside the imaging range is relatively small. FIG. 6B shows an example in which the number of parts 101 outside the imaging range is relatively large. The indication 103 may be displayed in a size according to the number of parts 101 outside the imaging range. In the example shown in FIG. 6A, the number of parts 101 outside the imaging range is relatively small, so the size of the indication 103 may be set to be relatively small. On the other hand, in the example shown in FIG. 6B, the number of parts 101 outside the imaging range is relatively large, so the size of the indication 103 may be set to be relatively large. In this way, the indication 103 may be displayed in a size according to the number of parts 101 outside the imaging range.

[0043] 7A and 7B are diagrams showing examples of images displayed on the display screen of the display unit. FIG. 7A shows an example in which the distance to the part 101 outside the imaging range is relatively small. FIG. 7B shows an example in which the distance to the part 101 outside the imaging range is relatively large. The indication 103 may be displayed with a length according to the distance to the part 101 outside the imaging range. In the example shown in FIG. 7A, the distance to the part 101 outside the imaging range is relatively small, so the length of the indication 103 may be set to be relatively short. On the other hand, in the example shown in FIG. 7B, the distance to the part 101 outside the imaging range is relatively large, so the length of the indication 103 may be set to be relatively long. In this way, the indication 103 may be displayed with a length according to the distance to the part 101 outside the imaging range.

[0044] 8A and 8B are diagrams showing examples of images displayed on the display screen of the display unit. An example in which the shape of the indication 103 is rectangular is shown in FIG. 8A and FIG. 8B. An example in which the number of the parts 101 outside the imaging range is relatively small is shown in FIG. 8A. An example in which the number of the parts 101 outside the imaging range is relatively large is shown in FIG. 8B. The indication 103 may be displayed in a size according to the number of the parts 101 outside the imaging range. In the example shown in FIG. 8A, the number of the parts 101 outside the imaging range is relatively small, so the size of the indication 103 may be set relatively small. On the other hand, in the example shown in FIG. 8B, the number of the parts 101 outside the imaging range is relatively large, so the size of the indication 103 may be set relatively large. In this way, the shape of the indication 103 may be rectangular. Also, the indication 103 may be displayed in a size according to the number of the parts 101 outside the imaging range.

[0045] FIG. 9 is a diagram showing an example of an image displayed on the display screen of the display unit. FIG. 9 shows an example in which the shape of the indication 103 is rectangular. Components 102S1 to 102S3 belonging to a first type and components 102T1 and 102T2 belonging to a second type different from the first type are located outside the imaging range of the imaging unit 16. FIG. 9 shows an example in which the number of out-of-imaging-range components 101 belonging to the first type is relatively large. FIG. 9 shows an example in which the number of out-of-imaging-range components 101 belonging to the second type is relatively small. The indication 103A corresponds to the out-of-imaging-range components 101 belonging to the first type, i.e., the components 102S1 to 102S3. The indication 103B corresponds to the out-of-imaging-range components 101 belonging to the second type, i.e., the components 102T1 and 102T2. The indications 103A and 103B can be displayed in a size according to the number of out-of-imaging-range components 101. Since the number of the outside imaging range parts 101 belonging to the first type is relatively large, the size of the indication 103A may be set to be relatively large. On the other hand, since the number of the outside imaging range parts 101 belonging to the second type is relatively small, the size of the indication 103B may be set to be relatively small. Since the indication 103A corresponds to the outside imaging range parts 101 belonging to the first type, it may be displayed in, for example, a first color. Since the indication 103B corresponds to the outside imaging range parts 101 belonging to the second type, it may be displayed in, for example, a second color different from the first color. In this way, the shape of the indication 103 may be rectangular. Also, the indication 103 may be displayed in a size according to the number of the outside imaging range parts 101. Also, the indication 103 may be displayed in a color according to the type of the outside imaging range parts 101.

[0046] FIG. 10 is a diagram showing an example of an image displayed on the display screen of the display unit. FIG. 10 shows an example in which the shape of the indication 103 is rectangular. FIG. 10 shows information indicating the number of the components 101 outside the imaging range corresponding to the indication 103. For example, in the example shown in FIG. 10, the number of the components 101 outside the imaging range located on the left side of the imaging range of the imaging unit 16 is two. Therefore, in the example shown in FIG. 10, the indication 103 displayed on the left side of the display screen 28A of the display unit 28 is marked with "2", which is the number of the components 101 outside the imaging range located on the left side of the imaging range. In the example shown in FIG. 10, the number of the components 101 outside the imaging range located on the right side of the imaging range of the imaging unit 16 is five. Therefore, in the example shown in FIG. 10, the indication 103 displayed on the right side of the display screen 28A of the display unit 28 is marked with "5", which is the number of the components 101 outside the imaging range located on the right side of the imaging range. The indication 103 can be displayed in a size according to the number of the components 101 outside the imaging range. Since the number of outside-imaging-range components 101 located on the left side of the imaging range of the imaging unit 16 is relatively small, the size of the indication 103 displayed on the left side of the display screen 28A of the display unit 28 may be set to be relatively small. Since the number of outside-imaging-range components 101 located on the right side of the imaging range of the imaging unit 16 is relatively large, the size of the indication 103 displayed on the right side of the display screen 28A of the display unit 28 may be set to be relatively large. In this manner, information indicating the number of outside-imaging-range components 101 may be attached to the indication 103.

[0047] FIG. 11 is a diagram showing an example of an image displayed on the display screen of the display unit. FIG. 11 shows an example in which rectangular indications 103A1 and 103A2 and a triangular indication 103B are displayed on the display screen 28A of the display unit 28. The components 101 outside the imaging range corresponding to the indications 103A1 and 103A2, i.e., the components 102U1 to 102U7, belong to, for example, a first type. Since the components 101 outside the imaging range corresponding to the indications 103A1 and 103A2 belong to the first type, the colors of the indications 103A1 and 103A2 may be set to, for example, a first color. Furthermore, since the components 101 outside the imaging range corresponding to the indications 103A1 and 103A2 belong to the first type, the shapes of the indications 103A1 and 103A2 may be set to, for example, a rectangle. The part 101 outside the imaging range corresponding to the indication 103B, i.e., the part 102V, belongs to, for example, a second type different from the first type. Since the part 101 outside the imaging range corresponding to the indication 103B belongs to the second type, the color of the indication 103B may be set to, for example, a second color different from the first color, and the shape of the indication 103B may be set to, for example, a triangle. The indications 103A1 and 103A2 may be displayed in a size according to the number of the parts outside the imaging range 101. Since the number of the parts outside the imaging range 101 corresponding to the indication 103A1 displayed on the left side of the display screen 28A of the display unit 28 is relatively small, the size of the indication 103A1 may be set to be relatively small. Since the number of the parts outside the imaging range 101 corresponding to the indication 103A2 displayed on the right side of the display screen 28A of the display unit 28 is relatively large, the size of the indication 103A2 may be set to be relatively large. In this way, the shape, color, etc. of the indication 103 may be made different depending on the type of the part 101 outside the imaging range.

[0048] Next, an example of an information providing method according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of an information providing method according to this embodiment.

[0049] First, in step S1, at least a part of the vehicle 11 is imaged by the imaging unit 16. As a result, the image acquisition unit 36 ​​acquires the captured image 110 captured by the imaging unit 16. After that, the process proceeds to step S2.

[0050] In step S2, the determination unit 42 determines whether or not the part 102 to be the target of the augmented reality display is included in the captured image 110. If the part 102 to be the target of the augmented reality display is not included in the captured image 110 (NO in step S2), the process proceeds to step S3. If the part 102 to be the target of the augmented reality display is included in the captured image 110 (YES in step S2), the process proceeds to step S4.

[0051] In step S3, the determination unit 42 determines whether or not there is an outside imaging range part 101. If there is no outside imaging range part 101 (NO in step S3), the process proceeds to step S5. If there is an outside imaging range part 101 (YES in step S3), the process proceeds to step S6.

[0052] In step S4, the determination unit 42 determines whether or not there is an outside imaging range part 101. If there is no outside imaging range part 101 (NO in step S4), the process proceeds to step S7. If there is an outside imaging range part 101 (YES in step S4), the process proceeds to step S8.

[0053] In step S5, the display control unit 44 displays the captured image 110 on the display screen 28A of the display unit .

[0054] In step S6, the display control unit 44 displays the captured image 110 on the display screen 28A of the display unit 28, and also displays the indication 103 on the display screen 28A.

[0055] In step S7, the display control unit 44 displays the captured image 110 on the display screen 28A of the display unit 28, and also displays information about the part 102 superimposed on the captured image 110.

[0056] In step S8, the display control unit 44 displays the captured image 110 on the display screen 28A of the display unit 28, superimposes information about the part 102 on the captured image 110, and also displays the indication 103 on the display screen 28A. In this manner, the process shown in FIG. 12 is completed.

[0057] As described above, according to this embodiment, the indication 103 indicating the presence of the part 101 outside the imaging range is displayed on the display screen 28A of the display unit 28, so that the user can easily grasp the presence of the part 101 outside the imaging range.

[0058] (Modification) An information providing method according to a modified example of this embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of an image displayed on the display screen of the display unit.

[0059] In this modification, when one part 102 is located within the imaging range of the imaging unit 16, and another part 102 that is a part 102 for operating the one part 102 is located outside the imaging range of the imaging unit 16, the following process may be performed by the display control unit 44. That is, in such a case, the display control unit 44 may display an indication 103 indicating the position of the other part 102 on the display screen 28A of the display unit 28.

[0060] An example of the one part 102W may be, for example, an interior light, but is not limited thereto. Also, an example of the other part 102 may be, for example, a switch for operating the interior light, but is not limited thereto. FIG. 13 shows an example in which the one part 102W is located within the imaging range of the imaging unit 16, while another part 102 (not shown) for operating the one part 102W is located outside the imaging range of the imaging unit 16. As shown in FIG. 13, an indication 103 indicating the position of the other part 102 may be displayed on the display screen 28A of the display unit 28. The indication 103 may be, for example, a moving image, but is not limited thereto.

[0061] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present invention.

[0062] For example, in the above embodiment, the case where the indication 103 is displayed semi-transparently has been described as an example, but the present invention is not limited to this. Only the outline of the indication 103 may be displayed. That is, only the outline of the indication 103 may be displayed opaquely or semi-transparently, and the part of the indication 103 other than the outline may be transparent. That is, a part of the captured image 110 may be displayed in the transparent part of the indication 103.

[0063] The above embodiment can be summarized as follows.

[0064] The information providing method includes an image acquiring step (S1) of acquiring an image (110) obtained by capturing an image of at least a part of a vehicle (11) by an imaging unit (16) provided in a terminal device (12), a determination step (S3, S4) of determining the presence of an outside-imaging-range part (101) that is one of a plurality of parts (102) provided in the vehicle and that is located outside the imaging range of the imaging unit, and a display step (S7, S8) of displaying the captured image on a display screen (28A) provided in the terminal device and, when at least one of the plurality of parts is included in the captured image, displaying information about the part together, and, in the display step (S8), when the presence of the outside-imaging-range part is determined in the determination step, an indication (103) indicating the presence of the outside-imaging-range part is also displayed together. According to this configuration, an indication indicating the presence of the outside-imaging-range part can be displayed on the display screen, so that the user can easily grasp the presence of the outside-imaging-range part.

[0065] The indication may be displayed in a color corresponding to a type of the outside image capture range part. With this configuration, the type of the outside image capture range part can be easily grasped based on the indication.

[0066] The indication may indicate a direction of the part outside the imaging range. With this configuration, the direction in which the part outside the imaging range is located can be ascertained based on the indication.

[0067] The indication may be displayed with a length corresponding to the distance to the item outside the imaging range. With this configuration, the distance to the item outside the imaging range can be grasped based on the indication.

[0068] The indication may be displayed in a size corresponding to the number of the items outside the imaging range. With this configuration, the number of the items outside the imaging range can be ascertained based on the indication.

[0069] The indication may be accompanied by information indicating the number of the items outside the imaging range. With this configuration, the number of the items outside the imaging range can be more easily grasped based on the indication.

[0070] The indication may be displayed on an edge of the display screen.

[0071] The shape of the indication may be rectangular or triangular.

[0072] The indication may be displayed semi-transparently.

[0073] Only the outline of the indication may be displayed.

[0074] When one of the components (102W) is located within the imaging range of the imaging unit, and another component that is a component for operating the one component is located outside the imaging range of the imaging unit, the display step may be configured to display the indication showing the position of the other component on the display screen.

[0075] The indication may comprise a moving image.

[0076] The program causes a computer to execute the information providing method described above.

[0077] The information providing device (12) comprises an imaging unit, a display unit (28) having a display screen, an image acquisition unit (36) that acquires an imaged image obtained by imaging at least a part of the vehicle with the imaging unit, a determination unit (42) that determines the presence of an outside-imaging-range part which is a part that is located outside the imaging range of the imaging unit among a plurality of parts provided on the vehicle, and a display control unit (44) that displays the imaged image on the display screen and, when at least one of the plurality of parts is included in the imaged image, also displays information regarding the part. When the determination unit determines the presence of the outside-imaging-range part, the display control unit displays the image on the display screen and also displays an indication of the presence of the outside-imaging-range part. [Explanation of symbols]

[0078] 10: Information provision system 11: Vehicle 12: Terminal device 14: Server device 16: Imaging unit 18: Sensor group 20, 48: Arithmetic section 22, 50: Storage section 24, 52: Communication unit 26: Operation unit 28: Display section 28A: Display screen 30: Acceleration sensor 32: Gyro sensor 34: Control unit 36: Image acquisition unit 38: Calibration section 40: Position and orientation determination section 42: Determination unit 44: Display control unit 46: Information acquisition unit 54: Relay station 56: Network 60A~60C: Operation buttons 60D: Recalibration button 60E: Voice search button 64a-64c, 64P-64R: Information 100: Inside the car 101: Outside the imaging area 102: Equipment, parts 102A~102O: Equipment 102P~102R, 102S1, 102T1, 102T2, 102U1, 102V, 102W: Parts 102a to 102c: Parts, components 103, 103A, 103A1, 103A2, 103B, 103C, 103P, 103Q, 103R: Indication 110: Captured image

Claims

1. an image acquisition step of acquiring an image obtained by imaging at least a part of the vehicle using an imaging unit provided in the terminal device; a determination step of determining the presence of an outside-imaging-range part, which is a part that is located outside the imaging range of the imaging unit, among a plurality of parts provided in the vehicle; a display step of displaying the captured image on a display screen of the terminal device, and, when at least one of the plurality of components is included in the captured image, displaying information about the component together; having In the display step, when the presence of the part outside the imaging range is determined in the determination step, an indication of the presence of the part outside the imaging range is also displayed, An information provision method in which, when one of the parts is located within the imaging range of the imaging unit and another part that is a part for operating the one part is located outside the imaging range of the imaging unit, the display step displays the indication showing the position of the other part on the display screen.

2. The information providing method according to claim 1, The information providing method, wherein the indication is displayed in a color corresponding to a type of the item outside the imaging range.

3. The information providing method according to claim 1 or 2, The information providing method, wherein the indication indicates a direction of an item outside the imaging range.

4. In the information providing method according to any one of claims 1 to 3, The information providing method, wherein the indication is displayed with a length corresponding to a distance to an item outside the imaging range.

5. In the information providing method according to any one of claims 1 to 4, The information providing method, wherein the indication is displayed in a size according to the number of items outside the imaging range.

6. The information providing method according to any one of claims 1 to 5, The information providing method includes adding information indicating the number of items outside the imaging range to the indication.

7. The information providing method according to any one of claims 1 to 6, The method of providing information, wherein the indication is displayed on an edge of the display screen.

8. The information providing method according to any one of claims 1 to 7, An information providing method, wherein the shape of the indication is rectangular or triangular.

9. The information providing method according to any one of claims 1 to 8, The information providing method, wherein the indication is displayed semi-transparently.

10. The information providing method according to any one of claims 1 to 8, A method of providing information, wherein only an outline of the indication is displayed.

11. The information providing method according to any one of claims 1 to 10, The method of providing information, wherein the indication comprises a moving image.

12. A program for causing a computer to execute the information providing method according to any one of claims 1 to 11.

13. An imaging unit; A display unit having a display screen; an image acquisition unit that acquires an image obtained by imaging at least a part of a vehicle using the imaging unit; a determination unit that determines the presence of an outside-imaging-range part, which is a part that is located outside an imaging range of the imaging unit, among a plurality of parts provided on the vehicle; a display control unit that displays the captured image on the display screen and, when at least one of the plurality of components is included in the captured image, also displays information about the component; Equipped with The display control unit, when the determination unit determines that the part outside the imaging range is present, displays the captured image on the display screen and also displays an indication of the presence of the part outside the imaging range, The display control unit of the information providing device displays the indication showing the position of the other part on the display screen when one of the parts is located within the imaging range of the imaging unit and another part that is a part for operating the one part is located outside the imaging range of the imaging unit.

Images