Information provision device, information provision method, information provision program, and storage medium

Abstract

To improve convenience.SOLUTION: An information provision apparatus 3 comprises: an image acquisition unit 323 that acquires an imaged image imaging a periphery of a mobile object; an attitude detection unit 327 that detects an attitude of a crewman within the mobile object; an area extraction unit 324 that extracts a plurality of noted areas in which visual lines within the imaged image are converged; an object recognition unit 325 that identifies any one noted area of the plurality of noted areas on the basis of the attitude, and recognizes an object included in the identified noted area; and an information provision unit 326 that provides object information associated with the object included in the noted area.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information providing apparatus, an information providing method, an information providing program, and a storage medium.

Background Art

[0002] Conventionally, there has been known an object specifying apparatus that specifies an object existing around a vehicle and reads out information such as a name related to the object by voice (see, for example, Patent Document 1). In the object specifying apparatus described in Patent Document 1, a facility or the like on a map existing in the instruction direction indicated by a vehicle occupant with a hand or a finger is specified as an object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Patent Document 1, it is necessary to cause a vehicle occupant who desires to obtain information related to an object to perform an operation of indicating the object with a hand or a finger, and there is a problem that convenience cannot be improved, for example.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information providing apparatus, an information providing method, an information providing program, and a storage medium that can improve convenience, for example.

Means for Solving the Problems

[0006] The information providing device according to claim 1 is characterized by comprising: an image acquisition unit that acquires captured images of the surroundings of a moving body; an attitude detection unit that detects the attitude of an occupant inside the moving body; an area extraction unit that extracts a plurality of areas of interest in which the gaze is concentrated within the captured images; an object recognition unit that identifies one of the plurality of areas of interest based on the attitude and recognizes an object included in the identified area of ​​interest; and an information providing unit that provides object information relating to the object included in the area of ​​interest. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a block diagram showing the configuration of the information provision system according to Embodiment 1. [Figure 2] Figure 2 is a block diagram showing the configuration of the in-vehicle terminal. [Figure 3] Figure 3 is a block diagram showing the configuration of the information provision device. [Figure 4] Figure 4 is a flowchart showing the method of providing information. [Figure 5] Figure 5 is a diagram illustrating the method of providing information. [Figure 6] Figure 6 is a block diagram showing the configuration of the information providing device according to Embodiment 2. [Figure 7] Figure 7 is a flowchart showing the methods of providing information. [Figure 8] Figure 8 is a diagram illustrating the method of providing information. [Figure 9] Figure 9 is a block diagram showing the configuration of an in-vehicle terminal according to Embodiment 3. [Figure 10] Figure 10 is a block diagram showing the configuration of the information providing device according to Embodiment 3. [Figure 11] Figure 11 is a flowchart showing the information provision method. [Figure 12] Figure 12 is a block diagram showing the configuration of the information providing device according to Embodiment 4. [Figure 13] Figure 13 is a flowchart showing the information provision method. [Figure 14]Figure 14 is a diagram illustrating the method of providing information. [Modes for carrying out the invention]

[0008] The embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0009] (Embodiment 1) [Outline of the Information Provision System] Figure 1 is a block diagram showing the configuration of the information provision system 1 according to Embodiment 1. Information provision system 1 is a system that provides object information (for example, the name of the object) regarding buildings and other objects present around a moving vehicle VE (Figure 1) to the occupant PA (see Figure 5). As shown in Figure 1, this information provision system 1 comprises an in-vehicle terminal 2 and an information provision device 3. These in-vehicle terminal 2 and information provision device 3 communicate via a wireless communication network NE (Figure 1). In Figure 1, the in-vehicle terminal 2 that communicates with the information provision device 3 is shown as an example of a single unit, but it may also be multiple units installed in multiple vehicles. Furthermore, multiple in-vehicle terminals 2 may be installed in a single vehicle in order to provide object information to multiple occupants riding in that vehicle.

[0010] [Configuration of the in-vehicle terminal] Figure 2 is a block diagram showing the configuration of the in-vehicle terminal 2. The in-vehicle terminal 2 is, for example, a stationary navigation device or a drive recorder installed in the vehicle VE. Note that the in-vehicle terminal 2 is not limited to a navigation device or a drive recorder, and a portable terminal such as a smartphone used by the passenger PA of the vehicle VE may be adopted. As shown in FIG. 2, this in-vehicle terminal 2 includes a voice input unit 21, a voice output unit 22, an imaging unit 23, a display unit 24, and a terminal main body 25.

[0011] The voice input unit 21 includes a microphone 211 (see FIG. 5) that inputs voice and converts it into an electrical signal, and generates voice information by performing A / D (Analog / Digital) conversion or the like on the electrical signal. In the first embodiment, the voice information generated by the voice input unit 21 is a digital signal. Then, the voice input unit 21 outputs the voice information to the terminal main body 25. The voice output unit 22 includes a speaker 221 (see FIG. 5), converts the digital voice signal input from the terminal main body 25 into an analog voice signal by D / A (Digital / Analog) conversion, and outputs voice corresponding to the analog voice signal from the speaker 221.

[0012] The imaging unit 23 captures the surroundings of the vehicle VE under the control of the terminal main body 25 and generates a captured image. Then, the imaging unit 23 outputs the generated captured image to the terminal main body 25. The display unit 24 is composed of a display display using liquid crystal or organic EL (Electro Luminescence) or the like, and displays various images under the control of the terminal main body 25.

[0013] As shown in FIG. 2, the terminal main body 25 includes a communication unit 251, a control unit 252, and a storage unit 253. The communication unit 251 transmits and receives information to and from the information providing device 三 through the network NE under the control of the control unit 252. The control unit 252 is implemented by a controller such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory unit 253, thereby controlling the operation of the entire in-vehicle terminal 2. The control unit 252 may be composed of integrated circuits other than a CPU or MPU, such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The memory unit 253 stores various programs executed by the control unit 252, as well as data necessary when the control unit 252 performs processing.

[0014] [Configuration of the information provision device] Figure 3 is a block diagram showing the configuration of the information providing device 3. The information providing device 3 is, for example, a server device. As shown in Figure 3, this information providing device 3 comprises a communication unit 31, a control unit 32, and a storage unit 33.

[0015] The communication unit 31, under the control of the control unit 32, transmits and receives information with the in-vehicle terminal 2 (communication unit 251) via the network NE. The control unit 32 is implemented by a controller such as a CPU or MPU executing various programs (including the information provision program according to this embodiment) stored in the storage unit 33, and controls the operation of the entire information provision device 3. Note that the control unit 32 is not limited to a CPU or MPU, but may also be composed of an integrated circuit such as an ASIC or FPGA. As shown in Figure 3, this control unit 32 includes a request information acquisition unit 321, a voice analysis unit 322, an image acquisition unit 323, a region extraction unit 324, an object recognition unit 325, and an information provision unit 326.

[0016] The request information acquisition unit 321 acquires request information from the vehicle VE's occupant PA requesting the provision of object information. In this embodiment 1, the request information is voice information generated by the voice input unit 21 based on the words (voice) spoken by the vehicle VE's occupant PA, which are captured by the voice input unit 21. In other words, the request information acquisition unit 321 acquires the request information (voice information) from the in-vehicle terminal 2 via the communication unit 31. The voice analysis unit 322 analyzes the request information (voice information) acquired by the request information acquisition unit 321.

[0017] The image acquisition unit 323 acquires the captured image generated by the imaging unit 23 from the in-vehicle terminal 2 via the communication unit 31. The region extraction unit 324 extracts (predicts) areas of interest within the captured image acquired by the image acquisition unit 323 where the gaze is concentrated (or where the gaze is likely to be concentrated). In this embodiment 1, the region extraction unit 324 extracts areas of interest within the captured image using so-called visual splendor technology. More specifically, the region extraction unit 324 extracts areas of interest within the captured image by image recognition (image recognition using AI (Artificial Intelligence)) using the first learning model shown below. The first learning model is obtained by using an eye tracker to identify the region where the subject's gaze is concentrated, using images with those regions pre-labeled as training images, and then using those training images to perform machine learning (e.g., deep learning) on ​​those regions.

[0018] The object recognition unit 325 recognizes objects included in the region of interest extracted by the region extraction unit 324 within the captured image. In this embodiment 1, the object recognition unit 325 recognizes objects included in the region of interest within the captured image by image recognition (AI-based image recognition) using the second learning model shown below. The second learning model is obtained by using photographs of various objects such as animals, mountains, rivers, lakes, and facilities as training images, and by using machine learning (e.g., deep learning) to learn the features of those objects based on those training images.

[0019] The information provision unit 326 provides object information relating to the object recognized by the object recognition unit 325. More specifically, the information provision unit 326 reads object information corresponding to the object recognized by the object recognition unit 325 from the object information DB (Data Base) 333 in the storage unit 33. Then, the information provision unit 326 transmits this object information to the in-vehicle terminal 2 via the communication unit 31.

[0020] The storage unit 33 stores various programs executed by the control unit 32 (information provision programs according to this embodiment), as well as data necessary for processing by the control unit 32. As shown in Figure 3, the storage unit 33 includes a first learning model DB 331, a second learning model DB 332, and an object information DB 333. The first learning model DB331 stores the first learning model described above. The second learning model DB332 stores the second learning model described above. The object information DB333 stores the object information described above. Here, the object information DB333 stores multiple pieces of object information associated with various objects. This object information includes information describing the object, such as the object's name, and is composed of text data, audio data, or image data.

[0021] [Information provision method] Next, we will explain the information provision method performed by the information provision device 3 (control unit 32). Figure 4 is a flowchart illustrating the information provision method. Figure 5 is a diagram illustrating the information provision method. Specifically, Figure 5 is a diagram showing the captured image IM generated by the imaging unit 23 and acquired in step S4. Here, Figure 5 illustrates a case where the imaging unit 23 is installed inside the vehicle VE so that the front of the vehicle VE is photographed from inside the vehicle VE through the windshield. Furthermore, Figure 5 illustrates a case where the captured image IM includes an occupant PA sitting in the passenger seat of the vehicle VE as a subject. In addition, Figure 5 illustrates a case where the occupant PA says the words "What's that?". The installation location of the imaging unit 23 is not limited to the locations described above. For example, the imaging unit 23 may be installed inside the vehicle VE so as to capture the left side, right side, or rear of the vehicle VE from inside the vehicle VE, or it may be installed outside the vehicle VE so as to capture the area around the vehicle VE. Furthermore, the occupants of the vehicle in this embodiment are not limited to occupants sitting in the passenger seat of the vehicle VE, but also include occupants sitting in the driver's seat or rear seats, etc. Also, the number of imaging units 23 is not limited to one, but may be multiple.

[0022] First, the request information acquisition unit 321 acquires request information (voice information) from the in-vehicle terminal 2 via the communication unit 31 (step S1). After step S1, the voice analysis unit 322 analyzes the request information (voice information) acquired in step S1 (step S2). After step S2, the voice analysis unit 322 determines whether or not a specific keyword is included in the request information (voice information) as a result of analyzing the request information (voice information) in step S2 (step S3). Here, the specific keywords in question are words used by the vehicle VE's occupant PA to request information about objects, and examples include words such as "what," "what is it," "I wonder what it is," and "tell me."

[0023] If it is determined that the specific keyword is not included (step S3: No), the control unit 32 returns to step S1. On the other hand, if it is determined that a specific keyword is included (Step S3: Yes), the image acquisition unit 323 acquires the captured image IM generated by the imaging unit 23 from the in-vehicle terminal 2 via the communication unit 31 (Step S4: Image acquisition step). In Figures 4 and 5, the image acquisition unit 323 acquires the captured image IM generated by the imaging unit 23 from the in-vehicle terminal 2 via the communication unit 31 at the moment when the vehicle VE occupant PA says "What's that?" (Step S3: Yes), but the system is not limited to this configuration. For example, the information providing device 3 sequentially acquires captured images generated by the imaging unit 23 from the in-vehicle terminal 2 via the communication unit 31. The image acquisition unit 323 may then acquire the captured image acquired at the moment when the vehicle VE occupant PA says "What's that?" (Step S3: Yes) from among the sequentially acquired captured images as the captured image to be used in processing from Step S4 onwards.

[0024] After step S4, the region extraction unit 324 extracts the region of interest Ar1 (Figure 5) in the captured image IM where the gaze is concentrated by image recognition using the first learning model stored in the first learning model DB331 (step S5: region extraction step). After step S5, the object recognition unit 325 recognizes the object OB1 included in the region of interest Ar1 extracted in step S5 within the captured image IM by image recognition using the second learning model stored in the second learning model DB332 (step S6: object recognition step). After step S6, the information provision unit 326 reads object information corresponding to object OB1 recognized in step S6 from the object information DB 333 and transmits the object information to the in-vehicle terminal 2 via the communication unit 31 (step S7: information provision step). The control unit 252 then controls the operation of at least one of the voice output unit 22 and the display unit 24 to notify the vehicle VE occupant PA of the object information transmitted from the information provision device 3 by at least one of voice, text, and image. For example, if object OB1 is "Moulin Rouge", the object information will be communicated to the vehicle VE occupant PA with voice such as "That's Moulin Rouge. They have a spectacular dance show at night." Also, for example, if object OB1 is not a building but an animal called a buffalo, the object information will be communicated to the vehicle VE occupant PA with voice such as "That's a buffalo. Buffaloes move in herds."

[0025] According to the first embodiment described above, the following effects are achieved. The information providing device 3 according to this embodiment 1 acquires a captured image IM of the area around the vehicle VE and extracts a region of interest Ar1 within the captured image IM where the gaze is concentrated. The information providing device 3 then recognizes an object OB1 included in the region of interest Ar1 within the captured image IM and transmits object information about the object OB1 to the in-vehicle terminal 2. As a result, the occupant PA of the vehicle VE who wishes to obtain object information about the object OB1 recognizes the object information about the object OB1 when the object information is notified from the in-vehicle terminal 2. Therefore, for the occupant PA of a vehicle VE who wishes to obtain object information about object OB1, it is no longer necessary to have them point to the object OB1 with their hand or finger, as in the conventional method, thus improving convenience.

[0026] In particular, the information providing device 3 uses so-called visual splendor technology to extract the attention region Ar1 within the captured image IM where the gaze is concentrated. Therefore, even if the vehicle VE's occupant PA does not point to the object OB1 with their hand or finger, the region containing the object OB1 can be accurately extracted as the attention region Ar1.

[0027] Furthermore, the information providing device 3 provides object information in response to a request from the vehicle VE's occupant PA. Therefore, compared to a configuration that provides object information at all times regardless of the request, the processing load on the information providing device 3 can be reduced.

[0028] (Embodiment 2) Next, we will describe Embodiment 2. In the following description, components similar to those in Embodiment 1 described above are denoted by the same reference numerals, and their detailed descriptions are omitted or simplified. Figure 6 is a block diagram showing the configuration of the information providing device 3A according to Embodiment 2. In the information providing device 3A according to this second embodiment, as shown in Figure 6, the control unit 32 has the added function of a posture detection unit 327 compared to the information providing device 3 (see Figure 3) described in the first embodiment. In addition, the function of the object recognition unit 325 has been modified in the information providing device 3A. For the sake of explanation, the object recognition unit according to this second embodiment will be referred to as the object recognition unit 325A (see Figure 6) below. Furthermore, in the information providing device 3A, a third learning model DB334 (see Figure 6) has been added to the storage unit 33.

[0029] The posture detection unit 327 detects the posture of the occupant PA of the vehicle VE. In this second embodiment, the posture detection unit 327 detects the posture by so-called skeletal detection. More specifically, the posture detection unit 327 detects the posture of the occupant PA by detecting the skeleton of the occupant PA of the vehicle VE that is included as a subject in the captured image IM using image recognition (AI-based image recognition) using the third learning model shown below. The third learning model is obtained by using images of a person taken by a person, with the positions of the person's joint points pre-labeled as training images, and then using machine learning (e.g., deep learning) to determine the positions of the joint points based on these training images. The third learning model DB334 then stores this third learning model.

[0030] The object recognition unit 325A has the same functions as the object recognition unit 325 described in Embodiment 1 above, as well as a function (hereinafter referred to as the additional function) that is executed when multiple areas of interest are extracted in the captured image IM by the region extraction unit 324. The additional function is as follows. In other words, the object recognition unit 325A identifies one of the multiple areas of interest based on the attitude of the occupant PA detected by the attitude detection unit 327. Then, similar to the object recognition unit 325 described in Embodiment 1 above, the object recognition unit 325A recognizes the object included in the identified area of ​​interest within the captured image IM by image recognition using a second learning model.

[0031] Next, we will explain the information provision method performed by the information provision device 3A. Figure 7 is a flowchart illustrating the information provision method. Figure 8 is a diagram illustrating the information provision method. Specifically, Figure 8 corresponds to Figure 5 and shows the captured image IM generated by the imaging unit 23 and acquired in step S4. In the information provision method according to this second embodiment, as shown in Figure 7, steps S6A1 to S6A3 are added to the information provision method described in the first embodiment (see Figure 4). Therefore, only steps S6A1 to S6A3 will be mainly described below. Steps S6A1 to S6A3 and S6 correspond to the object recognition step according to this embodiment.

[0032] Step S6A1 is executed after step S5. Specifically, in step S6A1, the control unit 32 determines whether there are multiple regions of interest extracted in step S5. Figure 8 illustrates the case where three regions of interest, Ar1 to Ar3, are extracted in step S5. If it is determined that there is only one area of ​​interest (step S6A1: No), the control unit 32 proceeds to step S6 and recognizes an object (e.g., object OB1) included in that single area of ​​interest (e.g., area of ​​interest Ar1, as in the first embodiment described above).

[0033] On the other hand, if the control unit 32 determines that there are multiple areas of interest (step S6A1: Yes), the control unit 32 proceeds to step S6A2. Then, in step S6A2, the attitude detection unit 327 detects the attitude of the occupant PA of the vehicle VE, which is included as a subject in the captured image IM, by image recognition using the third learning model stored in the third learning model DB334.

[0034] After step S6A2, the object recognition unit 325A identifies the orientation DI (Figure 8) of the occupant PA's face FA and fingers FI from the occupant PA's posture detected in step S6A2. Then, in the captured image IM, the object recognition unit 325A identifies one of the three areas of interest Ar1 to Ar3 extracted in step S5, namely area Ar2, which is located at orientation DI relative to the occupant PA (step S6A3). Then, after step S6A3, the control unit 32 proceeds to step S6 and recognizes the object OB2 (Figure 8) included in the single region of interest Ar2.

[0035] According to this embodiment 2 described above, in addition to the same effects as in embodiment 1 described above, the following effects are achieved. The information providing device 3A according to this second embodiment detects the posture of the vehicle occupant PA when multiple focus regions Ar1 to Ar3 are extracted from the captured image IM, and identifies one focus region Ar2 from the multiple focus regions Ar1 to Ar3 based on that posture. The information providing device 3 then recognizes the object OB2 included in the identified focus region Ar2. Therefore, even when multiple regions of interest Ar1 to Ar3 are extracted from the captured image IM, the region containing object OB2 for which the vehicle VE's occupant PA wishes to obtain object information can be accurately identified as region of interest Ar1. Thus, appropriate object information can be provided to the vehicle VE's occupant PA.

[0036] In particular, the information providing device 3A detects the posture of the vehicle VE's occupant PA by so-called skeletal detection. Therefore, it is possible to detect the posture with high accuracy, and even when multiple areas of interest Ar1 to Ar3 are extracted from the captured image IM, it is possible to provide appropriate object information to the vehicle VE's occupant PA.

[0037] (Embodiment 3) Next, Embodiment 3 will be described. In the following description, components similar to those in Embodiment 1 described above are denoted by the same reference numerals, and their detailed descriptions are omitted or simplified. Figure 9 is a block diagram showing the configuration of the in-vehicle terminal 2B according to Embodiment 3. In the in-vehicle terminal 2B according to this third embodiment, as shown in Figure 9, a sensor unit 26 is added to the in-vehicle terminal 2 (see Figure 2) described in the first embodiment above. As shown in Figure 9, the sensor unit 26 includes a lidar 261 and a GNSS (Global Navigation Satellite System) sensor 262. The LIDA261 discretely measures the distance to an object in the external environment, recognizes the object's surface as a three-dimensional point cloud, and generates point cloud data. Note that any sensor capable of measuring the distance to an object in the external environment, such as millimeter-wave radar or sonar, may be used instead of the LIDA261. The GNSS sensor 262 uses GNSS to receive radio waves containing positioning data transmitted from navigation satellites. This positioning data is used to detect the absolute position of the vehicle VE from latitude and longitude information, etc., and corresponds to the position information in this embodiment. The GNSS used may be, for example, GPS (Global Positioning System) or another system. The sensor unit 26 then outputs output data, such as the point cloud data and the positioning data, to the terminal unit 25.

[0038] Figure 10 is a block diagram showing the configuration of the information providing device 3B according to Embodiment 3. Furthermore, in the information providing device 3B according to this third embodiment, the function of the object recognition unit 325 is modified compared to the information providing device 3 described in the first embodiment (see Figure 3). For convenience of explanation, the object recognition unit according to this third embodiment will be referred to as the object recognition unit 325B (see Figure 10) below. In addition, the second learning model DB 332 is omitted in the information providing device 3B, and a map DB 335 (see Figure 10) is added to the storage unit 33.

[0039] Map DB335 stores map data. This map data includes road data represented by links corresponding to roads and nodes corresponding to road connections (intersections), as well as facility information, which associates each facility with its location (hereinafter referred to as "facility location"). The object recognition unit 325B acquires output data from the in-vehicle terminal 2 via the communication unit 31 (point cloud data generated by the lidar 261 and positioning data received by the GNSS sensor 262). Based on this output data, the captured image IM, and the map data stored in the map DB 335, the object recognition unit 325B recognizes objects within the captured image IM that are included in the area of ​​interest extracted by the area extraction unit 324. The object recognition unit 325B described above corresponds to the object recognition unit according to this embodiment, as well as the location information acquisition unit and the facility information acquisition unit.

[0040] Next, we will describe the information provision method performed by the information provision device 3B. Figure 11 is a flowchart showing the information provision method. In the information provision method according to this third embodiment, as shown in Figure 11, steps S6B1 to S6B5 are added in place of step S6 compared to the information provision method described in the first embodiment (see Figure 4). Therefore, only steps S6B1 to S6B5 will be mainly described below. Steps S6B1 to S6B5 correspond to the object recognition step according to this embodiment.

[0041] Step S6B1 is performed after step S5. Specifically, in step S6B1, the object recognition unit 325B acquires output data from the in-vehicle terminal 2 via the communication unit 31, specifically the output data from the sensor unit 26 (point cloud data generated by the lidar 261 and positioning data generated by the GNSS sensor 262). In Figure 11, the object recognition unit 325B is configured to acquire output data from the in-vehicle terminal 2 via the communication unit 31 when the vehicle VE occupant PA utters a word containing a specific keyword (step S3: Yes), but this configuration is not limited to this. For example, the information providing device 3B sequentially acquires output data from the in-vehicle terminal 2 via the communication unit 31. The object recognition unit 325B may then acquire the output data acquired at the moment the vehicle VE occupant PA utters a word containing a specific keyword (step S3: Yes) from the sequentially acquired output data as output data to be used in processing from step S6B1 onwards.

[0042] After step S6B1, the object recognition unit 325B estimates the position of the vehicle VE based on the output data acquired in step S6B1 (positioning data received by the GNSS sensor 262) and the map data stored in the map DB 335 (step S6B2). After step S6B2, the object recognition unit 325B estimates the position of the object included in the region of interest in the captured image IM extracted in step S5 (step S6B3). Here, the object recognition unit 325B estimates the position of the object using the output data (point cloud data) acquired in step S6B1, the position of the vehicle VE estimated in step S6B2, and the position of the region of interest in the captured image IM extracted in step S5.

[0043] After step S6B3, the object recognition unit 325B acquires facility information from the map DB 335, including facility locations that are approximately the same as the location of the object estimated in step S6B3 (step S6B4). After step S6B4, the object recognition unit 325B recognizes the facilities included in the facility information acquired in step S6B4 as objects included in the area of ​​interest in the captured image IM extracted in step S5 (step S6B5). Then, after step S6B5, the control unit 32 proceeds to step S7.

[0044] According to this embodiment 3 described above, in addition to the same effects as in embodiment 1 described above, the following effects are achieved. The information providing device 3B according to this third embodiment recognizes objects included in the area of ​​interest within the captured image IM based on location information (positioning data received by the GNSS sensor 262) and facility information. In other words, the information providing device 3B recognizes objects included in the area of ​​interest within the captured image IM based on information commonly used in navigation devices (location information and facility information). Therefore, it is not necessary to provide the second learning model DB332 described in Embodiment 1 above, and the configuration of the information providing device 3B can be simplified.

[0045] (Embodiment 4) Next, we will describe Embodiment 4. In the following description, components similar to those in Embodiment 1 described above are denoted by the same reference numerals, and their detailed descriptions are omitted or simplified. Figure 12 is a block diagram showing the configuration of the information providing device 3C according to Embodiment 4. In the information providing device 3C according to this fourth embodiment, as shown in Figure 12, the functions of the object recognition unit 325 and the information providing unit 326 have been modified compared to the information providing device 3 (see Figure 3) described in the first embodiment described above. For the sake of convenience of explanation, the object recognition unit according to this fourth embodiment will be referred to as the object recognition unit 325C (see Figure 12), and the information providing unit according to this fourth embodiment will be referred to as the information providing unit 326C (see Figure 12).

[0046] The object recognition unit 325C has the same functions as the object recognition unit 325 described in Embodiment 1 above, as well as a function (hereinafter referred to as the additional function) that is executed when multiple areas of interest are extracted in the captured image IM by the region extraction unit 324. The additional function is as follows. In other words, the object recognition unit 325C recognizes objects contained in each of the multiple areas of interest within the captured image IM by image recognition using a second learning model.

[0047] The information provision unit 326C has the same functions as the information provision unit 326 described in Embodiment 1 above, as well as a function (hereinafter referred to as the additional function) that is executed when multiple areas of interest are extracted within the captured image IM by the area extraction unit 324. The additional function is as follows. In other words, the information provision unit 326C identifies one object from among the objects recognized by the object recognition unit 325C based on the analysis results from the voice analysis unit 322 and the object information stored in the object information DB 333. Then, the information provision unit 326C transmits the object information corresponding to the identified object to the in-vehicle terminal 2 via the communication unit 31.

[0048] Next, we will describe the information provision method performed by the information provision device 3C. Figure 13 is a flowchart illustrating the information provision method. Figure 14 is a diagram illustrating the information provision method. Specifically, Figure 14 corresponds to Figure 5 and shows the captured image IM generated by the imaging unit 23 and acquired in step S4. Here, unlike the example in Figure 5, Figure 14 illustrates a case where the occupant PA sitting in the passenger seat of vehicle VE says, "What is that red building?" In the information provision method according to this embodiment 4, as shown in Figure 13, steps S6C1, S6C2, and S7C are added to the information provision method described in the above-described embodiment 1 (see Figure 4). Therefore, in the following, only steps S6C1, S6C2, and S7C will be mainly described. Steps S6C1, S6C2, and S6 correspond to the object recognition step according to this embodiment, respectively. Steps S7C and S7 correspond to the information provision step according to this embodiment, respectively.

[0049] Step S6C1 is performed after step S5. Specifically, in step S6C1, the control unit 32 determines whether there are multiple regions of interest extracted in step S5, similar to step S6A1 described in the second embodiment above. Figure 14 illustrates the case where three regions of interest, Ar1 to Ar3, are extracted in step S5, similar to Figure 8. If it is determined that there is only one area of ​​interest (step S6C1: No), the control unit 32 proceeds to step S6 and recognizes an object (e.g., object OB1) included in that single area of ​​interest (e.g., area of ​​interest Ar1, as in the first embodiment described above).

[0050] On the other hand, if it is determined that there are multiple areas of interest (step S6C1: Yes), the control unit 32 proceeds to step S6C2. Then, the object recognition unit 325C recognizes objects OB1 to OB3, which are contained in the three areas of interest Ar1 to Ar3 extracted in step S5, respectively, within the captured image IM by image recognition using the second learning model stored in the second learning model DB332 (step S6C2).

[0051] After step S6C2, the information provision unit 326C executes step S7C. Specifically, in step S7C, the information providing unit 326C identifies one object from among the objects recognized in step S6C2. Here, the information providing unit 326C identifies the object based on the attributes of the object included in the request information (voice information) and the three object information entries stored in the object information DB 333 that correspond to each of the objects OB1 to OB3 recognized in step S6C2. The object attributes included in the request information (voice information) are generated in step S2 when the request information (voice information) is analyzed. For example, as shown in Figure 14, when the occupant PA of vehicle VE says "What is that red building?", the words "red" and "building" become the object attributes. Specifically, object attributes are information indicating the color such as red, the shape such as a square, and the type such as a building. Then, in step S7C, the information providing unit 326C refers to the three object information corresponding to each object OB1 to OB3 and identifies one object (for example, object OB3) that corresponds to the object information containing the text data of "red" and "building". The information providing unit 326C also transmits the object information corresponding to the identified object to the in-vehicle terminal 2 via the communication unit 31.

[0052] According to this embodiment 4 described above, in addition to the same effects as in embodiment 1 described above, the following effects are achieved. The information providing device 3C according to this fourth embodiment provides object information relating to one of the objects OB1 to OB3 contained in each of the multiple areas of interest Ar1 to Ar3, based on the analysis results of the requested information (audio information) when multiple areas of interest Ar1 to Ar3 are extracted from the captured image IM. Therefore, even when multiple areas of interest Ar1 to Ar3 are extracted from the captured image IM, the object OB3 for which the vehicle VE's occupant PA wishes to obtain object information can be accurately identified. Thus, appropriate object information can be provided to the vehicle VE's occupant PA.

[0053] (Other embodiments) While embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the embodiments 1 to 4 described above. The information providing devices 3, 3A to 3C according to the above-described embodiments 1 to 4 executed various processes such as the image acquisition step, region extraction step, object recognition step, and information provision step, triggered by the acquisition of request information (voice information) containing a specific keyword. However, the information providing device according to this embodiment may be configured to execute these processes at all times, even without acquiring request information (voice information) containing a specific keyword. Furthermore, the request information according to this embodiment is not limited to voice information, but may also be operation information corresponding to operations performed by the vehicle VE's PA on switches or other operating parts provided on the in-vehicle terminals 2, 2B.

[0054] In the embodiments 1 to 4 described above, all components of the information providing devices 3, 3A to 3C may be provided in the in-vehicle terminals 2 and 2B. In this case, the in-vehicle terminals 2 and 2B correspond to the information providing devices according to this embodiment. Alternatively, some functions of the control unit 32 and some functions of the storage unit 33 in the information providing devices 3, 3A to 3C may be provided in the in-vehicle terminals 2 and 2B. In this case, the entire information providing system 1 corresponds to the information providing devices according to this embodiment. [Explanation of symbols]

[0055] 3,3A~3C Information providing device 321 Request information acquisition unit 322 Voice Analysis Unit 323 Image acquisition unit 324 Region extraction part 325,325A~325C Object recognition part 326,326C Information Provision Department 327 Attitude detection unit

Claims

1. An image acquisition unit that acquires images taken around a moving object, A posture detection unit for detecting the posture of the occupant within the moving vehicle, A region extraction unit that predicts multiple areas of interest in the captured image where the gaze is likely to be concentrated, Based on the aforementioned posture, an object recognition unit identifies one of the multiple areas of interest and recognizes an object included in the identified area of ​​interest. The system includes an information providing unit that provides object information relating to objects included in the area of ​​interest. An information-providing device characterized by the following features.

2. The aforementioned captured image is The subject includes the occupants inside the aforementioned moving vehicle, The aforementioned attitude detection unit is The posture is detected by detecting the skeletal structure of the occupant based on the captured image. The information providing device according to feature 1.

3. A position information acquisition unit that acquires position information relating to the position of the moving object, It further includes a facility information acquisition unit that acquires facility information related to the facility, The object recognition unit, Based on the location information and facility information, recognize objects included in the area of ​​interest. The information providing device according to claim 1 or 2, characterized by the above.

4. The system further includes a request information acquisition unit that acquires request information from an occupant in the mobile vehicle requesting the provision of the object information, The aforementioned information provision unit, The object information is provided in accordance with the requested information. An information providing device according to any one of features 1 to 3.

5. The aforementioned request information is, This is audio information relating to the voice spoken by the aforementioned crew member. The system further comprises a voice analysis unit that analyzes the aforementioned voice information, The region extraction unit, Multiple of the aforementioned areas of interest are extracted, The object recognition unit, Each of the objects included in the aforementioned multiple regions of interest is recognized, The aforementioned information provision unit, Based on the analysis results of the aforementioned audio information, the object information relating to one of the objects included in each of the multiple areas of interest is provided. The information providing device according to feature 4.

6. An information provision method performed by an information provision device, Image acquisition step: Obtaining images taken around a moving object, A posture detection step for detecting the posture of the occupant within the moving vehicle, A region extraction step predicts multiple areas of interest within the captured image where the gaze is likely to be concentrated, Based on the aforementioned posture, an object recognition step is performed to identify one of the multiple areas of interest and to recognize an object included in the identified area of ​​interest. The step includes providing information about an object that is included in the region of interest, and providing information about the object. A method for providing information characterized by the following features.

7. Image acquisition step: Obtaining images taken around a moving object, A posture detection step for detecting the posture of the occupant within the moving vehicle, A region extraction step predicts multiple areas of interest within the captured image where the gaze is likely to be concentrated, An object recognition step of recognizing an object included in the region of interest within the captured image, Based on the aforementioned posture, an information provision step is performed to identify one of the multiple areas of interest and to provide object information relating to an object included in the identified area of ​​interest. A program that provides information to cause a computer to execute a command.

8. Image acquisition step: Obtaining images taken around a moving object, A posture detection step for detecting the posture of the occupant within the moving vehicle, A region extraction step predicts multiple areas of interest within the captured image where the gaze is likely to be concentrated, Based on the aforementioned posture, an object recognition step is performed to identify one of the multiple areas of interest and to recognize an object included in the identified area of ​​interest. The system stores an information provision program that causes a computer to perform an information provision step of providing object information about an object included in the aforementioned region of interest. A storage medium characterized by the following features.

Images