Ground object position information output device, assist control device, ground object position information output method, and computer-readable storage medium

Abstract

The invention provides a ground object position information output apparatus, an assist control apparatus, a ground object position information output method, and a computer readable storage medium. A ground object position information output device is provided with: an image information acquisition unit that acquires a plurality of images obtained from a plurality of different viewpoints on a road, and first information indicating the viewpoint position and observation direction of each of the plurality of images; a ground object position specifying unit that specifies the position of the ground object on the basis of the difference in the appearance pattern of the image of the ground object in each of the plurality of images, and the viewpoint position and observation direction of each of the plurality of images specified from the first information; and a ground object position information output unit that outputs information indicating the position of the ground object determined by the ground object position determination unit.

Description

Technical Field

[0001] The invention relates to a ground object position information output device, an auxiliary control device, a ground object position information output method and a computer-readable storage medium. Background Art

[0002] In recent years, efforts to provide access to sustainable transportation systems for disadvantaged people among traffic participants have become increasingly active. In order to achieve this goal, research and development are being devoted to further improving the safety and convenience of traffic through research and development related to preventive safety technologies. Patent documents 1 to 3 describe technologies related to the generation of map information.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2020-24693

[0004] Patent Document 2: Japanese Patent Application Publication No. 2023-27065

[0005] Patent Document 3: Japanese Patent Application Publication No. 2013-30006 Summary of the invention

[0006] However, in terms of preventive safety technology, effective use of ground objects has become a problem. In order to solve the above-mentioned problem, the present application aims to effectively obtain the position of ground objects from images obtained by shooting during road driving. Moreover, it also contributes to the development of sustainable transportation systems.

[0007] In a first aspect of the present invention, a device for outputting information on a location of a ground feature is provided. The device for outputting information on a location of a ground feature includes an image information acquisition unit that acquires a plurality of images obtained from a plurality of different viewpoints on a road, and first information indicating the viewpoint positions and observation directions of each of the plurality of images. The device for outputting information on a location of a ground feature includes a device for determining a location of the ground feature based on the difference in the appearance of the image of the ground feature in each of the plurality of images, and the viewpoint positions and observation directions of each of the plurality of images determined based on the first information. The device for outputting information on a location of a ground feature includes a device for outputting information indicating the location of the ground feature determined by the device for determining a location of the ground feature.

[0008] In the above-mentioned ground feature position information output device, the ground feature position determination unit can determine the position of the ground feature based on whether each of the multiple images contains the image of the ground feature, and the respective viewpoint positions and observation directions of the multiple images determined according to the first information.

[0009] In any of the above-mentioned devices for outputting information on the position of a ground object, when a first image among the plurality of images includes an image of a first ground object, and a second image among the plurality of images does not include an image of the ground object, the ground object position determination unit may determine a range between a viewpoint position of the first image and a viewpoint position of the second image as the position of the ground object. The second image may be an image whose viewpoint position is different from that of the first image and whose observation direction differs from that of the first image by less than a predetermined value.

[0010] In any of the above-mentioned ground object position information output devices, the ground object position determination unit can perform the following processing: determine the position of the ground object from multiple first images in a first observation direction among the multiple images, and when the image of the ground object is included in a second image in a second observation direction different from the first observation direction, correct the position of the ground object based on the viewpoint position of the second image and the position of the ground object determined based on the multiple first images.

[0011] In any of the above-mentioned ground object position information output devices, the ground object position determination unit can perform the following processing: determine a first position as the position of the ground object based on a plurality of first images in a first observation direction among the plurality of images, determine a third position as the position of the ground object based on a plurality of third images in a third observation direction opposite to the first observation direction among the plurality of images, and determine the position of the ground object based on the first position and the third position.

[0012] The above-ground features may include overpasses, vehicle stop lines, pedestrian crossings and / or road signs.

[0013] In a second aspect of the present invention, an auxiliary control device is provided. The auxiliary control device includes any of the above-mentioned feature position information output devices. The auxiliary control device includes an information acquisition unit that acquires second information including the position information of a user terminal and the position information of a vehicle traveling on the road. The auxiliary control device includes an auxiliary control unit that performs traffic assistance for a vehicle traveling on the road and / or a user associated with the user terminal based on the second information and the information indicating the position of the feature output from the feature position information output unit.

[0014] In the auxiliary control device, the above-mentioned feature may be a pedestrian overpass. The auxiliary control device may further include a determination unit for determining whether the user terminal is located at the pedestrian overpass based on the second information. The auxiliary control unit may determine whether to perform the traffic assistance in consideration of the determination result of the determination unit.

[0015] In any of the above-mentioned auxiliary control devices, when the determination unit determines that the position of the user terminal is located at the position of the overpass, and the height of the user terminal determined based on the position information of the user terminal is above a predetermined value, the auxiliary control unit may determine not to perform the traffic assistance.

[0016] In any of the above-mentioned auxiliary control devices, when the determination unit determines that the position of the user terminal is not at the position of the overpass, or when the determination unit determines that the position of the user terminal is at the position of the overpass, and the height of the user terminal determined based on the position information of the user terminal is less than a predetermined value, the auxiliary control unit may perform control for executing the traffic assistance on the condition that it is determined that the position of the vehicle is close to the position of the user terminal in the latitude and longitude coordinate system.

[0017] In a third aspect of the present invention, a method for outputting information on the position of a ground object is provided. The method for outputting information on the position of a ground object comprises the steps of acquiring a plurality of images obtained from a plurality of different viewpoints on a road, and first information indicating the viewpoint positions and observation directions of each of the plurality of images. The method for outputting information on the position of a ground object comprises the steps of determining the position of the ground object based on the difference in the appearance of the image of the ground object in each of the plurality of images, and the viewpoint positions and observation directions of each of the plurality of images determined based on the first information. The method for outputting information on the position of a ground object comprises the steps of outputting information indicating the position of the ground object.

[0018] In a fourth aspect of the present invention, there is provided a program that causes a computer to function as any of the above-mentioned ground object position information output devices.

[0019] In addition, the above summary of the invention does not list all the features of the present invention. In addition, a sub-combination of these feature groups may also be an invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A usage scenario of the assistance system 10 is schematically shown.

[0021] Figure 2 The functional structure of the auxiliary device 60 is shown.

[0022] Figure 3 is a diagram for explaining an example of a process of determining the position of the overpass 50 based on an image provided by the environment image server 62 .

[0023] Figure 4 is a diagram for explaining another example of a process of determining the position of the overpass 50 based on an image provided by the environment image server 62 .

[0024] Figure 5 is a diagram for explaining another example of a process of determining the position of the overpass 50 based on an image provided by the environment image server 62 .

[0025] Figure 6 An example of a flowchart related to processing related to the method of outputting feature position information executed in the support device 60 is shown.

[0026] Figure 7 An example of the execution flow of processing performed in the user terminal 82 , the vehicle-mounted processing device 40 , the imaging device 70 , and the support device 60 included in the vehicle 20 is shown.

[0027] Figure 8 An example of a flowchart related to a process related to the assist control method executed by the assist control device 202 is shown.

[0028] Fig. 9 An example of a computer 2000 is shown. DETAILED DESCRIPTION

[0029] Hereinafter, the present invention will be described by way of the embodiments of the invention, but the following embodiments do not limit the claimed invention. In addition, not all combinations of features described in the embodiments are essential to the solution means of the invention.

[0030] Figure 1 The following schematically shows a usage scenario of the support system 10 . The support system 10 includes vehicles 20 a , 20 b , and 20 c , user terminals 82 a and 82 b , a support device 60 , a camera 70 , an environment image server 62 , and a map server 64 .

[0031] The vehicle 20a includes the vehicle-mounted processing device 40a, the vehicle 20b includes the vehicle-mounted processing device 40b, and the vehicle 20c includes the vehicle-mounted processing device 40c. The user terminal 82a is a terminal held by the user 80a, and the user terminal 82b is a terminal held by the user 80b. In this embodiment, the user 80a and the user 80b are pedestrians. In this embodiment, the user 80a walks on the overpass 50, and the user 80b walks on the road 90.

[0032] In this embodiment, the vehicle 20a, the vehicle 20b, and the vehicle 20c are sometimes collectively referred to as "the vehicle 20". The vehicle-mounted processing device 40a, the vehicle-mounted processing device 40b, and the vehicle-mounted processing device 40c are sometimes collectively referred to as "the vehicle-mounted processing device 40". The user terminal 82a and the user terminal 82b are sometimes collectively referred to as "the user terminal 82". The user 80a and the user 80b are sometimes collectively referred to as "the user 80".

[0033] The vehicle 20 is a vehicle traveling on the road 90. The vehicle 20 is an example of a mobile body. The vehicle-mounted processing device 40 is configured to include various sensors including a position sensor of a global navigation satellite system (GNSS) receiver, a speed sensor such as a vehicle speed sensor, etc. The vehicle-mounted processing device 40 has a function of processing information acquired by various sensors provided in the vehicle-mounted processing device 40, and a function of communicating with the auxiliary device 60. The vehicle-mounted processing device 40 provides an advanced driving assistance system (ADAS) function provided in the vehicle 20.

[0034] The user terminal 82 is a portable terminal such as a smartphone. The user terminal 82 is an example of a mobile object. The user terminal 82 periodically transmits the current position information of the user terminal 82 detected by the position sensor including the GNSS receiver to the auxiliary device 60. For example, the user terminal 82 transmits the latitude and longitude information and the altitude information indicating the current position of the user terminal 82 in the three-dimensional space.

[0035] The camera 70 is a camera installed on a traffic infrastructure, and acquires the positions of the vehicle 20 and the user 80 within the camera range of the camera 70 by analyzing the captured images, and transmits the acquired positions of the vehicle 20 and the user 80 to the assisting device 60 .

[0036] The auxiliary device 60 receives information sent from the vehicle-mounted processing device 40, the user terminal 82, and the camera 70 via mobile communication. The auxiliary device 60 can receive information sent from the vehicle-mounted processing device 40, the user terminal 82, and the camera 70 via a communication line such as mobile communication, the Internet, and a dedicated line.

[0037] The environment image server 62 provides images obtained from multiple viewing directions for each of multiple viewpoints on the road. The environment image server 62, for example, collects images of the environment around the vehicle captured from a vehicle traveling on the road 90, and generates images obtained from multiple viewing directions for each of multiple viewpoints on the road based on the collected images. The environment image server 62 generates images obtained from multiple viewing directions for each of multiple viewpoints on the road based on three-dimensional information obtained from a vehicle traveling on the road 90 by a fisheye camera or the like. For example, the environment image server 62 provides images obtained from a direction specified by the user with the position specified by the user as the viewpoint. By changing the position specified by the user on the road, the environment image server 62 can provide images that are visually confirmed when the user actually moves on the road.

[0038] The auxiliary device 60 determines the position of the overpass 50 based on the multiple images obtained from the environment image server 62 with different positions as viewpoints. For example, the auxiliary device 60 determines the position of the overpass 50 in the driving direction on the road 90 based on the viewpoint on the road 90 corresponding to the image where the overpass 50 is displayed and the viewpoint on the road 90 corresponding to the image where the overpass 50 is not displayed, and stores the determined position of the overpass 50. Figures 3 to 5 etc. describe a method of determining the position of the overpass 50. The overpass 50 is an example of a land feature.

[0039] The map server 64 provides map information. The map information provided by the map server 64 includes the location of the road, the type of the road, the type of the road, etc., but does not include information indicating the location of the overpass 50.

[0040] The assistance device 60 performs traffic assistance for the vehicle 20 based on the position information of the vehicle 20 and the user 80 received from the vehicle-mounted processing device 40, the user terminal 82, and the camera 70, the position of the overpass 50 determined from the image provided by the environment image server 62, and the information obtained from the map server 64. For example, the assistance device 60 determines whether the user 80 and the vehicle 20 are close to each other within a predetermined time, and if it is determined that the user 80 and the vehicle 20 are close to each other within the predetermined time, the assistance device 60 performs traffic assistance for the vehicle 20 and / or the user 80. For example, the assistance device 60 performs traffic assistance by sending alarm information indicating output of an alarm to the vehicle-mounted processing device 40 and the user terminal 82.

[0041] exist Figure 1 In the case shown, since the user terminal 82b moves toward the road 90 in a direction intersecting the traveling direction of the vehicle 20b, the auxiliary device 60 determines that the user terminal 82b and the vehicle 20b are close within a predetermined time, and sends an alarm message to the vehicle 20b and the user terminal 82b. On the other hand, the user terminal 82a moves in a direction intersecting the traveling direction of the vehicle 20a, and it is predicted that the vehicle 20a and the user terminal 82a are close within a predetermined time in the latitude and longitude coordinate system. However, since it is determined that the user terminal 82a is located at the position of the overpass 50 and the height information of the user terminal 82a exceeds the predetermined height, the auxiliary device 60 does not send an alarm message to the vehicle 20a and the user terminal 82a as it is not an object of traffic assistance.

[0042] According to the assistance system 10, the location where the overpass 50 is installed can be determined by a simple method. As a result, pedestrians walking on the overpass 50 can be excluded from the objects of traffic assistance. As a result, useless traffic assistance can be suppressed.

[0043] Figure 2 The following is a functional configuration of the support device 60. The support device 60 includes a control device 200, a communication device 290, and a storage device 280.

[0044] The communication device 290 is responsible for the communication between the vehicle-mounted processing device 40 and the user terminal 82 and the auxiliary device 60, respectively, based on the control of the control device 200. The control device 200 is implemented by a circuit including a calculation processing device including a processor, for example. The control device 200 can be implemented by a microcomputer including a CPU, ROM, RAM, I / O, bus, etc. The storage device 280 is implemented by a non-volatile storage medium. The control device 200 performs processing using information stored in the storage device 280.

[0045] The control device 200 includes an auxiliary control device 202 and a floor object position information output device 204. The floor object position information output device 204 includes an image information acquisition unit 210, a floor object position determination unit 220, and a floor object position information output unit 230. The auxiliary control device 202 includes an information acquisition unit 250, a determination unit 240, and an auxiliary control unit 260. Figure 2 The functional form of a part of the functional structure shown.

[0046] The image information acquisition unit 210 acquires a plurality of images obtained from a plurality of different viewpoints on the road 90, and first information indicating the viewpoint position and the viewing direction of each of the plurality of images. The feature position determination unit 220 determines the position of the feature based on the difference in the appearance of the image of the feature in each of the plurality of images, and the viewpoint position and the viewing direction of each of the plurality of images determined based on the first information. The feature position information output unit 230 outputs information indicating the position of the feature determined by the feature position determination unit 220. The information output by the feature position information output unit 230 may be stored in the storage device 280.

[0047] The feature position identification unit 220 may identify the feature position based on whether each of the plurality of images includes an image of the feature and the viewpoint positions and viewing directions of each of the plurality of images determined based on the first information.

[0048] When the first image among the multiple images includes an image of the first ground object, and the second image among the multiple images does not include an image of the ground object, the ground object position determination unit 220 may determine the range between the viewpoint position of the first image and the viewpoint position of the second image as the position of the ground object. Here, the second image is an image whose viewpoint position is different from that of the first image and whose observation direction differs from that of the first image by less than a predetermined value.

[0049] The ground object position determination unit 220 determines the position of the ground object based on multiple first images in a first observation direction among multiple images, and when a second image in a second observation direction different from the first observation direction contains an image of the ground object, the position of the ground object is corrected based on the viewpoint position of the second image and the position of the ground object determined based on the multiple first images.

[0050] The ground object position determination unit 220 determines a first position as the position of the ground object based on multiple first images in a first observation direction among multiple images, determines a third position as the position of the ground object based on multiple third images in a third observation direction opposite to the first observation direction among multiple images, and determines the position of the ground object based on the first position and the third position.

[0051] In this embodiment, the ground feature is a pedestrian overpass 50. The ground feature may be a vehicle stop line, a pedestrian crossing and / or a road sign, etc.

[0052] In the assistance control device 202, the information acquisition unit 250 acquires second information including the position information of the user terminal 82 and the position information of the vehicle 20 traveling on the road 90. The assistance control unit 260 performs traffic assistance for the vehicle 20 traveling on the road 90 and / or the user associated with the user terminal 82 based on the second information and the information indicating the position of the feature output from the feature position information output unit.

[0053] When the above-ground object is the overpass 50, the determination unit 240 determines whether the position of the user terminal 82 is located at the position of the overpass 50 based on the second information. The assistance control unit 260 determines whether to perform traffic assistance in consideration of the determination result of the determination unit 240. When the determination unit 240 determines that the position of the user terminal 82 is located at the position of the overpass 50, and the height of the user terminal 82 determined based on the position information of the user terminal 82 is greater than a predetermined value, the assistance control unit 260 determines not to perform traffic assistance.

[0054] On the other hand, when the determination unit 240 determines that the position of the user terminal 82 is not at the position of the overpass 50, or when the determination unit 240 determines that the position of the user terminal 82 is at the position of the overpass 50 and the height of the user terminal 82 determined based on the position information of the user terminal 82 is less than a predetermined value, the auxiliary control unit 260 performs control for executing traffic assistance on the condition that the position of the vehicle 20 is determined to be close to the position of the user terminal 82 in the latitude and longitude coordinate system.

[0055] Figure 3 is a diagram for explaining an example of a process of determining the position of the overpass 50 based on an image provided by the environment image server 62 .

[0056] Here, it is assumed that between the viewpoint position P1 and the viewpoint position P3, the images obtained from the viewpoint position P1, the viewpoint position P2, and the viewpoint position P3 are provided from the environment image server 62 in correspondence with the respective viewpoint positions and the viewing directions. That is, it is assumed that the environment image server 62 can provide images obtained from discrete viewpoint positions. Here, it is assumed that the viewing direction of each image is along the driving direction of the vehicle on the road 90.

[0057] When the image information acquisition unit 210 acquires an image obtained from each of the viewpoint positions P1, P2, and P3, the ground feature position determination unit 220 analyzes each image and determines whether each image contains the image of the overpass 50. The ground feature position determination unit 220 determines that the image obtained from the viewpoint positions P1 and P2 contains the image of the overpass 50, and determines that the image obtained from the viewpoint position P3 does not contain the image of the overpass 50. Figure 3 In the figure, a “yes” corresponding to each viewpoint position indicates that the image includes the image of the overpass 50, and a “no” corresponding to each viewpoint position indicates that the image does not include the image of the overpass 50.

[0058] When it is determined that the image of the overpass 50 is included in the image at the viewpoint position P1 and the image at the viewpoint position P2, and the image of the overpass 50 is not included in the image at the viewpoint position P3, the feature position determination unit 220 determines that the overpass 50 exists between the viewpoint position P2 and the viewpoint position P3. Thus, the feature position determination unit 220 determines that the overpass 50 exists within the range 300 along the travel direction in the road 90. The feature position information output unit 230 outputs information indicating the range 300 as information indicating the position of the overpass 50.

[0059] However, in the case where the viewing angle of the image provided by the environment image server 62 is limited, there is a possibility that the image of the overpass 50 is not included in the image obtained from the viewpoint position before passing through the overpass 50. Therefore, the ground feature position determination unit 220 can determine that the overpass 50 exists in a larger range that at least includes the range 300, and the ground feature position information output unit 230 can output information indicating the larger range as information indicating the position of the overpass 50.

[0060] Figure 4 is a diagram for explaining another example of a process of determining the position of the overpass 50 based on an image provided by the environment image server 62 .

[0061] Here, it is assumed that images obtained from viewpoint positions P11, P12, and P13, and images obtained from viewpoint positions P14, P15, and P16 are provided from the environment image server 62 in association with their respective viewpoint positions and viewing directions.

[0062] Here, it is assumed that the observation direction of the images of viewpoint position P11, viewpoint position P12, and viewpoint position P13 is along the first direction in which the vehicle can travel on the road 90. On the other hand, it is assumed that the observation direction of the images of viewpoint position P14, viewpoint position P15, and viewpoint position P16 is along the third direction in which the vehicle can travel on the road 90. The third direction is a direction opposite to the first direction.

[0063] The feature position determination unit 220 analyzes the images obtained from each of the viewpoint position P11, viewpoint position P12, and viewpoint position P13, and determines that the images obtained from the viewpoint position P11 and viewpoint position P12 contain the image of the overpass 50, and determines that the image obtained from the viewpoint position P13 does not contain the image of the overpass 50. Thus, the feature position determination unit 220 determines that the overpass 50 exists between the viewpoint position P12 and the viewpoint position P13, and determines that the overpass 50 exists within the range 400 along the travel direction of the road 90.

[0064] The feature position determination unit 220 analyzes the images obtained from each of the viewpoint positions P14, P15, and P16, and determines that the images obtained from the viewpoint positions P14 and P15 contain the image of the overpass 50, and determines that the image obtained from the viewpoint position P16 does not contain the image of the overpass 50. Thus, the feature position determination unit 220 determines that the overpass 50 exists between the viewpoint positions P15 and P16.

[0065] The feature position identification unit 220 determines that the overpass 50 is present in the range 420 where the range 400 and the range 410 overlap. The feature position information output unit 230 outputs information indicating the range 420 as information indicating the position of the overpass 50.

[0066] As described above, in the case where the viewing angle of the image provided by the environment image server 62 is limited, there is a possibility that the image obtained from the viewpoint position before passing the overpass 50 does not include the image of the overpass 50. Therefore, the ground feature position determination unit 220 can determine that the overpass 50 exists in a wider first range that includes at least the range 400, and at the same time determine that the overpass 50 exists in a wider second range that includes at least the range 410, and determine that the overpass 50 exists in a range where the first range and the second range overlap. In this case, the ground feature position information output unit 230 can output information indicating the range where the first range and the second range overlap as information indicating the position of the overpass 50.

[0067] Figure 5 is a diagram for explaining another example of a process of determining the position of the overpass 50 based on an image provided by the environment image server 62 .

[0068] Here, it is assumed that images obtained from viewpoint positions P21 , P22 , and P23 and an image obtained from viewpoint position P24 are provided from the environment image server 62 in association with the respective viewpoint positions and viewing directions.

[0069] Here, it is assumed that the observation direction of the images of the viewpoint position P21, the viewpoint position P22, and the viewpoint position P23 is along the first direction in which the vehicle can travel on the road 90. On the other hand, it is assumed that the observation direction of the image of the viewpoint position P24 is along the second direction intersecting the first direction in the road 90. As an example, the second direction is a direction orthogonal to the first direction.

[0070] The feature position determination unit 220 analyzes the images obtained from each of the viewpoint position P21, the viewpoint position P22, and the viewpoint position P23, and determines that the image obtained from the viewpoint position P21 includes the image of the overpass 50, and determines that the images obtained from the viewpoint position P22 and the viewpoint position P23 do not include the image of the overpass 50. Thus, the feature position determination unit 220 determines that the overpass 50 exists between the viewpoint position P21 and the viewpoint position P22, and determines that the overpass 50 exists within the range 500 along the traveling direction in the road 90. In this way, when the viewing angle of the image provided by the environment image server 62 is limited, there is a possibility that the image obtained from the viewpoint position before passing the overpass 50, such as the viewpoint position P22, does not include the image of the overpass 50.

[0071] Here, the feature position determination unit 220 analyzes the image at the viewpoint position P24 and determines that the image obtained from the viewpoint position P24 includes the image of the overpass 50. In this case, it can be determined that the overpass 50 exists somewhere along the viewing direction of the viewpoint position P24. Figure 5 In the example shown, it is determined that the image obtained from the viewpoint position P24 includes the image of the overpass 50 on the viewpoint position P22 side relative to the center. Therefore, the feature position identification unit 220 determines that the overpass 50 exists in the range 510 between P21 and P24 in the direction connecting P21 and P22.

[0072] The feature position information output unit 230 outputs information indicating the range 510 as information indicating the position of the overpass 50 .

[0073] Figure 6 An example of a flowchart related to a process related to the method of outputting position information of a feature executed in the support device 60 is shown. The process of this flowchart may be repeatedly executed at predetermined time intervals.

[0074] In S600, the image information acquisition unit 210 acquires images obtained from a plurality of viewpoint positions from the environment image server 62. In the images acquired from the environment image server 62, the viewpoint positions and the viewing directions are associated with each other.

[0075] In S602, the feature position determination unit 220 determines the position of the overpass 50 based on the image of the overpass 50 in the plurality of images and the viewpoint position and observation direction corresponding to each image. Figures 3 to 5 The range of the overpass 50 is determined by the method described in the above description.

[0076] In S604, the feature position information output unit 230 outputs information indicating the position of the overpass 50 determined in S602. For example, the feature position information output unit 230 stores information indicating the range in which the overpass 50 determined in S602 is located in the storage device 280.

[0077] In S606 , the support control device 202 performs control related to traffic support based on the information indicating the position of the overpass 50 output by the feature position information output unit 230 .

[0078] Figure 7 An example of the execution flow of processing performed in the user terminal 82 , the vehicle-mounted processing device 40 , the imaging device 70 , and the support device 60 included in the vehicle 20 is shown.

[0079] In S700, the in-vehicle processing device 40a transmits the position information of the vehicle 20 to the support device 60. The position information is transmitted from the in-vehicle processing device 40a to the support device 60 periodically.

[0080] In S720, the user terminal 82a transmits position information indicating the current position of the user terminal 82a based on the signal received from the GNSS satellite to the support device 60. The position information is transmitted from the user terminal 82a to the support device 60 periodically.

[0081] In S721, the user terminal 82b transmits position information indicating the current position of the user terminal 82b based on the signal received from the GNSS satellite to the support device 60. The position information is transmitted from the user terminal 82b to the support device 60 periodically.

[0082] In S730, the imaging device 70 transmits position information indicating the position of the user 80 and the position of the vehicle 20 obtained by analyzing the image captured by the imaging device 70 to the support device 60. The position information is transmitted from the imaging device 70 to the support device 60 periodically.

[0083] In S714, the auxiliary control unit 260 of the auxiliary device 60 determines whether to perform traffic assistance for the vehicle 20 and the user terminal 82 based on the position information of the vehicle 20 obtained from the on-board processing device 40, the position information of the user terminal 82 obtained from the user terminal 82, the position information of the vehicle 20 and the user terminal 82 obtained from the shooting device 70, the information indicating the position of the overpass 50 output by the ground object position information output device 204, and the map information provided by the map server 64.

[0084] Specifically, the assistance control unit 260 determines whether the vehicle 20 and the user terminal 82 approach to within a predetermined distance within a predetermined time based on the location information of the vehicle 20, the location information of the user terminal 82, and the map information obtained from the map server 64. When it is determined that the vehicle 20 and the user terminal 82 approach to within a predetermined distance within a predetermined time, the assistance control unit 260 determines to perform traffic assistance.

[0085] In this embodiment, the auxiliary control unit 260 determines that the vehicle 20b and the user terminal 82b approach to within a predetermined distance within a predetermined time. On the other hand, for the user terminal 82a, although it can be determined that the user terminal 82a and the vehicle 20a approach to within a predetermined distance within a predetermined time, based on the position information of the user terminal 82a and the information indicating the position of the overpass 50, it is determined that the position of the user terminal 82a in the longitude and latitude coordinate system is within the range where the overpass 50 exists, and the height of the user terminal 82a is higher than the predetermined threshold. Therefore, it is excluded from the processing related to traffic assistance performed by the auxiliary control device 202.

[0086] Thus, in S716, the auxiliary control unit 260 controls the communication device 290 to send auxiliary information indicating that the vehicle-mounted processing device 40b outputs an alarm. In S718, the auxiliary control unit 260 controls the communication device 290 to send auxiliary information indicating that the user terminal 82b outputs an alarm.

[0087] Thus, when the user terminal 82b receives the assistance information from the assistance device 60, in S726, the user terminal 82b notifies the user 80b of the presence of the vehicle 20 approaching the user 80b through the HMI (Human Machine Interface) function of the user terminal 82b. The user terminal 82b can notify the user 80b of the presence of the vehicle 20 approaching the user 80b by sound.

[0088] When the vehicle-mounted processing device 40b receives the auxiliary information from the auxiliary device 60, in S706, the vehicle-mounted processing device 40b notifies the passengers of the vehicle 20b of the presence of a user approaching the vehicle 20b through the HMI function of the vehicle-mounted processing device 40b. The vehicle-mounted processing device 40b can notify the passengers of the vehicle 20b of the presence of a vehicle approaching the vehicle 20b through sound and display on a display device provided in the vehicle 20b.

[0089] Figure 8 An example of a flowchart related to processing related to the assist control method executed by the assist control device 202 is shown. A portion of the processing in this flowchart may be performed in Figure 7 The processing of this flowchart can be performed repeatedly. The processing of this flowchart can be performed for each user terminal 82.

[0090] In S802, the auxiliary control unit 260 determines whether the user terminal 82 is located within the range where the overpass 50 exists. The auxiliary control unit 260 determines whether the user terminal 82 is located within the range where the overpass 50 exists based on the position information of the user terminal 82 and the information indicating the position of the overpass 50 output by the feature position information output device 204. If it is determined that the user terminal 82 is not located within the range where the overpass 50 exists, the process proceeds to S808.

[0091] If the user terminal 82 is determined to be within the range of the overpass 50, the assist control unit 260 determines in S804 whether the height of the user terminal 82 is higher than a predetermined threshold. If the height of the user terminal 82 is lower than the predetermined threshold, the process proceeds to S808.

[0092] When it is determined that the height of the user terminal 82 is higher than the predetermined threshold, in S806 , the user terminal 82 is excluded from the objects of auxiliary determination, and the processing of this flowchart ends.

[0093] In S808, the auxiliary control unit 260 determines whether the vehicle 20 and the user terminal 82 are within a predetermined distance within a predetermined time based on the position of the user terminal 82, the position of the vehicle 20, and the map information obtained from the map server 64. If it is determined that traffic assistance is to be performed for the vehicle 20 and the user terminal 82, control related to traffic assistance is performed in S810.

[0094] In the above description, the case where the ground feature as the object of determining the position in the ground feature position information output device 204 is the overpass 50 is mainly explained. However, the ground feature is not limited to the overpass 50. As the ground feature, a vehicle stop line, a pedestrian crossing, a road sign, a railway line, any building, etc. can be applicable. In the present embodiment, a method of determining the position of a ground feature based on an image obtained from the environment image server 62 is explained, but the image used to determine the position of the ground feature is not limited to the image taken for the environment image server 62. For example, the image used to determine the position of the ground feature can be an image taken by an ADAS camera provided in the vehicle 20, or it can be an image taken by a driving recorder provided on the vehicle 20.

[0095] According to the assistance system 10 described above, the position of the ground feature such as the overpass 50 can be determined by a simple method. As a result, for example, pedestrians walking on the overpass 50 can be excluded from the objects of traffic assistance, so unnecessary traffic assistance can be suppressed. In addition, when a ground feature is newly installed, before the position information of the newly installed ground feature is reflected in the map information provided by the map server 64, the position information of the newly installed ground feature can sometimes be obtained and used for traffic assistance.

[0096] Fig. 9 An example of a computer 2000 that can fully or partially embody multiple embodiments of the present invention is shown. The program installed in the computer 2000 can cause the computer 2000 to function as a device such as the auxiliary device 60 or each unit of the device involved in the embodiment, perform operations associated with the device or each unit of the device, and / or perform a process involved in the embodiment or steps of the process. Such a program can be executed by the CPU 2012 in order to cause the computer 2000 to perform the processing flow described in this specification and specific operations associated with some or all of the functional blocks in the block diagram.

[0097] The computer 2000 according to the present embodiment includes a CPU 2012 and a RAM 2014, which are connected to each other via a main controller 2010. The computer 2000 further includes a ROM 2026, a flash memory 2024, a communication interface 2022, and an input / output chip 2040. The ROM 2026, the flash memory 2024, the communication interface 2022, and the input / output chip 2040 are connected to the main controller 2010 via the input / output controller 2020.

[0098] The CPU 2012 operates according to the programs stored in the ROM 2026 and the RAM 2014 , thereby controlling each unit.

[0099] The communication interface 2022 communicates with other electronic devices via a network. The flash memory 2024 stores programs and data used by the CPU 2012 in the computer 2000. The ROM 2026 stores a startup program executed by the computer 2000 when activated, and / or a program dependent on the hardware of the computer 2000. The input / output chip 2040 can also connect various input / output units such as a keyboard, a mouse, and a monitor to the input / output controller 2020 via an input / output port such as a serial port, a parallel port, a keyboard port, a mouse port, a monitor port, a USB port, and an HDMI (registered trademark) port.

[0100] The program is provided via a computer-readable storage medium such as a CD-ROM, DVD-ROM, or a USB flash drive, or a network. RAM 2014, ROM 2026, or flash memory 2024 is an example of a computer-readable storage medium. The program is installed in flash memory 2024, RAM 2014, or ROM 2026 and executed by CPU 2012. The information processing described in these programs is read by computer 2000, and the cooperation between the program and the various types of hardware resources described above is realized. The device or method can be constructed by implementing the operation or processing of information in accordance with the use of computer 2000.

[0101] For example, when the computer 2000 communicates with an external device, the CPU 2012 may execute a communication program loaded into the RAM 2014, and based on the processing described in the communication program, instruct the communication interface 2022 to perform communication processing. Under the control of the CPU 2012, the communication interface 2022 reads the transmission data stored in the transmission buffer processing area provided in the RAM 2014 and the recording medium such as the flash memory 2024, transmits the read transmission data to the network, and writes the reception data received from the network to the reception buffer processing area provided on the recording medium.

[0102] In addition, the CPU 2012 can read all or a necessary part of a file or database stored in a recording medium such as the flash memory 2024 to the RAM 2014, and execute various processes on the data in the RAM 2014. The CPU 2012 then writes the processed data back to the recording medium.

[0103] Various types of information such as programs, data, tables, and databases can be saved to a recording medium and applied to information processing. CPU2012 can perform various operations, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval / replacement, etc., which are specified by the instruction sequence of the program, on the data read from RAM2014, and write the results back to RAM2014. In addition, CPU2012 can retrieve information in files, databases, etc. in the recording medium. For example, when multiple items having attribute values ​​of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, CPU2012 can retrieve an item that specifies the attribute value of the first attribute and is consistent with the condition from the multiple items, read the attribute value of the second attribute stored in the item, and thereby obtain the attribute value of the second attribute associated with the first attribute that meets the pre-set condition.

[0104] The program or software module described above can be stored in a computer-readable storage medium on or near the computer 2000. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium. The program stored in the computer-readable storage medium can be provided to the computer 2000 via a network.

[0105] The program installed in the computer 2000 and causing the computer 2000 to function as the auxiliary device 60 can be run in the CPU 2012 or the like, thereby causing the computer 2000 to function as each unit of the auxiliary device 60 (e.g., the auxiliary control device 202 and the feature position information output device 204, etc.). The information processing described in these programs is read into the computer 2000, thereby causing the computer 2000 to function as a specific unit in which software and the various hardware resources described above cooperate, that is, each unit of the auxiliary device 60. Furthermore, by using these specific units to realize the calculation or processing of information corresponding to the purpose of use of the computer 2000 in this embodiment, a unique auxiliary device 60 corresponding to the purpose of use is constructed.

[0106] Various embodiments are described with reference to block diagrams, etc. In the block diagram, each functional block may represent (1) a step of a process for performing an operation or (2) each unit of a device having the function of performing an operation. Specific steps and each unit may be implemented by a dedicated circuit, a programmable circuit supplied together with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied together with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc., including memory elements.

[0107] The computer-readable storage medium may include any tangible device capable of storing instructions to be executed by an appropriate device, and as a result, the computer-readable storage medium having the instructions stored therein constitutes at least a part of a product containing instructions executable to implement a unit for performing the operations specified in the process flow or block diagram. As examples of computer-readable storage media, electrical storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. may be included. As more specific examples of computer-readable storage media, floppy disks (registered trademark), floppy disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), electrically erasable programmable read-only memories (EEPROM), static random access memories (SRAM), compact disk read-only memories (CD-ROM), digital versatile disks (DVD), Blu-ray (registered trademark) optical disks, memory sticks, integrated circuit cards, etc. may be included.

[0108] Computer-readable instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine delegate instructions, microcode, firmware instructions, state setting data, or any source code or object code described in any combination of one or more programming languages ​​including object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0109] Computer readable instructions are provided to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device via a local or local area network (LAN), a wide area network (WAN) such as the Internet, and the computer readable instructions can be executed to implement a unit for performing the operations specified in the described processing flow or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0110] The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is obvious from the description of the claims that such changes or improvements can also be included in the technical scope of the present invention.

[0111] Regarding the execution order of each process of actions, processes, steps, and steps in the devices, systems, programs, and methods shown in the claims, specifications, and drawings, it should be noted that there is no special indication of "before", "preceding", etc., and in addition, as long as the output of the previous process is not used in the subsequent process, it can be implemented in any order. For the action flow in the claims, specifications, and drawings, even if "first", "next", etc. are used for convenience, it does not mean that it must be implemented in such an order.

[0112] [Description of Reference Numerals]

[0113] 10Assistance Systems

[0114] 20 vehicles

[0115] 40 On-board processing device

[0116] 50 Overpass

[0117] 60 auxiliary devices

[0118] 62 Environment Image Server

[0119] 64 map servers

[0120] 70 Camera

[0121] 80 users

[0122] 82 user terminals

[0123] 90 Road

[0124] 200 control devices

[0125] 202 Auxiliary control device

[0126] 204 Ground object position information output device

[0127] 210 Image Information Acquisition Department

[0128] 220 Ground object position determination unit

[0129] 230 Ground object position information output unit

[0130] 240 Judgment Department

[0131] 250 Information Acquisition Department

[0132] 260 Auxiliary Control Unit

[0133] 280 storage devices

[0134] 290 communication device

[0135] 300 range

[0136] 400, 410, 420, 500, 510 range

[0137] 2000 Computer

[0138] 2010 Main Controller

[0139] 2012CPU

[0140] 2014 RAM

[0141] 2020 Input / Output Controller

[0142] 2022 Communication Interface

[0143] 2024 Flash

[0144] 2026ROM

[0145] 2040 input / output chips.

Claims

1. A device for outputting information on the position of a ground object, characterized in that: have: an image information acquisition unit that acquires a plurality of images obtained from a plurality of different viewpoints on a road, and first information indicating the viewpoint positions and observation directions of each of the plurality of images; a floor feature position determination unit that determines the position of the floor feature based on the difference in appearance of the image of the floor feature in each of the plurality of images and the viewpoint position and the observation direction of each of the plurality of images determined based on the first information; as well as The planimetric feature position information output unit outputs information indicating the position of the planimetric feature specified by the planimetric feature position specifying unit.

2. The device for outputting information on the position of a ground object according to claim 1, wherein: The feature position identification unit identifies the position of the feature based on whether each of the plurality of images includes an image of the feature and the viewpoint position and the observation direction of each of the plurality of images identified based on the first information.

3. The device for outputting position information of a ground object according to claim 1 or 2, wherein: When a first image among the plurality of images includes an image of a first land object and a second image among the plurality of images does not include the image of the land object, the land object position determination unit determines a range between a viewpoint position of the first image and a viewpoint position of the second image as the position of the land object, The second image is an image having a different viewpoint position from that of the first image and having an observation direction whose difference from that of the first image is less than a predetermined value.

4. The device for outputting position information of a ground object according to claim 1 or 2, wherein: The above-ground object position determination unit performs the following processing: determining the position of the ground object according to a plurality of first images in a first observation direction among the plurality of images, When the image of the feature is included in a second image in a second observation direction different from the first observation direction, the position of the feature is corrected based on the viewpoint position of the second image and the position of the feature determined from the plurality of first images.

5. The device for outputting information on the position of a ground object according to claim 1 or 2, wherein: The above-ground object position determination unit performs the following processing: determining a first position as the position of the ground object according to a plurality of first images in a first observation direction among the plurality of images, determining a third position as the position of the feature based on a plurality of third images in a third observation direction opposite to the first observation direction among the plurality of images, The position of the feature is determined based on the first position and the third position.

6. The device for outputting position information of a ground object according to claim 1 or 2, wherein: The ground features include pedestrian overpasses, vehicle stop lines, pedestrian crossings and / or road signs.

7. An auxiliary control device, characterized in that: have: The device for outputting information on the position of a ground object according to any one of claims 1 to 6; an information acquisition unit that acquires second information including location information of a user terminal and location information of a vehicle traveling on the road; as well as An assistance control unit performs traffic assistance for a vehicle traveling on the road and / or a user associated with the user terminal based on the second information and the information indicating the position of the feature output from the feature position information output unit.

8. The auxiliary control device according to claim 7, wherein: The above-ground feature is a pedestrian overpass. further comprising a determination unit for determining whether the user terminal is located at the overpass based on the second information, The assistance control unit determines whether to perform the traffic assistance in consideration of a determination result of the determination unit.

9. The auxiliary control device according to claim 8, wherein: When the determination unit determines that the user terminal is located at the overpass and the height of the user terminal determined based on the location information of the user terminal is above a predetermined value, the assistance control unit determines not to perform the traffic assistance.

10. The auxiliary control device according to claim 8, wherein: When the determination unit determines that the user terminal is not located at the overpass, or when the determination unit determines that the user terminal is located at the overpass and the height of the user terminal determined based on the location information of the user terminal is less than a predetermined value, the auxiliary control unit performs control for executing the traffic assistance on the condition that the position of the vehicle is determined to be close to the position of the user terminal in the latitude and longitude coordinate system.

11. A method for outputting location information of ground objects, characterized in that: have: A step of acquiring a plurality of images obtained from a plurality of different viewpoints on a road, and first information indicating the viewpoint positions and viewing directions of each of the plurality of images; A step of determining the position of the ground feature based on the difference in the appearance of the image of the ground feature in each of the plurality of images and the viewpoint position and the observation direction of each of the plurality of images determined according to the first information; as well as The step of outputting information indicating the position of the feature.

12. A computer-readable storage medium, characterized in that: A program for causing a computer to function as the ground object position information output device according to any one of claims 1 to 6 is stored.

Images