Driving environment decision device, vehicle, driving environment decision procedure and storage medium

Abstract

A driving environment decision device (100) comprises an image acquisition unit (105), an analysis unit (106), and a decision unit (107). The image acquisition unit (105) captures a photographic image, which is to be taken by a photographic device installed in a vehicle. The analysis unit (106) performs analysis processing with respect to the photographed image and determines a first area, which is an area associated with the sky in the photographed image. Based on the first area, the decision unit (107) decides whether the vehicle is inside a structure.

Description

Technical area

[0001] The present invention relates to a driving environment decision device, a vehicle, a driving environment decision method and a storage medium. Background technology

[0002] A vehicle described in Patent Document 1 includes a camera and a detection device for detecting a tunnel. The camera described in Patent Document 1 is capable of photographing at least one image region around the vehicle and is arranged at the front of the vehicle. The image region includes a plurality of pixels. The tunnel detection device described in Patent Document 1 is configured to determine an average luminance of at least one image region. The tunnel detection device includes a feature detection device and is capable of detecting a difference in luminance between pixels that differ from each other using the device. Therefore, the tunnel detection device is capable of detecting a feature characterized by a drastic change in luminance in at least one image region.

[0003] An automatic light system described in Patent Document 2 is a system installed in a vehicle and configured to detect a driving environment (mainly a scene ahead) of the vehicle while driving and automatically control the turning on and off of a light based on the detected image. In the automatic light system, an image with a wider range (angle of view) in a vertical direction is captured by a camera using a dedicated wide-angle lens. Furthermore, an image inside a vehicle interior is captured at a lower region within a captured image by installing a prism at a lower portion of a windshield inside the vehicle interior.This allows an image captured by the camera to become an image in a sufficiently wide range from an area (an area where a structure and the like are very unlikely to be captured) in front of the vehicle and high in the sky as a higher position, to an area inside the vehicle interior as a lower position.

[0004] Then, four areas are defined within the captured image: a front and upper brightness decision area, a front and mid-air brightness decision area, a front and far brightness decision area, and a vehicle interior brightness decision area. For each decision area, a decision is made about brightness and darkness, and based on the decision result, a decision is made about how a vehicle light should be controlled.

[0005] An object candidate area decision device described in Patent Document 3 detects, in an input image photographed by a camera, an area where a specific object may be present as an object candidate area. The object candidate area decision device includes a reference pattern storage unit, a background area division unit, a decision area cutting unit, a reference pattern selection unit, and a detection unit.

[0006] The reference pattern storage unit described in Patent Document 3 stores, with respect to a background of an input image, a plurality of reference patterns, each different for a road area and a non-road area. The background area division unit described in Patent Document 3 divides an input image currently photographed by a camera into a road area and a non-road area with respect to a background. The decision area cutting unit described in Patent Document 3 cuts out a decision area from an input image currently photographed by a camera.

[0007] The reference pattern selection unit described in Patent Document 3 selects a reference pattern from a plurality of reference patterns according to a background area depending on which of the background areas, which are a road area and a non-road area, a region to be decided is cut out. The detection unit described in Patent Document 3 detects an object candidate region from a region to be decided by comparing a selected reference pattern with the region to be decided.

[0008] Non-Patent Document 1 describes region detection (also referred to as region division, segmentation, etc.) as one of the image recognition techniques. Region detection is a technique for estimating a type of an object represented in a region for each region included in an image by setting the image as an input. Related DocumentsPatent Documents Patent Document 1: Japanese Patent Application (Translation of PCT Application) No. 2014-517388 Patent Document 2: Japanese Patent Application No. 2009-255722 Patent Document 3: Japanese Patent Application No. 2007-328630 Non-patent document

[0009] Non-Patent Document 1: S. Shoulder et al., “Learning to Look around Objects for Top-View Representations of Outdoor Scenes,” ECCV, 2018, pp. 787 to 802 Brief description of the inventionTechnical task

[0010] In the technique described in Patent Document 1, a tunnel is detected based on the difference in the luminance of a pixel. However, if the presence of a vehicle in a tunnel is detected based only on the luminance of a pixel, the presence of the vehicle in the tunnel may be erroneously detected due to the different shooting environments of a camera.

[0011] In the technique described in Patent Document 2, the presence of a vehicle in a tunnel is detected based on the brightness of each judgment area within a captured image. However, since the brightness of a judgment area is a value to be detected based on a luminance of a pixel, the technique described in Patent Document 2, similar to the technique described in Patent Document 1, may also erroneously detect the presence of a vehicle in a tunnel. In addition, the technique described in Patent Document 2 requires a camera with a dedicated wide-angle lens, and detecting that a vehicle is in a tunnel may become difficult with an image captured by a camera with a general viewing angle.

[0012] Patent Document 3 describes that an image currently captured by a camera is divided into a road area and a non-road area with respect to the background. Furthermore, Non-Patent Document 1 describes an example of area detection. However, neither Patent Document 3 nor Non-Patent Document 1 discloses a technique for determining whether a vehicle is inside a structure such as a tunnel.

[0013] An example of an object of the present invention, in view of the above-described problem, is to provide a driving environment decision device, a vehicle, a driving environment decision method, and a storage medium that achieve the object of correctly deciding whether a vehicle is inside a structure. Solution to the problem

[0014] According to one aspect of the present invention, there is provided a driving environment decision device comprising: an image capturing unit that captures a photographed image to be taken by photographing by a photographing device installed in a vehicle; an analysis unit that performs analysis processing on the photographed image and determines a first area, which is an area associated with the sky in the photographed image; and a decision unit that decides whether the vehicle is within a structure based on the first area.

[0015] According to one aspect of the present invention, there is provided a vehicle comprising: the driving environment decision device described above, and the photographing device installed in the vehicle that creates the photographed image by taking photographs.

[0016] According to one aspect of the present invention, there is provided a driving environment decision method comprising: the steps performed by a computer: Acquiring a photographed image to be taken by photographing by a photographing device installed in a vehicle; performing analysis processing on the photographed image and determining a first area which is an area associated with the sky in the photographed image; and Decide whether the vehicle is inside a structure based on the first area.

[0017] According to one aspect of the present invention, there is provided a storage medium storing a program that causes a computer to perform the steps of: Acquiring a photographed image to be taken by photographing by a photographing device installed in a vehicle; performing analysis processing on the photographed image and determining a first area which is an area associated with the sky in the photographed image; and Decide whether the vehicle is inside a structure based on the first area. Advantageous effects of the invention

[0018] According to one aspect of the present invention, it becomes possible to provide a driving environment decision device, a vehicle, a driving environment decision method, and a storage medium that achieve the object of correctly deciding whether a vehicle is inside a structure. Short description of the drawings Fig. 1 is a diagram showing an overview of a driving environment decision device according to an exemplary embodiment 1; Fig. 2 is a diagram showing an overview of a vehicle according to the exemplary embodiment 1; Fig. 3 is a flowchart showing an overview of driving environment decision processing according to the exemplary embodiment 1; Fig. 4 is a diagram showing a detailed configuration example of the vehicle according to Example Embodiment 1; Fig. 5 is a diagram showing a physical configuration example of the driving environment decision device according to Example Embodiment 1; Fig. 6 is a flowchart showing an example of the driving environment decision processing according to the exemplary embodiment 1; Fig. Fig. 7 is a diagram showing a first example of a photographed image; Fig. Fig. 8 is a diagram showing a second example of a photographed image; Fig. 9 is a diagram showing an example of a functional configuration of a driving environment decision device according to an exemplary embodiment 2; Fig. 10 is a flowchart showing an example of driving environment decision processing according to the exemplary embodiment 2; and Fig. 11 is a flowchart showing a detailed example of the decision processing according to Example Embodiment 2. Exemplary embodiments

[0019] Exemplary embodiments according to the present invention will be described below using the drawings. It should be noted that throughout the drawings, a similar component is designated by a similar reference numeral, and its description will be omitted where appropriate. Exemplary embodiment 1 (overview)

[0020] Fig. 1 is a diagram illustrating an overview of a driving environment decision device 100 according to an exemplary embodiment 1. The driving environment decision device 100 includes an image acquisition unit 105, an analysis unit 106, and a decision unit 107.

[0021] The image acquisition unit 105 acquires a photographed image to be captured by a photographing device installed in a vehicle. The analysis unit 106 performs analysis processing on the photographed image and determines a first area, which is an area associated with the sky in the photographed image. The decision unit 107 decides whether the vehicle is located within a structure based on the first area.

[0022] According to the driving environment decision device 100, it becomes possible to correctly decide whether a vehicle is inside a structure.

[0023] Fig. 2 is a diagram illustrating an overview of a vehicle 120 according to Example 1. The vehicle 120 includes the driving environment decision device 100 and a photographing device 121. The photographing device 121 is installed in a vehicle and generates a photographed image by photographing.

[0024] According to the vehicle 120, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0025] Fig. 3 is a flowchart showing an overview of the driving environment decision processing according to Example Embodiment 1.

[0026] The image acquisition unit 105 acquires a photographed image to be captured by the photography device 121 installed in the vehicle 120 (step S101). The analysis unit 106 performs analysis processing on the photographed image and determines a first area, which is an area associated with the sky in the photographed image (step S102). The decision unit 107 decides whether the vehicle 120 is located within a structure based on the first area (step S103).

[0027] According to the driving environment decision method, it becomes possible to correctly decide whether a vehicle is inside a structure. Details

[0028] A detailed example of the vehicle 120 according to the exemplary embodiment 1 will be described below.

[0029] Fig. 4 is a diagram illustrating a detailed configuration example of the vehicle 120 according to the present exemplary embodiment. The vehicle 120 is, for example, a standard car, a truck, a bus, or the like. Note that the vehicle 120 may be a motorcycle, a bicycle, or the like.

[0030] The vehicle 120 according to the present exemplary embodiment includes the photography device 121, a vehicle control device 122, and the driving environment decision device 100. The photography device 121, the vehicle control device 122, and the driving environment decision device 100 are connected to each other in such a way that they are capable of mutually transmitting and receiving information via a communication line configured by a wired connection, a wireless connection, or a combination thereof.

[0031] The photography device 121 is, for example, a camera or a terminal device such as a smartphone with a photography function. The photography device 121 is installed in the vehicle 120 such that it can photograph the surroundings of the vehicle 120. Fig. 4 shows an example in which the photographing device 121 is installed to photograph a front area of ​​the vehicle 120. The photographing device 121 according to the present exemplary embodiment generates a photographed image in which a front area of ​​the vehicle 120 is photographed. The angle of view of the photographing device 121 may be a view angle of a camera generally installed in a vehicle, a camera included in a general terminal, or the like, and is, for example, 80 to 110 degrees. Note that the angle of view of the photographing device 121 is not limited to this.

[0032] It should be noted that the photographing device 121 can be installed such that, in addition to the front area of ​​the vehicle 120, it photographs, for example, a rear or side area of ​​the vehicle 120 and generates a photographed image in which the rear or side area of ​​the vehicle 120 is photographed. Furthermore, Fig. 4 shows an example in which the photographing device 121 is installed in a vehicle interior of the vehicle 120. However, as long as the photographing device 121 is installed in the vehicle 120, an installation location of the photographing device 121 is not limited to a vehicle interior.

[0033] The vehicle control device 122 is an electronic control unit (ECU) or the like. The vehicle control device 122 controls the vehicle 120.

[0034] For example, in a case where the vehicle 120 has an autonomous driving function, the vehicle control device 122 controls the driving of the vehicle 120 using map information, current position information, and the like. The vehicle control device 122 acquires the current position information using, for example, a global positioning system (GPS).

[0035] For example, in a case where the vehicle 120 has an automatic lighting function for controlling a vehicle exterior lamp that illuminates the exterior of the vehicle 120, the vehicle control device 122 controls the turning on and off of the vehicle exterior lamp. The vehicle exterior lamp is, for example, at least one lamp among a headlight HL, a tail lamp (also referred to as a rear lamp, tail lamp, and the like) TL, and a vehicle width lamp (not shown). (Example of a functional configuration for a driving environment decision device 100)

[0036] The driving environment decision device 100 according to the present exemplary embodiment is a device for deciding an environment in which the vehicle 120 is traveling. For example, the driving environment decision device 100 decides, as a driving environment of the vehicle 120, whether the vehicle 120 is in a tunnel on a highway, a regular road, or the like, or whether the vehicle 120 is inside a structure such as a covered parking lot.

[0037] As mentioned above with reference to Fig. 1, the driving environment decision device 100 functionally includes the image acquisition unit 105, the analysis unit 106, and the decision unit 107.

[0038] The image acquisition unit 105 acquires a photographed image from the photographing device 121 via a communication line, which image is generated by photographing by the photographing device 121.

[0039] The analysis unit 106 performs analysis processing with respect to the photographed image acquired by the image acquisition unit 105 and determines various regions included in the photographed image.

[0040] For example, region detection (also referred to as region division, segmentation, or the like), which is one of the techniques for image recognition, can be applied to the analysis processing to be performed by the analysis unit 106 to determine various regions included in a photographed image. Region detection is a technique for estimating a type of an object depicted in a region for each region included in an image by setting the image as an input. As an example of the region detection described above, reference is made to a technique described in Non-Patent Document 1.

[0041] Furthermore, the analysis unit 106 may determine various regions included in a photographed image using a pre-learned learning model through machine learning. In this case, for example, the analysis unit 106 determines various regions using a learning model to which a photographed image is input, and region information for dividing into each region included in the photographed image is output. In this case, during the learning time of the learning model, it is preferable to perform supervised learning in which a photographed image in which an object type is added to each region of the photographed image is set as training data.

[0042] Generally, a road along which the vehicle 120 travels is one among various roads, such as a highway and a general road. For example, while on a general road, a pedestrian sidewalk and a roadside tree are highly likely to be present near the road, on a highway, a pedestrian sidewalk and a roadside tree are less likely to be present on the road. In this way, an object included in a photographed image differs depending on an attribute of a road. Therefore, the analysis unit 106 may include a plurality of pre-learned learning models according to an attribute of a road.

[0043] Then, the analysis unit 106 may determine various areas included in a photographed image, such as a first area, using one of the plurality of learning models according to an attribute of a road along which the vehicle 120 is traveling. In this case, a type of road along which the vehicle 120 is traveling may be detected, for example, by the vehicle control device 122. For example, the vehicle control device 122 may distinguish a road type from position information of the vehicle 120 and map information, distinguish a road type from a traveling speed of the vehicle 120, or distinguish a road type from an analysis result of a photographed image.

[0044] For example, the analysis unit 106 determines a first area using a technique as described above. As described above, the first area is an area associated with the sky in a photographed image.

[0045] Furthermore, for example, an area to be determined by the analysis unit 106 includes a reference area. The reference area is an area above a position associated with a road in a photographed image. In the present exemplary embodiment, the reference area is a first area and a second area, which will be described later, in an area above a position associated with a road in a photographed image.

[0046] The position assigned to a street is, for example, the position of the top of a street in a photographed image. The position of the top of a street in a photographed image is determined, for example, as the position of a vanishing point on a street in the photographed image.

[0047] To derive a vanishing point, the analysis unit 106 determines from a photographed image a line that is related to a line extending parallel to an actual road along which the vehicle 120 is traveling. The line extending parallel to the actual road is, for example, a line associated with both ends of a road, a white line on a road, a yellow line on a road, and the like. Then, the analysis unit 106 derives a vanishing point on a road in the photographed image using the determined line.

[0048] In addition, the analysis unit 106 determines, for example, a second area, which is an area associated with an object other than the sky of a predetermined type in a photographed image.

[0049] The second area includes, for example, an area associated with an obstacle and / or a structure. The obstacle includes another vehicle in the vicinity of vehicle 120. The structure may include a tunnel and / or a covered parking area. The structure may further include a building in the vicinity of a road.

[0050] It should be noted that the second area is not limited to an obstacle and a structure and may, for example, include at least one element under a road, a person, a traffic light, a white line, a yellow line, a street post (street lamp), a motorcycle, a traffic sign, a stop line, a pedestrian crossing, a parking lot (parking space on a roadside), a road marking, a sidewalk, a driveway (vehicle passage on a sidewalk connecting a roadway and a facility or the like), a railway line, a tree, a plant and other elements.

[0051] The decision unit 107 decides whether the vehicle 120 is located within a structure based on a reference area and a first area determined by the analysis unit 106. A structure through which the vehicle 120 moves is typically a tunnel, a building, or a structure such as a covered parking lot, and the like.

[0052] So far, an example of a functional configuration of the driving environment decision device 100 according to Example 1 has been described. Next, a physical configuration example of the driving environment decision device 100 according to Example 1 will be described. (Physical configuration example of the driving environment decision device 100)

[0053] Fig. 5 is a diagram illustrating a physical configuration example of the driving environment decision device 100 according to the present exemplary embodiment. The driving environment decision device 100 physically includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, and a user interface 1060.

[0054] Bus 1010 is a data transmission path along which processor 1020, memory 1030, storage device 1040, network interface 1050, and user interface 1060 mutually transmit and receive data. However, a method for mutually connecting processor 1020 and the like is not limited to a bus connection.

[0055] The processor 1020 is a processor that can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0056] The memory 1030 is a main storage device that can be realized by a random access memory (RAM) or the like.

[0057] The storage device 1040 is an auxiliary storage device that can be implemented by a hard disk drive (HDD), a solid-state drive (SSD), a memory card, a read-only memory (ROM), or the like. The storage device 1040 stores a program module that implements each functional unit of the driving environment decision device 100. Each functional unit associated with a program module is implemented by having the processor 1020 read each program module in the memory 1030 and execute the program module.

[0058] The network interface 1060 is an interface for connecting the driving environment decision device 100 to a communication line.

[0059] The user interface 1050 is, for example, an interface that allows a user to connect a terminal or the like to make various settings and the like with respect to the driving environment decision device 100.

[0060] So far, the examples of the functional and physical configuration of the vehicle 120 according to Example Embodiment 1 have been described.

[0061] An operation example of the vehicle 120 according to Example Embodiment 1 will be described below. (Operation Example of the Vehicle 120 According to Exemplary Embodiment 1)

[0062] Fig. 6 is a flowchart illustrating an example of the driving environment decision processing according to the present exemplary embodiment. The driving environment decision processing is processing for deciding an environment in which the vehicle 120 is traveling. The environment decision processing is repeatedly executed, for example, during the traveling of the vehicle 120. The photographing device 121 photographs a region ahead of the vehicle 120 while the vehicle 120 is traveling and generates a photographed image captured by the photography.

[0063] The image capturing unit 105 captures the photographed image generated in the photographing device 121 (step S101).

[0064] Each of the Fig. 7 and Fig. 8 shows an example of a photographed image produced in the photographing device 121. Fig. 7 shows an example of a photographed image photographed by the photographing device 121 installed in the vehicle 120 while the vehicle is traveling outside a tunnel on a highway. Fig. 8 shows an example of a photographed image photographed by the photographing device 121 installed in the vehicle 120 while the vehicle is traveling within a tunnel on the highway.

[0065] The analysis unit 106 analyzes the photographed image acquired in step S101 (step S102).

[0066] In particular, the analysis unit 106 determines a first area, which is an area associated with the sky in the photographed image (cf. Fig. 7 and Fig. 8).

[0067] Further, the analysis unit 106 determines a reference area in the photographed image. As described above, in the present exemplary embodiment, the reference area is the first area and a second area in an area above a position associated with a road in the photographed image. As described above, the second area is an area associated with an object of a predetermined type other than the sky; however, here, it is assumed that the second area is an obstacle and a structure.

[0068] In a case where a position associated with a street in the photographed image is a vanishing point on a street, as in the example of the Fig. 7 and Fig. 8, the analysis unit 106 determines the first area and the second area in an area above a vanishing point on a road in the photographed image.

[0069] In the Fig. In the example shown in Figure 7, the second area includes a building, which is a structure, and another vehicle, which is an obstacle. Fig. In the example shown in Figure 8, the second area includes a tunnel (inner wall), which is a structure, and another vehicle, which is an obstacle.

[0070] It should be noted that the position of the top of a road is not limited to a vanishing point on a road, but can be determined by any method. For example, the analysis unit 106 may determine an area associated with a road as the second area and determine a point located at an upper side of the photographed image in the area associated with the road as the top of a road (see FIG. Fig. 8).

[0071] The decision unit 107 decides whether the vehicle 120 is located within a structure based on the reference area and the first area determined in step S102 (step S103).

[0072] As the decision method in step S103, an appropriate method may be used.

[0073] For example, the decision unit 107 may decide whether the vehicle 120 is located within a structure based on a proportion occupied by the first area in the reference area. Generally, in a case where the vehicle 120 is located within a structure, a proportion occupied by the first area in the reference area is small compared to a case where the vehicle 120 is located outside the structure. Therefore, for example, in a case where a proportion occupied by the first area in the reference area is less than or equal to a predetermined threshold, the decision unit 107 decides that the vehicle 120 is located within a structure. Furthermore, in a case where a proportion occupied by the first area in the reference area is greater than the predetermined threshold, the decision unit 107 decides that the vehicle 120 is not located within the structure.

[0074] Furthermore, the decision unit 107 may decide, for example, based on whether the surroundings of the first area are surrounded by the second area in the reference area, whether the vehicle 120 is located within a structure. As shown in the Fig. 7 and Fig. As shown in Figure 8, in a case where the vehicle 120 is located within a structure, surroundings of the first area are surrounded by the second area, but in a case where the vehicle 120 is not located within the structure, surroundings of the first area are not surrounded by the second area. Herein, the fact that surroundings of the first area are surrounded by the second area means that the second area is present above the first area and to one side of the first area.

[0075] Therefore, for example, in a case where the surroundings of the first area are surrounded by the second area in the reference area, the decision unit 107 decides that the vehicle 120 is located within a structure. Furthermore, in a case where the surroundings of the first area are not surrounded by the second area in the reference area, the decision unit 107 decides that the vehicle 120 is not located within the structure.

[0076] The vehicle control device 122 controls the vehicle 120 based on a decision result in step S103 (step S104).

[0077] For example, in a case where the vehicle control device 122 controls autonomous driving of the vehicle 120, the vehicle control device 122 switches a control mode between a normal mode and a structure interior mode.

[0078] Normal mode is a control mode intended for use outside a structure. In normal mode, the vehicle control device 122 controls autonomous driving of the vehicle 120 using current position information acquired, for example, using a GPS.

[0079] The structure-inside mode is a control mode for use within a structure. Generally, in a structure such as a tunnel or a covered parking lot, the accuracy of the current position information to be acquired using GPS is reduced due to diffuse reflection of a radio wave or the like. Therefore, in the structure-inside mode, the vehicle control device 122 controls autonomous driving of the vehicle 120 using, for example, current position information acquired using self-motion information such as a travel route, a travel direction, and the like of the vehicle 120, rather than using GPS.

[0080] Furthermore, for example, during daytime or the like, when a vehicle exterior light outside a building is not turned on, the vehicle control device 122 causes the vehicle exterior light to be turned on when the vehicle 120 is inside the structure and causes the vehicle exterior light to be turned off when the vehicle 120 is outside the structure. Specifically, the vehicle control device 122 causes the vehicle exterior light to be turned on during a time when the vehicle 120 is inside a structure. Beneficial effect

[0081] According to the present exemplary embodiment, the driving environment decision device 100 includes the image acquisition unit 105, the analysis unit 106, and the decision unit 107. The image acquisition unit 105 acquires a photographed image to be captured by photographing with a photography device installed in a vehicle. The analysis unit 106 performs analysis processing on the photographed image and determines a first area, which is an area associated with the sky in the photographed image. The decision unit 107 decides whether the vehicle 120 is located inside a structure based on the first area.

[0082] This makes it possible to determine the first area, which is a sky-related area, from the photographed image, and to decide whether the vehicle 120 is located within a structure based on the first area. Therefore, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0083] According to the present exemplary embodiment, the analysis unit 106 further determines a reference area in the photographed image in which a predetermined criterion is met. The decision unit 107 decides, based on the reference area and the first area, whether the vehicle 120 is located within a structure.

[0084] This makes it possible to determine whether the vehicle 120 is located within a structure based on the photographed image, based on the first area being an area associated with the sky, and based on the reference area in which the predetermined criterion is met. Therefore, it becomes possible to correctly determine whether the vehicle 120 is located within a structure.

[0085] According to the present exemplary embodiment, the reference area is an area above a position associated with a road in the photographed image.

[0086] This makes it possible to determine whether the vehicle 120 is located within a structure based on the photographed image, based on the first area being an area associated with the sky, and based on the reference area in which the predetermined criterion is met. Therefore, it becomes possible to correctly determine whether the vehicle 120 is located within a structure.

[0087] The position assigned to a street is a position of a vanishing point on a street in the photographed image.

[0088] This makes it possible to determine whether the vehicle 120 is located inside a structure based on the first area being an area associated with the sky and the reference area being an area above a vanishing point position on a road in the photographed image. Therefore, it becomes possible to correctly determine whether the vehicle 120 is located inside a structure.

[0089] The analysis unit 106 performs analysis processing with respect to the photographed image, determines a second area which is an area associated with an object other than the sky of a predetermined type in the photographed image, and determines, as a reference area, the first area and the second area in an area above a position associated with a road in the photographed image.

[0090] This makes it possible to decide whether the vehicle 120 is located within a structure based on the first area and the reference area based on the first area and the second area. Therefore, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0091] The second area contains an area associated with an obstacle and / or structure.

[0092] This makes it possible to decide whether the vehicle 120 is located within a structure based on the first area and the reference area based on the first area and the second area. Therefore, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0093] The decision unit 107 decides whether the vehicle 120 is located within a structure based on a proportion that the first area occupies in the reference area.

[0094] This makes it possible to decide whether the vehicle 120 is located within a structure based on the first area and the reference area. Therefore, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0095] The decision unit 107 decides whether the vehicle 120 is located within a structure based on whether surroundings of the first area in the reference area are surrounded by the second area.

[0096] This makes it possible to decide whether the vehicle 120 is located within a structure based on the first area and the reference area based on the first area and the second area. Therefore, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0097] The analysis unit 106 determines the first region using a learning model to which the photographed image is input, and outputs region information for dividing into each region included in the photographed image.

[0098] This makes it possible to decide whether the vehicle 120 is located within a structure based on the first area by determining from the photographed image that the first area is an area associated with the sky. Therefore, it becomes possible to correctly decide whether the vehicle 120 is located within a structure.

[0099] A learning model is one of a plurality of learning models according to an attribute of a road. The analysis unit 106 determines the first area using the learning model from the plurality of learning models according to an attribute of a road along which the vehicle 120 is traveling.

[0100] This makes it possible to determine the first area using the learning model according to an attribute of a road along which the vehicle 120 is traveling, making it possible to determine the first area more accurately. Then, based on the correctly determined first area, it is possible to decide whether the vehicle 120 is located within a structure. Therefore, it becomes possible to decide more accurately whether the vehicle 120 is located within a structure. Modification example 1

[0101] In exemplary embodiment 1, an example in which the analysis unit 106 determines a reference range has been described. However, the analysis unit 106 does not need to determine a reference range.

[0102] In the present modification example, the decision unit 107 may decide whether the vehicle 120 is located within a structure based on a proportion occupied by a first area in an entire photographed image and a predetermined threshold. Specifically, for example, the decision unit 107 decides that the vehicle 120 is located within a structure if the proportion occupied by the first area is less than or equal to the threshold, and decides that the vehicle 120 is not located within the structure if the proportion occupied by the first area is greater than the threshold.

[0103] Furthermore, the decision unit 107 may decide whether the vehicle 120 is located within a structure, for example, based on whether surroundings of the first area are surrounded by a second area including a road, a structure, an obstacle, and the like. Specifically, the decision unit 107 decides, for example, that the vehicle 120 is located within a structure when surroundings of the first area are surrounded by the second area in the reference area, and decides that the vehicle 120 is not located within the structure when surroundings of the first area are not surrounded by the second area in the reference area.

[0104] According to the present modification example, it is also possible to judge whether the vehicle 120 is present within a structure based on a first area by judging from a photographed image that the first area is an area associated with the sky. Therefore, it becomes possible to correctly judge whether the vehicle 120 is present within a structure. Modification example 2

[0105] In exemplary embodiment 1, an example in which the analysis unit 106 determines a second area has been described. However, the analysis unit 106 does not need to determine a second area and may determine an area above a position associated with a road in a photographed image as a reference area.

[0106] In the present modification example, for example, similarly to the exemplary embodiment 1, the decision unit 107 may decide whether a vehicle 120 is located inside a structure based on a proportion that the first area occupies in the reference area.

[0107] Furthermore, for example, the decision unit 107 may decide whether the vehicle 120 is located within a structure based on whether surroundings of the first area are surrounded by the reference area. Herein, the fact that surroundings of the first area are surrounded by the reference area means that the second area exists above the first area and to one side of the first area. Specifically, for example, the decision unit 107 may decide that the vehicle 120 is located within a structure in a case where surroundings of the first area are surrounded by the reference area, and decide that the vehicle 120 is not located within the structure in a case where surroundings of the first area are not surrounded by the reference area.

[0108] According to the present modification example, it is also possible to judge whether the vehicle 120 is located inside a structure based on the first area by judging from a photographed image that the first area is an area associated with the sky. Therefore, it becomes possible to correctly judge whether the vehicle 120 is located inside a structure. Exemplary embodiment 2

[0109] In Example 1, an example was described in which a decision as to whether a vehicle 120 is located inside a structure is made using a photographed image. However, the decision as to whether the vehicle 120 is located inside a structure can also be made using a photographed time-series image, specifically, a plurality of photographed images. In the present Example 1, an example is described in which the decision as to whether the vehicle 120 is located inside a structure is made using a photographed time-series image.

[0110] A vehicle according to an exemplary embodiment 2 includes a driving environment decision device 200 according to the exemplary embodiment 1 instead of the driving environment decision device 100 according to the exemplary embodiment 1. Apart from this point, a vehicle according to the present exemplary embodiment may be configured similarly to the vehicle according to the exemplary embodiment 1.

[0111] Fig. 9 is a diagram illustrating a functional configuration example of the driving environment decision device 200 according to the present exemplary embodiment. The driving environment decision device 200 functionally includes an image acquisition unit 205, an analysis unit 206, and a decision unit 207.

[0112] The image acquisition unit 205 acquires a photographed time-series image from a photographing device 121.

[0113] The analysis unit 206 performs analysis processing on each time-series photographed image acquired by the image acquisition unit 205 and determines various regions included in each photographed image. For example, the analysis unit 206 determines a first region, a second region, and a reference region.

[0114] Similar to the decision unit 107 according to the exemplary embodiment 1, the decision unit 207 decides whether a vehicle is located inside a structure based on the reference area and the first area determined by the analysis unit 206.

[0115] As in Fig. As shown in Figure 9, the decision unit 207 according to the present exemplary embodiment includes a first processing unit 207a and a second processing unit 207b.

[0116] The first processing unit 107a decides whether each photographed time-series image is a structure interior region image based on the first region determined in each photographed image by the analysis unit 206. The structure interior region image is an image photographed inside a structure.

[0117] The second processing unit 207b decides whether a vehicle is located within a structure based on a decision result of the first processing unit 207a with respect to each photographed time series image.

[0118] Each component among the image acquisition unit 205, the analysis unit 206, and the decision unit 207 may be configured substantially similarly to the image acquisition unit 105, the analysis unit 106, and the decision unit 107 according to Example Embodiment 1, except for the point described above.

[0119] The driving environment decision device 200 according to the present exemplary embodiment may be physically configured similarly to the driving environment decision device 100 according to the exemplary embodiment 1.

[0120] Fig. 10 is a flowchart illustrating an example of the driving environment decision processing according to the present exemplary embodiment. The driving environment decision processing according to the present exemplary embodiment includes steps S201 to S103 instead of steps S101 to S103 in the driving environment decision processing according to Exemplary Embodiment 1.

[0121] The image acquisition unit 205 acquires a plurality of photographed images generated in the photographing device 121 (step S201).

[0122] The analysis unit 206 analyzes each photographed image acquired in step S201 (step S202). The content of the analysis processing to be performed with respect to each photographed image in step S202 may be similar to the analysis processing to be performed with respect to a photographed image in step S102 according to Example Embodiment 1.

[0123] The decision unit 207 decides whether the vehicle 120 is located within a structure based on a reference area and a first area determined in step S202 (step S203).

[0124] Fig. 11 is a flowchart showing a detailed example of the decision processing (step S203) according to the present exemplary embodiment.

[0125] The first processing unit 207a decides whether each photographed time series image is a structure interior area image based on the first area determined for each photographed image in step S202 (step S203a).

[0126] The second processing unit 207b decides whether the vehicle is located within a structure based on a decision result in step S203 with respect to each photographed time series image (step S203b).

[0127] Specifically, for example, in a case where it is judged that a number of photographed images greater than or equal to a predetermined threshold among a predetermined number of consecutive photographed images are structure interior images, the second processing unit 207b judges that the vehicle is located inside a structure. Furthermore, in a case where it is judged that a number of photographed images less than the threshold among the predetermined number of consecutive photographed images are structure interior images, the second processing unit 207b judges that the vehicle is not located inside the structure.

[0128] For example, in step S203a, it may be erroneously decided whether the photographed image is a structure interior image for some of the predetermined number of photographed images. In the present exemplary embodiment, the decision as to whether the vehicle is inside a structure is made based on whether the number of photographed images determined as a structure interior image among the predetermined number of consecutive photographed images is greater than or equal to the threshold. This enables a correct decision as to whether the vehicle is inside a structure even in a case where an erroneous decision is made regarding some photographed images in step S203a. Beneficial effect

[0129] According to the present exemplary embodiment, the image acquisition unit 205 acquires a photographed time-series image. The analysis unit 206 performs analysis processing on each photographed time-series image and determines a first region in each photographed image.

[0130] The decision unit 207 includes the first processing unit 207a and the second processing unit 207b. The first processing unit 207a decides whether each photographed time-series image is a structure interior image photographed inside a structure based on the first area determined in each photographed image. The second processing unit 207b decides whether the vehicle is inside a structure based on a decision result of the first processing unit 207a regarding each photographed time-series image.

[0131] As described above, this enables a correct decision as to whether a vehicle is located inside a structure, even in a case where an erroneous decision is made regarding some of the photographed images in step S203a. Therefore, it becomes possible to correctly decide whether the vehicle is located inside a structure.

[0132] In the above, exemplary embodiments of the present invention have been described with reference to the drawings, but these are examples of the present invention, and configurations other than the above-mentioned configurations may also be employed.

[0133] Furthermore, in a plurality of flowcharts used in the above description, a plurality of processes (processing steps) are described in a specific sequence. The sequence of execution of the processes to be executed in each exemplary embodiment is not limited to the sequence illustrated in the description. In each exemplary embodiment, the illustrated sequence of processes can be changed within a range that does not adversely affect the content. Furthermore, the exemplary embodiments described above can be combined as long as the contents do not conflict with each other.

[0134] Some or all of the exemplary embodiments described above may also be described as the following supplementary notes, without being limited thereto. 1. A driving environment decision device comprising: an image capturing unit that captures a photographed image to be captured by photographing by a photographing device installed in a vehicle; an analysis unit that performs analysis processing on the photographed image and determines a first area, which is an area associated with the sky in the photographed image; and a decision unit that decides whether the vehicle is within a structure based on the first area. 2. The driving environment decision device according to Supplementary Note 1, wherein the analysis unit further determines a reference area in the photographed image in which a predetermined criterion is met, and the decision unit decides whether the vehicle is within a structure based on the reference area and the first area. 3. The driving environment decision device according to Supplementary Note 2, wherein the reference area is an area above a position associated with a street in the photographed image. 4. The driving environment decision device according to Supplementary Note 3, wherein the position assigned to a street is a position of a vanishing point on a street in the photographed image. 5. The driving environment decision device according to any one of Supplementary Notes 2 to 4, wherein the analysis unit performs analysis processing with respect to the photographed image, determines a second area which is an area associated with an object other than the sky of a predetermined type in the photographed image, and determines, as the reference area, the first area and the second area in an area above a position associated with a road in the photographed image. 6. The driving environment decision device according to Supplementary Note 5, wherein the second area has an area associated with an obstacle and / or a structure. 7. The driving environment decision device according to any one of Supplementary Notes 2 to 6, wherein the decision unit decides whether the vehicle is located within a structure based on a proportion of the first area in the reference area. 8. The driving environment decision device according to Supplementary Note 5 or 6, wherein the decision unit decides whether the vehicle is inside a structure based on whether environments of the first area are surrounded by the second area in the reference area. 9. The driving environment decision device according to any one of Supplementary Notes 1 to 8, wherein the analysis unit determines the first region using a learning model to which the photographed image is input, and outputs region information for dividing into each region included in the photographed image. 10. The driving environment decision device according to Supplementary Note 9, wherein the learning model is one of several learning models according to an attribute of a road, and the analysis unit determines the first area using the learning model among the plurality of learning models according to an attribute of a road along which the vehicle travels. 11. The driving environment decision device according to any one of Supplementary Notes 1 to 10, wherein the image acquisition unit captures the photographed image of a time series, the analysis unit performs analysis processing with respect to each photographed time series image and determines the first area in each photographed image, and the decision-making unit a first processing unit that decides, based on the first area determined in each photographed image, whether each photographed time series image is a structure interior area image photographed within a structure, and a second processing unit that decides whether the vehicle is located within a structure based on a decision result of the first processing unit with respect to each photographed time series image. 12. A vehicle that has: the driving environment decision device according to any one of Supplementary Notes 1 to 11; and the photographing device installed in the vehicle that creates the photographed image by taking photographs. 13. The vehicle as defined in supplementary note 12, further comprising: a vehicle control unit that is installed in the vehicle and controls the vehicle. 14. A driving environment decision-making process comprising the steps performed by a computer: Acquiring a photographed image to be taken by photographing by a photographing device installed in a vehicle; performing analysis processing on the photographed image and determining a first area which is a sky-related area in the photographed image; and Decide whether the vehicle is inside a structure based on the first area. 15. A storage medium that stores a program that causes a computer to perform the steps: Acquiring a photographed image to be taken by photographing by a photographing device installed in a vehicle; performing analysis processing on the photographed image and determining a first area which is a sky-related area in the photographed image; and Decide whether the vehicle is inside a structure based on the first area. 16. A program that causes a computer to perform the steps: Acquiring a photographed image to be captured by photographing by a photographing device installed in a vehicle; Performing analysis processing on the photographed image, and determining a first area which is a sky-related area in the photographed image; and Decide whether the vehicle is inside a structure based on the first area. List of reference symbols 100, 200 Driving environment decision device 105, 205 Image acquisition unit 106, 206 Analysis unit 107, 207 Decision-making unit 120 vehicles 121 Photographic device 122 Vehicle control device 207a First processing unit 207b Second processing unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2014-517388

[0008] JP 2009-255722

[0008] JP 2007-328630

[0008] Cited non-patent literature

[0000] S. Shoulder et al., “Learning to Look around Objects for Top-View Representations of Outdoor Scenes”, ECCV, 2018, pp. 787 to 802

[0009]

Claims

[1] Driving environment decision device, comprising: an image capturing unit that captures a photographed image to be taken by photographing by a photographing device installed in a vehicle; an analysis unit that performs analysis processing on the photographed image and determines a first area, which is an area associated with the sky in the photographed image; and a decision unit that decides whether the vehicle is within a structure based on the first area. [2] A driving environment decision device according to claim 1, wherein the analysis unit further determines a reference area in the photographed image in which a predetermined criterion is met, and the decision unit decides whether the vehicle is within a structure based on the reference area and the first area. [3] A driving environment decision device according to claim 2, wherein the reference area is an area above a position associated with a road in the photographed image. [4] A driving environment decision device according to claim 3, wherein the position associated with a road is a position of a vanishing point on a road in the photographed image. [5] A driving environment decision device according to any one of claims 2 to 4, wherein the analysis unit performs analysis processing with respect to the photographed image, determines a second area which is an area associated with an object other than the sky of a predetermined type in the photographed image, and determines, as the reference area, the first area and the second area in an area above a position associated with a road in the photographed image. [6] The driving environment decision device according to claim 5, wherein the second area comprises an area associated with an obstacle and / or a structure. [7] A driving environment decision device according to any one of claims 2 to 4, wherein the decision unit decides whether the vehicle is inside a structure based on a proportion occupied by the first area in the reference area. [8] The driving environment decision device according to claim 5, wherein the decision unit decides whether the vehicle is inside a structure based on whether surroundings of the first area are surrounded by the second area in the reference area. [9] The driving environment decision device according to any one of claims 1 to 4, wherein the analysis unit determines the first area by using a learning model to which the photographed image is input, and area information for dividing into each area included in the photographed image is output. [10] A driving environment decision device according to claim 9, wherein the learning model is one of several learning models according to an attribute of a road, and the analysis unit determines the first area using the learning model among the plurality of learning models according to an attribute of a road along which the vehicle travels. [11] A driving environment decision device according to any one of claims 1 to 4, wherein the image acquisition unit acquires the photographed image of a time series, the analysis unit performs analysis processing with respect to each photographed time series image and determines the first area in each photographed image, and the decision-making unit has: a first processing unit that decides, based on the first area determined in each photographed image, whether each photographed time series image is a structure interior area image photographed within a structure, and a second processing unit that decides whether the vehicle is located within a structure based on a decision result of the first processing unit with respect to each photographed time series image. [12] Vehicle with: the driving environment decision device according to any one of claims 1 to 4; and the photographing device installed in the vehicle which creates the photographed image by taking photographs. [13] A vehicle according to claim 12, further comprising: a vehicle control unit that is installed in the vehicle and controls the vehicle. [14] Driving environment decision-making process, with the steps carried out by a computer: Acquiring a photographed image to be taken by photographing by a photographing device installed in a vehicle; performing analysis processing on the photographed image and determining a first area which is a sky-related area in the photographed image; and Decide whether the vehicle is inside a structure based on the first area. [15] Storage medium that stores a program that causes a computer to perform the steps: Acquiring a photographed image to be taken by photographing by a photographing device installed in a vehicle; performing analysis processing on the photographed image and determining a first area which is a sky-related area in the photographed image; and Decide whether the vehicle is inside a structure based on the first area.

Images