Information processing system, information processing method, information processing program, and data structure

Abstract

To objectively and efficiently acquire information related to safety of a sidewalk.SOLUTION: An information processing system performs: acquiring an image which indicates a road constituted by a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle (S1, S2); acquiring vehicle position information which indicates a position of the vehicle and photographing direction information which indicates a photographing direction of the camera (S1, S2); determining sidewalk position information which indicates a position of the sidewalk based on at least the vehicle position information and the photographing direction information (S3); determining a safety level of the sidewalk based on the image and an image recognition model (S6); and generating sidewalk data including sidewalk safety level information which indicates the safety level of the sidewalk and the sidewalk position information (S9).SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, an information processing program, and a data structure.

Background Art

[0002] There is known a commuting route safety support system capable of visualizing the safety and dangerous locations of sidewalks on the way to school on a map (see, for example, Non-Patent Document 1). Each guardian or the like can register the location of a child's home, the commuting route, and dangerous locations around the commuting route through the commuting route safety support system disclosed in Non-Patent Document 1. In this way, guardians who use this system can share the safety of sidewalks on the way to school and dangerous locations.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, on the system disclosed in Non-Patent Document 1, each user registers information regarding the safety of sidewalks on the way to school based on their own subjectivity, so the objectivity of the information is not sufficiently guaranteed. In addition, since each user registers the information through manual input onto the system, there is a problem in that the input work of each user becomes a burden. Furthermore, when the number of users using the system is not large, it is realistically difficult to comprehensively provide information regarding the safety of sidewalks on the way to school and to update the information regularly.

[0005] In view of the above viewpoints, an object of the present disclosure is to provide an information processing system, an information processing method, and an information processing program that enable objective and efficient acquisition of information regarding the safety of sidewalks. Further, the present disclosure provides an information processing method and an information processing program capable of clearly grasping the safety of sidewalks and dangerous locations on a map, and also provides an information processing method and an information processing program capable of performing route search considering the safety of sidewalks. Furthermore, an object of the present disclosure is to provide a data structure for realizing these information processing methods and information processing programs.

Means for Solving the Problems

[0006] An information processing system according to an aspect of the present disclosure acquires an image showing a road composed of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle, acquires vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera, determines sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information, determines the safety level of the sidewalk based on the image and an image recognition model, and generates sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0007] An information processing method according to an aspect of the present disclosure is executed by a computer and includes steps of acquiring an image showing a road composed of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle, acquiring vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera, determining sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information, determining the safety level of the sidewalk based on the image and an image recognition model, and generating sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0008] An information processing method according to an aspect of the present disclosure is executed by a computer and includes a step of visualizing a safety level of a sidewalk on a map based on sidewalk data. The sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk and sidewalk position information indicating the position of the sidewalk.

[0009] An information processing method according to an aspect of the present disclosure is executed by a computer and includes a step of performing route search considering the safety level of a sidewalk based on sidewalk data and road network data. The sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, information regarding the position of the sidewalk, and identification information of a link associated with a road composed of the sidewalk and a lane adjacent to the sidewalk. The road network data is associated with the sidewalk data via the identification information of the link.

[0010] Also, an information processing program for causing a computer to execute the information processing method is provided.

[0011] A data structure according to an aspect of the present disclosure includes sidewalk position information indicating the position of a sidewalk and sidewalk safety level information indicating the safety level of the sidewalk. The sidewalk position information is determined based on vehicle position information indicating the position of a vehicle and imaging direction information indicating the imaging direction of a camera mounted on the vehicle. The sidewalk safety level information is determined based on an image captured by the camera and showing a road composed of a lane and the sidewalk adjacent to the lane, and an image recognition model.

Advantages of the Invention

[0012] According to the present disclosure, it is possible to provide an information processing system, an information processing method, and an information processing program that enable objective and efficient acquisition of information regarding the safety of sidewalks. Further, according to the present disclosure, it is possible to provide an information processing method and an information processing program that enable clear understanding of the safety of sidewalks and dangerous locations on a map, and it is also possible to provide an information processing method and an information processing program that enable route search considering the safety of sidewalks. Furthermore, according to the present disclosure, it is possible to provide a data structure for realizing these information processing methods and information processing programs.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0014] (Overview of This Embodiment) The overview of this embodiment is shown below.

[0015] An information processing system according to an aspect of the present disclosure acquires an image showing a road composed of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle, acquires vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera, determines sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information, determines the safety level of the sidewalk based on the image and an image recognition model, and generates sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0016] According to the above configuration, it is possible to acquire sidewalk safety level information indicating the safety level of a sidewalk based on an image captured by a camera mounted on a vehicle and an image recognition model. Thus, an information processing system can be provided that enables objective and efficient acquisition of information regarding the safety of sidewalks. For example, a user can clearly grasp the safety of sidewalks and dangerous locations on a map through the safety level of sidewalks visualized on the map. Furthermore, the user can perform route search to a destination considering the safety of sidewalks.

[0017] In addition, the information processing system may visualize the safety level of the sidewalk on a map based on the sidewalk data.

[0018] According to the above configuration, the user can clearly grasp the safety and dangerous points of the sidewalk (especially the safety and dangerous points of the sidewalk on the way to school) through the safety level of the sidewalk visualized on the map.

[0019] In addition, the information processing system may determine the coloring of the plot points corresponding to the sidewalk position information according to the sidewalk safety level information, and visualize the colored plot points on the map.

[0020] According to the above configuration, the user can clearly grasp the safety and dangerous points of the sidewalk by looking at the plot points displayed on the map.

[0021] In addition, the information processing system may determine the safety level of the sidewalk from the viewpoints of the presence or absence of a protective fence installed on the sidewalk, the presence or absence of a step between the sidewalk and the roadway, and the presence or absence of a boundary line between the sidewalk and the roadway.

[0022] According to the above configuration, it is possible to objectively determine the safety level of the sidewalk from the viewpoints of the presence or absence of a protective fence installed on the sidewalk, the presence or absence of a step between the sidewalk and the roadway, and the presence or absence of a boundary line between the sidewalk and the roadway.

[0023] In addition, the information processing system may acquire attribute information of the protective fence based on the image. The sidewalk data may include the attribute information of the protective fence.

[0024] According to the above configuration, it is possible to evaluate the safety of the sidewalk in more detail based on the attribute information of the protective fence.

[0025] In addition, the information processing system may acquire external environment information regarding the external environment around the road based on the image. The sidewalk data may include the external environment information.

[0026] According to the above configuration, it becomes possible to evaluate the safety of the sidewalk in more detail based on external environment information (for example, the presence or absence of an evacuation place adjacent to the sidewalk, etc.).

[0027] Further, the information processing system may specify the sidewalk position information based on the vehicle position information, the imaging direction information, the field angle information of the camera, and the map data indicating the road.

[0028] According to the above configuration, since it becomes possible to more accurately specify the sidewalk position information, it is possible to acquire information regarding the safety of the sidewalk with higher accuracy.

[0029] Further, the information processing system may specify the identification information of the link associated with the road based on the vehicle position information. The link may constitute road network data for route search. The sidewalk data may include the identification information of the link. The sidewalk data may be associated with the road network data via the identification information of the link.

[0030] According to the above configuration, since the sidewalk data is associated with the road network data via the identification information of the link, it becomes possible to execute route search considering the safety level of the sidewalk by utilizing both the sidewalk data and the road network data.

[0031] Further, the information processing system may determine the in-link sidewalk position information indicating the relative position of the sidewalk on the link based on the information indicating the start point position and the end point position of the link and the sidewalk position information. The sidewalk data may include the in-link sidewalk position information.

[0032] According to the above configuration, since the sidewalk data includes the identification information of the link and the in-link sidewalk position information, it becomes possible to execute route search considering the safety level of the sidewalk by utilizing both the sidewalk data and the road network data.

[0033] Further, the information processing system may execute route search considering the safety level of the sidewalk based on the sidewalk data and the road network data.

[0034] According to the above configuration, it is possible to contribute to a reduction in the number of traffic accidents caused by the danger of the sidewalk through route search considering the safety level of the sidewalk.

[0035] An information processing method according to an aspect of the present disclosure is executed by a computer, and includes steps of: acquiring an image showing a road composed of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle; acquiring vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera; determining sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information; determining the safety level of the sidewalk based on the image and an image recognition model; and generating sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0036] According to the above configuration, it is possible to acquire sidewalk safety level information indicating the safety level of the sidewalk based on an image captured by a camera mounted on a vehicle and an image recognition model. In this way, it is possible to provide an information processing method that enables objective and efficient acquisition of information regarding the safety of sidewalks. For example, a user can clearly grasp the safety of sidewalks and dangerous locations on a map through the safety level of sidewalks visualized on the map. Furthermore, the user can perform route search to a destination considering the safety of the sidewalk.

[0037] An information processing method according to an aspect of the present disclosure is executed by a computer and includes a step of visualizing the safety level of a sidewalk on a map based on sidewalk data. The sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk and sidewalk position information indicating the position of the sidewalk.

[0038] According to the above method, a user can clearly understand the safety of a sidewalk and dangerous locations (especially the safety and dangerous locations of the sidewalk on the way to school) through the safety level of the sidewalk visualized on a map.

[0039] An information processing method according to an aspect of the present disclosure is executed by a computer and includes a step of performing route search considering the safety level of a sidewalk based on sidewalk data and road network data. The sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, information regarding the position of the sidewalk, and identification information of a link associated with a road including the sidewalk and a lane adjacent to the sidewalk. The road network data is associated with the sidewalk data via the identification information of the link.

[0040] According to the above method, a user (for example, a pedestrian, a driver of a vehicle, etc.) can perform route search to a destination considering the safety of the sidewalk. In particular, a pedestrian can perform route search to a destination considering the safety of the sidewalk on a map application installed on a mobile terminal. A driver of a vehicle can perform route search to a destination considering the safety of the sidewalk through a navigation system mounted on the vehicle. Furthermore, an autonomous driving vehicle (autonomous driving robot) that autonomously travels from a departure place to a destination in an autonomous driving mode can perform route search to a destination considering the safety of the sidewalk.

[0041] Also provided is an information processing program that causes a computer to execute the information processing method.

[0042] A data structure according to an aspect of the present disclosure is determined based on identification information of an image taken by a camera mounted on a vehicle and showing a road including a lane and a sidewalk adjacent to the lane, vehicle position information indicating the position of the vehicle, and shooting direction information indicating the shooting direction of the camera, and includes sidewalk position information indicating the position of the sidewalk and sidewalk safety level information indicating the safety level of the sidewalk determined based on the image and an image recognition model.

[0043] Further, the data structure may be determined based on the vehicle position information and may further include identification information of a link associated with the road.

[0044] (Information processing system according to this embodiment) Hereinafter, the information processing system 1 according to this embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of the information processing system 1 according to this embodiment. As shown in FIG. 1, the information processing system 1 includes a vehicle 2, a server 3, a user terminal 4, and a vehicle 6. These are connected to a communication network 5. The vehicles 2 and 6 and the user terminal 4 are communicably connected to the server 3 via the communication network 5. The communication network 5 is constituted by, for example, the Internet or the like.

[0045] Next, the hardware configuration of the vehicle 2 will be described below with reference to FIG. 2. FIG. 2 is a diagram showing an example of the configuration of the vehicle 2 according to this embodiment. As shown in FIG. 2, the vehicle 2 includes a vehicle control unit 20, a camera 21, a wireless communication unit 22, a GPS (Global Positioning System) receiver 23, an HMI (Human Machine Interface) 24, a storage device 25, a drive system 26, and a direction sensor 27.

[0046] The vehicle 2 may be a vehicle (automobile) capable of traveling in an automatic driving mode. In this example, a four-wheel automobile is given as an example of the vehicle, but the number of wheels of the vehicle 2 is not particularly limited. The vehicle control unit 20 is configured to control various components provided in the vehicle 2 and is constituted by, for example, at least one electronic control unit (ECU: Electronic Control Unit). The electronic control unit includes a computer system including one or more processors and one or more memories.

[0047] The camera 21 is configured to image the surrounding environment in front of the vehicle 2. The camera 21 is disposed at a predetermined position of the vehicle 2 so as to image the surrounding environment in front of the vehicle 2 through, for example, the windshield of the vehicle 2. The image captured by the camera 21 may be a still image or a moving image. The frame rate of the moving image is not particularly limited. The image shows the road in front of the vehicle 2. Here, the road is composed of a lane and a sidewalk adjacent to the lane. It should be noted that in this specification, the "sidewalk" does not necessarily mean the sidewalk defined in the road traffic laws of each country. Further, the camera 21 may be detachably mounted on the vehicle 2. In this regard, the camera 21 may be portable by a driver or the like.

[0048] The wireless communication unit 22 is configured to connect the vehicle 2 to the communication network 5, and includes a transmission / reception antenna and a wireless transmission / reception circuit. The wireless communication unit 22 may be a wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be a wireless communication module compatible with a fourth-generation mobile communication system such as LTE or a fifth-generation mobile communication system.

[0049] The GPS receiver 23 is configured to acquire information regarding the current position of the vehicle 2. The HMI 24 is composed of an input unit that receives an input operation from the driver and an output unit that outputs information regarding the running of the vehicle 2 to the driver. The storage device 25 is an external storage device such as a hard disk drive (HDD) or a solid state drive (SSD). Map data and vehicle control programs may be stored in the storage device 25. The drive system 26 is configured to control the running state of the vehicle 2. For example, the drive system 26 is configured to control the accelerator, brake, and steering of the vehicle 2 respectively.

[0050] The azimuth sensor 27 is configured to detect the traveling direction of the vehicle 2. In this example, it is assumed that the shooting direction of the camera 21 coincides with the traveling direction of the vehicle 2. For this reason, the information regarding the traveling direction of the vehicle 2 detected by the azimuth sensor 27 becomes the information regarding the shooting direction of the camera 21. The vehicle 2 transmits, via the communication network 5, image data, vehicle position data, shooting direction data, and the angle of view information of the camera 21 to the server 3. The image data includes a plurality of images and a plurality of time information. Each of the plurality of images is associated with one of the plurality of time information. Each of the plurality of time information indicates the shooting time (shooting date and time) of the corresponding image.

[0051] The vehicle position data includes a plurality of vehicle position information and a plurality of time information. The vehicle position information is information indicating the position (longitude and latitude) of the vehicle 2. The longitude is displayed within the range of -180 degrees to +180 degrees. The latitude is displayed within the range of -90 degrees to +90 degrees. Each of the plurality of vehicle position information is associated with one of the plurality of time information. Each of the plurality of time information indicates the acquisition time of the corresponding vehicle position.

[0052] The shooting direction data includes information regarding the shooting directions (traveling direction of the vehicle 2) of the plurality of cameras 21 and a plurality of time information. The shooting direction of the camera 21 is indicated within the range of 0 degrees to 360 degrees. In this case, 0 degrees indicates the north direction, 90 degrees indicates the east direction, 180 degrees indicates the south direction, and 270 degrees indicates the west direction. Each of the information regarding the plurality of shooting directions is associated with one of the plurality of time information. Each of the plurality of time information indicates the detection time of the corresponding shooting direction.

[0053] Next, with reference to FIG. 3, the hardware configuration of the server 3 will be described below. FIG. 3 is a diagram showing an example of the configuration of the server 3 according to the present embodiment. As shown in FIG. 3, the server 3 includes a control unit 30, a storage device 31, an input / output interface 32, a communication unit 33, an input operation unit 34, and a display unit 35. These elements are connected to a communication bus 36.

[0054] The control unit 30 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory is composed of a ROM (Read Only Memory) storing various programs and the like, a RAM (Random Access Memory) having a plurality of work areas storing various programs and the like executed by the processor, and the like. The processor is composed of, for example, at least one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The CPU may be composed of a plurality of CPU cores. The GPU may be composed of a plurality of GPU cores. The processor may be configured to expand a program specified from various programs incorporated in the storage device 31 or the ROM onto the RAM and execute various processes in cooperation with the RAM. The memory may store an information processing program for causing the processor to execute a series of processes (information processing method) executed by the server 3 in FIGS. 4, 11, and 14.

[0055] The storage device 31 is, for example, a storage device (storage) such as an HDD or an SSD, and is configured to store programs and various data. The storage device 31 stores sidewalk data (described later) and map data. The map data includes road network data for route search, background data, annotation data, address data, and store information data.

[0056] As shown in FIG. 10, the road network data is composed of a plurality of links (lines) and a plurality of nodes (points). The nodes are connected to a plurality of links. Each node is connected to another node via one link. Each link is assigned unique identification information (ID). Each link is associated with, for example, road attribute information and road regulation information (e.g., one-way traffic, etc.). The road attribute information includes, for example, road type information, route cost information, number of lanes information, and width information. Each node is assigned unique identification information (ID). Each node is associated with, for example, lane information, direction guidance information, traffic signal information, and intersection name information. Thus, the road network data has information related to links and information related to nodes.

[0057] The background data includes illustration data regarding the visual background of the map (e.g., map shapes of buildings, sea, mountains, forests, roads, etc.). The annotation data includes character information to be marked on the map (e.g., names of buildings and mountains, etc.). The address data includes address information associated with each building on the map. The store information data includes information regarding stores on the map (opening hours, closing hours, etc.).

[0058] Returning to FIG. 3, the input / output interface 32 is an interface that enables connection between an external device and the server 3, and includes an interface according to a predetermined communication standard such as the USB standard or the HDMI (registered trademark) standard. The communication unit 33 may include various wired communication modules for communicating with an external terminal on the communication network 5. The input operation unit 34 is, for example, a touch panel, a mouse, and / or a keyboard, etc., configured to receive an operator's input operation and generate an operation signal according to the operator's input operation. The display unit 35 is composed of, for example, a video display and a video display circuit.

[0059] Returning to FIG. 1, the user terminal 4 is the terminal operated by the user U. The user terminal 4 is communicably connected to the server 3 via the communication network 5 and receives map information from the server 3. The user terminal 4 includes a control unit having a processor and a memory, a storage device, a wireless communication unit, an input operation unit, a GPS receiver configured to acquire the current position of the user terminal 4, and a display unit. A map application or a web browser for displaying a map may be installed in the user terminal 4.

[0060] The vehicle 6 may be a vehicle capable of traveling in an autonomous driving mode. The vehicle 6 may include, for example, a vehicle control unit, a sensing device such as a camera, a millimeter-wave radar, a LiDAR unit, a wireless communication unit, a GPS receiver, an HMI, a storage device for storing map data and the like, and a drive system. The vehicle 6 may further include a car navigation system.

[0061] (Generation process of sidewalk data) Next, with reference to FIG. 4, a series of processes for generating sidewalk data (an example of a data structure) according to the present embodiment will be described below. FIG. 4 is a flowchart for explaining a series of processes for generating sidewalk data according to the present embodiment. As shown in FIG. 5, an example of the information constituting the sidewalk data includes the shooting date and time of the image (the shooting time of the image), the image ID, the sidewalk position information (longitude, latitude), the link ID (the identification information of the link), the in-link sidewalk position information, the shooting direction information, the shooting area information (left-side shooting area or right-side shooting area), the sidewalk safety level information, the recognition score information, the evacuation space information, and the attribute information of the protection fence.

[0062] In step S1, the server 3 acquires image data, vehicle position data, shooting direction data, and camera field of view information from the vehicle 2 via the communication network 5. As described above, the image data is data composed of a plurality of images and a plurality of time information associated therewith. Each image shall show the road in front of the vehicle 2 (for example, refer to FIG. 6). The vehicle position data is data composed of a plurality of vehicle position information and a plurality of time information associated therewith. The shooting direction data is data composed of a plurality of shooting direction information and a plurality of time information associated therewith. The camera field of view information is information indicating the field of view of the camera 21 of the vehicle 2.

[0063] In step S2, the server 3 acquires an image, vehicle position information, and shooting direction information associated with each other from the image data, vehicle position data, and shooting direction data received from the vehicle 2. Here, since each of the image data, vehicle position data, and shooting direction data has time information, the server 3 associates each of the image, vehicle position information, and shooting direction information through common time information. For example, the server 3 associates the image Mt1 at time t1, the vehicle position Pt1 at time t1, and the shooting direction Dt1 at time t1 with each other. In this example, the server 3 extracts one image from the plurality of images included in the image data, and then extracts the vehicle position information and shooting direction information associated with the extracted one image from the vehicle position data and the shooting direction data.

[0064] In step S3, the server 3 acquires sidewalk position information indicating the position of the sidewalk based on the vehicle position information and the shooting direction information. More specifically, the server 3 determines the sidewalk position information corresponding to the vehicle position information based on the vehicle position information, the shooting direction information, the camera field of view information, and the map data indicating the road. In this regard, with reference to FIG. 7, the process of determining the sidewalk position information will be described below. FIG. 7 is a diagram for explaining the process of determining the sidewalk position information.

[0065] As shown in FIG. 7, assume that the position of the vehicle 2 traveling on the road lane 70 is P. In this case, the position P1 of the left sidewalk 71 adjacent to the left side of the road lane 70 is defined as the intersection point where the boundary line between the road lane 70 and the left sidewalk 71 (more specifically, the boundary area between the outer road line 72 and the left sidewalk 71) intersects with the left end VL of the camera angle V of the camera 21.

[0066] The server 3 acquires map information around the position P from the map data based on the vehicle position information indicating the position P of the vehicle 2. Here, the map information around the position P shall include position information regarding the road lane 70, the left sidewalk 71, and the outer road line 72. Next, the server 3 determines the position of the camera angle V of the camera 21 on the map shown in FIG. 7 (in other words, the position of the shooting area of the camera 21) based on the shooting direction information and the camera angle information. In the example shown in FIG. 7, since the vehicle 2 is moving in the west direction, the shooting direction of the camera 21 is the west direction (270 degrees), and the camera angle V of the camera 21 extends in the west direction from the vehicle 2. For example, in this example, when the vehicle 2 is moving in the east direction, since the shooting direction of the camera 21 is the east direction (90 degrees), the camera angle V of the camera 21 extends in the east direction from the vehicle 2. Thus, if the shooting direction of the camera 21 is not known, the position of the camera angle V cannot be determined, and therefore the sidewalk position information cannot be accurately acquired.

[0067] Next, the server 3 calculates the position (longitude, latitude) of the intersection point where the boundary line between the road lane 70 and the left sidewalk 71 (more specifically, the boundary area between the outer road line 72 and the left sidewalk 71) intersects with the left end VL of the camera angle V of the camera 21 based on the position information regarding the road lane 70, the left sidewalk 71, and the outer road line 72 and the positional relationship of the camera angle V of the camera 21. In this way, the server 3 determines the calculated position information of the intersection point as the sidewalk position information indicating the position of the left sidewalk 71.

[0068] Similarly, the position P2 of the right sidewalk 73 adjacent to the right side of the road lane 70 is defined as the intersection point where the boundary line between the road lane 70 and the right sidewalk 73 (more specifically, the boundary area between the outer road line 74 and the right sidewalk 73) intersects with the right end VR of the camera angle V of the camera 21.

[0069] Server 3 acquires position information regarding the roadway 70, the right sidewalk 73, and the outside lane line 74 from the map data. Next, based on the position information regarding the roadway 70, the right sidewalk 73, and the outside lane line 74 and the positional relationship with the angular field V of the camera 21, Server 3 calculates the position (longitude, latitude) of the intersection where the boundary line between the roadway 70 and the right sidewalk 73 (more specifically, the boundary area between the outside lane line 74 and the right sidewalk 73) intersects with the right end VR of the angular field V of the camera 21. In this way, Server 3 determines the calculated position information of the intersection as sidewalk position information indicating the position of the right sidewalk 73.

[0070] In this way, in step S3, two pieces of sidewalk position information (sidewalk position information indicating the position of the left sidewalk 71 and sidewalk position information indicating the position of the right sidewalk 73) corresponding to the vehicle position information are acquired. Note that in this example, the intersection of the boundary line between the sidewalk and the roadway and the end of the angular field V of the camera is defined as the position of the sidewalk, but the method for determining the sidewalk position in this embodiment is not limited to this. Also, when the right sidewalk 73 is determined as a median strip instead of a sidewalk from the map data, the sidewalk position information of the right sidewalk may not be acquired.

[0071] Next, in step S4, Server 3 identifies the link ID, which is the identification information of the link associated with the road including the vehicle position information, based on the vehicle position information. Each link constituting the road network data shown in FIG. 10 has one or more pieces of position information. Specifically, each link has the position of the starting point, the position of the ending point, and the positions of one or more constituent points existing between the starting point and the ending point. In this way, Server 3 identifies the link most relevant to the vehicle position information from the position information associated with each link. In particular, Server 3 identifies the link having position information that matches the vehicle position information or the link having position information closest to the vehicle position information, and then acquires the ID of the identified link. The link ID is, for example, an integer value.

[0072] Next, in step S5, the server 3 determines link-in sidewalk position information indicating the relative position of the sidewalk on the link based on the start point position and end point position of the link specified in step S4 and the sidewalk position information. The link-in sidewalk position information is displayed within the range of 0 to 1. In this regard, the link-in sidewalk position is calculated by (the distance between the start point position of the link and the sidewalk position) ÷ (the distance between the start point position and end point position of the link (the total link length)). For example, when the sidewalk position is located at the midpoint of the entire link, the link-in sidewalk position information is 0.5. Also, when the sidewalk position coincides with the end point position of the link, the link-in sidewalk position information is 1.0.

[0073] Next, in step S6, the server 3 determines the safety level of the sidewalk based on the image and the image recognition model 130 (see FIG. 9). In particular, the server 3 may determine the safety level of the left sidewalk existing in the image and the safety level of the right sidewalk existing in the image based on one image and the image recognition model. Specifically, as shown in FIG. 6, in the image captured by the camera 21 of the vehicle 2, there are two sidewalks (left sidewalk and right sidewalk) sandwiching the roadway. Therefore, the server 3 determines the safety level of the left sidewalk existing in the left capture area S L within one image and determines the safety level of the right sidewalk existing in the right capture area S R within one image. In this regard, the server 3 extracts the images of the two capture areas S L , S R from each image, and then inputs the extracted images of each capture area S L , S R into the input layer of the image recognition model, thereby determining the safety level of the sidewalk in the images of each capture area S L , S R .

[0074] As shown in FIG. 9, the pixel values of the image indicating the road are input into the input layer of the image recognition model 130. The image input into the input layer is the left capture area S L indicating the left sidewalk and the right capture area S RIt may be an image including only one of them, or an image including both the left shooting area S L and the right shooting area S R is also possible. In the output layer of the image recognition model 130, the probability of each safety level of the sidewalk is output. In this embodiment, in the output layer of the image recognition model 130, the probabilities of safety levels 4, 3, 2, 1, and 0 of the sidewalk are output.

[0075] In this regard, when an image of the left shooting area S L or the right shooting area S R is input to the image recognition model 130, the server 3 determines the safety level of the left sidewalk or the right sidewalk. Further, when an image including both the left shooting area S L and the right shooting area S R is input to the image recognition model 130, the server 3 determines both the safety level of the left sidewalk and the safety level of the right sidewalk.

[0076] The image recognition model 130 is a learning model constructed by machine learning. The image recognition model 130 is constructed by various learning images showing sidewalks with safety level 4, various learning images showing sidewalks with safety level 3, various learning images showing sidewalks with safety level 2, various learning images showing sidewalks with safety level 1, and various learning images showing safety level 0 (classification determination of the safety level of the sidewalk is impossible) (for example, images in which no sidewalk such as a median strip is shown). The image recognition model 130 is stored in the storage device 31 of the server 3, and the control unit 30 can classify the safety level of the sidewalk displayed on the image by using the image recognition model 130.

[0077] Referring to FIG. 8, each safety level of the sidewalk will be described below. FIG. 8(a) is an example of an image showing a sidewalk with safety level 4. FIG. 8(b) is an example of an image showing a sidewalk with safety level 3. FIG. 8(c) is an example of an image showing a sidewalk with safety level 2. FIG. 8(d) is an example of an image showing a sidewalk with safety level 1. As shown in FIG. 8, for the safety level of the sidewalk, level 4 has the highest safety, and level 1 has the lowest safety. That is, as going from level 4 to 1, the safety of the sidewalk decreases (in other words, the danger of the sidewalk increases).

[0078] As shown in FIG. 8(a), for the safety level 4 of the sidewalk, there is a step between the sidewalk and the roadway, and a protective fence for pedestrian protection is provided on the sidewalk. An image showing such a sidewalk is recognized as safety level 4 (the safety of the sidewalk is high) by the image recognition model 130. In other words, the probability of the unit of safety level 4 in the output layer of the image recognition model 130 is the highest. As shown in FIG. 8(b), for the safety level 3 of the sidewalk, there is a step between the sidewalk and the roadway, while a protective fence for pedestrian protection is not provided on the sidewalk. An image showing such a sidewalk is recognized as safety level 3 (the safety of the sidewalk is slightly high) by the image recognition model 130. That is, the probability of the unit of safety level 3 in the output layer of the image recognition model 130 is the highest.

[0079] As shown in FIG. 8(c), for the safety level 2 of the sidewalk, there is no step or protective fence between the sidewalk and the roadway, but an outside lane line indicating the boundary of the roadway is provided. An image showing such a sidewalk is recognized as safety level 2 (the danger of the sidewalk is slightly high) by the image recognition model 130. That is, the probability of the unit of safety level 2 in the output layer of the image recognition model 130 is the highest. As shown in FIG. 8(d), for the safety level 1 of the sidewalk, there is no step or protective fence between the sidewalk and the roadway, and no outside lane line indicating the boundary of the roadway is provided either. An image showing such a sidewalk is recognized as safety level 1 (the danger of the sidewalk is high) by the image recognition model 130. That is, the probability of the unit of safety level 1 in the output layer of the image recognition model 130 is the highest.

[0080] Furthermore, when the safety level of the sidewalk displayed on the image cannot be classified into any of safety levels 1 to 4 by the image recognition model 130, it is classified as safety level 0. In this case, the probability of the unit in the output layer associated with safety level 0 becomes the highest. For example, when there is no road in the image captured by the camera 21 mounted on the vehicle 2 (for example, when the image shows a wall of a house, etc.), it is impossible for the image recognition model 130 to classify the safety level of the sidewalk. Similarly, when an object that is not a sidewalk, such as a median strip of a two-lane road, is displayed in the image, it is also impossible for the image recognition model 130 to classify the safety level of the sidewalk. Considering such situations, a unit of safety level 0 indicating that the classification of the safety level is impossible is provided as a classification discriminant unit in the output layer of the image recognition model 130. Thus, since the unit of safety level 0 is provided in the output layer of the image recognition model 130, it is possible to further improve the classification accuracy of each safety level.

[0081] In this embodiment, the server 3 determines the safety level of the sidewalk displayed on the image based on the image and the image recognition model 130, and then inputs the sidewalk safety level information and the recognition score information into the sidewalk data. Here, the recognition score is the probability of the determined safety level, more specifically, the value of the probability of the output layer unit associated with the determined safety level.

[0082] Furthermore, in this example, the safety or danger of the sidewalk as the safety level of the sidewalk is classified into four safety levels (five safety levels when safety level 0 is added), but the number of classifications of the safety level is not particularly limited. Also, in this example, as a classification method of the safety level, the safety level of the sidewalk is objectively evaluated from the viewpoints of the presence or absence of a protective fence installed on the sidewalk, the presence or absence of a step between the sidewalk and the roadway, and the presence or absence of a boundary line (outer lane line) between the sidewalk and the roadway, but the evaluation viewpoints are not limited to these items.

[0083] Next, returning to FIG. 4, in step S7, the server 3 acquires external environment information regarding the external environment around the road based on the image and the external environment recognition model. Similarly, the external environment recognition model is constructed by various learning images in which the external environment (for example, various evacuation spaces) is displayed. In particular, the server 3 determines whether there is an evacuation space (avoidance space) where pedestrians can evacuate from dangerous vehicles around the road as the external environment information. In this regard, when an open space, a park, a parking lot, etc. are adjacent to the road (sidewalk), the server 3 determines that an evacuation space exists around the road. On the other hand, when the block wall of a house is adjacent to the road (sidewalk), the server 3 determines that an evacuation space does not exist around the road. Thus, since the presence or absence of an evacuation space (an example of external environment information) is one of the factors contributing to the safety of the sidewalk, the external environment information may be included in the sidewalk data in the present embodiment.

[0084] In step S8, the server 3 acquires the attribute information of the protection fence for pedestrians based on the image and the attribute information recognition model. In particular, the server 3 can estimate the material, strength, and height of the protection fence as the attribute information of the protection fence. For example, the server 3 may classify the material of the protection fence into any one of wood, iron, and concrete. Further, the server 3 can estimate the overall strength (N / cm) of the protection fence based on the shape information of the protection fence and the material information of the protection fence. Also, the server 3 can estimate the height of the protection fence (the height of the protection fence from the surface of the sidewalk) based on the vehicle position information and the protection fence displayed in the image. Thus, since the server 3 determines that the attribute information of the protection fence is one of the factors contributing to the safety of the sidewalk, the attribute information of the protection fence may be included in the sidewalk data in the present embodiment.

[0085] In step S9, the server 3 updates the sidewalk data shown in FIG. 5 based on the various types of information acquired in steps S1 to S8. As shown in FIG. 5, the sidewalk data includes information regarding the shooting time, image ID, sidewalk position, link ID, in-link sidewalk position, shooting direction, shooting area, sidewalk safety level, recognition score (probability), evacuation space, and protective fence. In this regard, the image ID may be generated based on the hash value of the image. Also, when two sidewalks, a left-side sidewalk and a right-side sidewalk, are displayed in one image, the above information is generated for the left-side sidewalk in the left-side shooting area S L and the above information is generated for the right-side sidewalk in the right-side shooting area S R . That is, the sidewalk position, in-link sidewalk position, sidewalk safety level, and recognition score are generated for the left-side sidewalk, and the sidewalk position, in-link sidewalk position, sidewalk safety level, and recognition score are generated for the right-side sidewalk.

[0086] According to the present embodiment, it is possible to acquire sidewalk safety level information indicating the safety level of a sidewalk based on the image captured by the camera 21 mounted on the vehicle 2 and the image recognition model 130. Thus, it is possible to provide the information processing system 1 that enables objective and efficient acquisition of information regarding the safety of sidewalks. Furthermore, it is possible to more precisely evaluate the safety of sidewalks based on information regarding the presence or absence of an evacuation space and attribute information of a protective fence. In this regard, when the safety level of the sidewalk is 1 and there is no evacuation space around the road, it is possible to objectively recognize that the risk of the sidewalk is high. Also, when the safety level of the sidewalk is 4, there is an evacuation space, and the height and strength of the protective fence are sufficiently high, it is possible to objectively recognize that the safety of the sidewalk is high.

[0087] In addition, according to the present embodiment, since the sidewalk position information corresponding to the vehicle position information is determined based on the vehicle position information, the shooting direction information, the camera field angle information, and the map data indicating the road, it is possible to more accurately identify the sidewalk position. In this way, information regarding the safety of the sidewalk can be acquired with higher accuracy. In particular, as will be described later, it is useful to improve the accuracy of the sidewalk position information when visualizing the safety level of the sidewalk as plot points on the map.

[0088] (Visualization process of sidewalk safety level) Next, with reference to FIG. 11, the process of visualizing the safety level of the sidewalk on the map will be described below. FIG. 11 is a flowchart for explaining the process of visualizing the safety level of the sidewalk on the map. As a prerequisite for this process, it is assumed that the server 3 has the sidewalk data and the map data shown in FIG. 5 in the storage device 31. Also, in this example, it is assumed that a map on which the safety level of the sidewalk is visualized is displayed on the user terminal 4 that is communicably connected to the server 3 through the communication network 5 (see FIG. 1). It is assumed that a map display application or browser is installed on the user terminal 4.

[0089] As shown in FIG. 11, in step S10, in response to an input operation by the user U, the user terminal 4 transmits the position information of the user terminal 4 to the server 3 through the communication network 5. Thereafter, the server 3 receives the position information of the user terminal 4. In step S11, the server 3 extracts the map information around the current position of the user terminal 4 from the map data. In step S12, the server 3 extracts the information related to the sidewalk existing within the map (coordinate area) around the current position of the user terminal 4 from the sidewalk data. Here, the information related to the sidewalk is the sidewalk position information and the sidewalk safety level information.

[0090] In step S13, the server 3 determines the coloring of the plot points corresponding to the sidewalk positions according to the sidewalk safety level information. For example, when the sidewalk safety level associated with the sidewalk position P1 is 4, the server 3 determines the coloring of the plot points plotted at the sidewalk position P1 according to the sidewalk safety level 4. The coloring of the plot points may be different from each other according to the sidewalk safety level. For example, the coloring of the plot points in the case of sidewalk safety level 1 may be the first color C1, the coloring of the plot points in the case of sidewalk safety level 2 may be the second color C2, the coloring of the plot points in the case of sidewalk safety level 3 may be the third color C3, and the coloring of the plot points in the case of sidewalk safety level 4 may be the fourth color C4. Here, the colors C1 to C4 may be different from each other. For example, the color C4 may be green, the color C3 may be yellow, the color C2 may be orange, and the color C1 may be red.

[0091] Next, in step S14, after the server 3 generates map image data in which the plot points corresponding to each sidewalk position are visualized, the server 3 transmits the map image data to the user terminal 4 through the communication network 5. Thereafter, a map in which the plot points corresponding to each sidewalk position are visualized is displayed on the display unit of the user terminal 4.

[0092] FIG. 12 is a schematic diagram showing an example of a map 140 in which the safety level of the sidewalk is visualized. As shown in FIG. 12, a large number of plot points K corresponding to the sidewalk positions are visualized along both ends of the road (i.e., the left sidewalk and the right sidewalk) shown in the map 140. Further, since the coloring of the large number of plot points K is determined according to the safety level of the sidewalk, the safety of the sidewalk is visualized on the map 140. In this way, the user U can intuitively grasp the safety of the sidewalk and dangerous locations (especially the safety and dangerous locations of the sidewalk on the way to school) by looking at the plot points K displayed on the map 140.

[0093] Also, in FIG. 12, there may be a defective area H of plotted points. For example, when the safety level of the sidewalk is 0 (in other words, when the safety level of the sidewalk cannot be classified), the plotted points associated with the safety level 0 may not be displayed on the map 140. Further, FIG. 13 is a diagram showing another example of a map 150 in which the safety level of the sidewalk is visualized. In FIG. 13, a wide-area map 150 covering the southern region of Tokyo is shown. A large number of plotted points indicating the safety of the sidewalk may be visualized in a partially designated area within the map 150.

[0094] In addition, the map in which the safety level of the sidewalk is visualized may be displayed on the display unit of the navigation system of the vehicle 6 (see FIG. 1). In this case, the sidewalk data and the map data stored in the server 3 may be periodically distributed to the vehicle 6 via the communication network 5. When the sidewalk data and the map data are stored in the storage device of the vehicle 6, the control unit of the navigation system mounted on the vehicle 6 may display a map in which the safety of the sidewalk is visualized on the display unit of the navigation system by executing each process shown in steps S10 to S15 shown in FIG. 11.

[0095] Also, in this example, the plotted points indicating the safety of the sidewalk are visualized on the map. In addition to this, the attribute information of the protective fence and / or the external environment information around the road (for example, evacuation space information) may be visualized on the map.

[0096] (Route search process considering the safety level of the sidewalk) Next, with reference to FIG. 14, the route search process considering the safety level of the sidewalk will be described below. FIG. 14 is a flowchart for explaining the route search process considering the safety level of the sidewalk. As a prerequisite for this process, it is assumed that the server 3 has the sidewalk data and the map data shown in FIG. 5 in the storage device 31. Also, in this example, it is assumed that the route search process result is displayed on the user terminal 4 that is communicably connected to the server 3 through the communication network 5. It is assumed that a map display application or a browser is installed on the user terminal 4.

[0097] As shown in FIG. 14, in step S20, in response to an input operation by user U, user terminal 4 transmits current location information (location information of the current location) and destination information (location information of the destination) to server 3 through communication network 5. Thereafter, server 3 receives the current location information and the destination information. In step S21, server 3 executes a route search process from the current location to the destination considering the route cost of each link and the sidewalk safety level based on the road network data and sidewalk data of the map data. Since the map data includes link IDs, the road network data composed of a plurality of links and a plurality of nodes is associated with the sidewalk data via the link IDs. Further, each link in the road network data has route cost information (for example, distance cost). In the sidewalk data, each sidewalk safety level information is associated with each link. Therefore, by using both the road network data and the map data, it is possible to execute a route search process from the current location to the destination considering both the route cost and the sidewalk safety level. Also, when a plurality of sidewalk safety levels are associated with one link, the average value of the sidewalk safety levels belonging to the one link may be considered in the route search process. For example, if 10 sidewalk safety levels are associated with a predetermined link and the average value of the 10 sidewalk safety levels belonging to the predetermined link is 3.5, the average value 3.5 of the sidewalk safety levels may be considered as the sidewalk safety level of the predetermined link in the route search process.

[0098] Server 3 may extract only the links associated with the average value of the sidewalk safety levels equal to or higher than a predetermined level, and then determine an optimal route from the perspective of the route cost based on the extracted links. Also, server 3 may extract a plurality of candidate routes through the route search process from the perspective of the route cost of each link, and then select the route with the highest average value of the sidewalk safety levels from among the plurality of candidate routes. In this way, a route search process considering both the route cost and the sidewalk safety level is executed.

[0099] In addition, based on the path cost of each link and the average value of the sidewalk safety level, the server 3 may calculate the corrected path cost considering the sidewalk safety level for each link, and then execute a path search from the departure point to the destination based on the corrected path cost of each link. As the path search method, a conventional path search algorithm such as Dijkstra's algorithm or A* algorithm may be used.

[0100] In step S22, the server 3 transmits the path search result to the user terminal 4 via the communication network 5. After that, the path search result is displayed on the display unit of the user terminal 4. According to this embodiment, since the sidewalk data is associated with the road network data via the link ID, it is possible to execute a path search considering the safety level of the sidewalk by utilizing both the sidewalk data and the road network data. Therefore, the user U can perform a path search to the destination considering the safety of the sidewalk.

[0101] Note that the path search result may be displayed on the display unit of the navigation system of the vehicle 6. In this case, the sidewalk data and the map data stored in the server 3 may be periodically distributed to the vehicle 6 via the communication network 5. When the sidewalk data and the map data are stored in the storage device of the vehicle 6, the control unit of the navigation system mounted on the vehicle 6 can execute the path search process from the current location to the destination considering the safety of the sidewalk by executing each process shown in steps S20 to S23 shown in FIG. 14. Note that the vehicle 6 may be an autonomous driving vehicle (autonomous driving robot) that autonomously travels from the departure point to the destination in the autonomous driving mode.

[0102] As described above, the embodiments of the present invention have been described, but the technical scope of the present invention should not be construed in a limited manner by the description of this embodiment. This embodiment is an example, and it is understood by those skilled in the art that various modifications of the embodiments are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalent scope. [Explanation of symbols]

[0103] 1: Information processing system 2: Vehicle 3: Server 4: User terminal 5: Communication network 6: Vehicle 20: Vehicle control unit 21: Camera 22: Wireless communication unit 23: GPS receiver 24:HMI 25: Storage device 26: Drive system 27: Orientation sensor 30: Control unit 31: Storage device 32: Input / output interface 33: Communications Department 34: Input operation section 35: Display section 70: Roadway 71: Left sidewalk 72: Outside lane 73: Right sidewalk 74: Outside lane 130: Image recognition model 140,150:Map S L : Left side shooting area S R :Right side shooting area VL: Left end VR: Right end H: Defect area K: plot point U:User V: Angle of view

Claims

1. An image showing a road consisting of a lane and a sidewalk adjacent to the lane is acquired from a camera mounted on a vehicle, vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera are acquired, sidewalk position information indicating the position of the sidewalk is determined based on at least the vehicle position information and the imaging direction information, the safety level of the sidewalk is determined based on the image and an image recognition model, sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information is generated. An information processing system.

2. The information processing system visualizes the safety level of the sidewalk on a map based on the sidewalk data. The information processing system according to claim 1.

3. The information processing system determines the coloring of plot points corresponding to the sidewalk position information according to the sidewalk safety level information, and visualizes the colored plot points on the map. The information processing system according to claim 2.

4. The information processing system determines the safety level of the sidewalk from the viewpoints of the presence or absence of a protective fence installed on the sidewalk, the presence or absence of a step between the sidewalk and the lane, and the presence or absence of a boundary line between the sidewalk and the lane. The information processing system according to claim 1.

5. The information processing system acquires attribute information of the protective fence based on the image, and the sidewalk data includes the attribute information of the protective fence. The information processing system according to claim 4.

6. The information processing system acquires external environment information regarding the external environment around the road based on the image, and the sidewalk data includes the external environment information. The information processing system according to claim 1.

7. The information processing system identifies the sidewalk position information based on the vehicle position information, the imaging direction information, the field of view angle information of the camera, and map data showing the road. The information processing system according to claim 1.

8. The information processing system identifies identification information of a link associated with the road based on the vehicle position information, the link constitutes road network data for route search, the sidewalk data includes the identification information of the link, and the sidewalk data is associated with the road network data via the identification information of the link. The information processing system according to claim 1.

9. The information processing system Based on the information indicating the start position and end position of the link and the sidewalk position information, determine the in-link sidewalk position information indicating the relative position of the sidewalk on the link. The sidewalk data includes the in-link sidewalk position information. The information processing system according to claim 8.

10. The information processing system executes route search considering the safety level of the sidewalk based on the sidewalk data and the road network data. The information processing system according to claim 8 or 9.

11. Obtaining an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle; Obtaining vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera; Determining sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information; Determining the safety level of the sidewalk based on the image and an image recognition model; Generating sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information; An information processing method executed by a computer, including:

12. An information processing method executed by a computer, including visualizing the safety level of a sidewalk on a map based on sidewalk data, wherein the sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, and sidewalk position information indicating the position of the sidewalk. An information processing method.

13. An information processing method executed by a computer, including executing route search considering the safety level of a sidewalk based on sidewalk data and road network data, wherein the sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, information regarding the position of the sidewalk, and identification information of a link associated with a road consisting of the sidewalk and a roadway adjacent to the sidewalk, and the road network data is associated with the sidewalk data via the identification information of the link. An information processing method.

14. An information processing program for causing a computer to execute the information processing method according to any one of claims 11 to 13.

15. A data structure including sidewalk position information indicating the position of a sidewalk, and sidewalk safety level information indicating the safety level of the sidewalk. An information processing method. The sidewalk position information is determined based on vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of a camera mounted on the vehicle. The sidewalk safety level information is determined based on an image captured by the camera and showing a road composed of a roadway and the sidewalk adjacent to the roadway, and an image recognition model. Data structure. **Claim 16** The data structure is determined based on the vehicle position information and further includes identification information of a link associated with the road. The data structure according to claim 15.

Images