Information processing system, information processing method, and non-transitory computer-readable medium

Abstract

This information processing system: acquires, from a camera mounted on a vehicle, images indicating a road comprising a roadway and a sidewalk adjacent to the roadway (S1, S2); acquires vehicle position information indicating the position of the vehicle and shooting direction information indicating the shooting direction of the camera (S1, S2); determines sidewalk position information indicating the position of the sidewalk on the basis of at least the vehicle position information and the shooting direction information (S3); determines a safety level of the sidewalk on the basis of the images and an image recognition model (S6); and generates sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk, and the sidewalk position information (S9).

Description

Information processing system, information processing method, and non-transitory computer-readable medium

[0001] The present disclosure relates to an information processing system, an information processing method, and a non-transitory computer-readable medium.

[0002] A school route safety support system is known that can visualize the safety and dangerous areas of sidewalks on school routes on a map (see, for example, Non-Patent Document 1). Each parent or guardian can register their child's home location, the school route, and dangerous areas around the school route through the school route safety support system disclosed in Non-Patent Document 1. In this way, parents who use this system can share information about the safety and dangerous areas of sidewalks on school routes.

[0003] "School Route Safety Support System: Manages school routes and dangerous areas to ensure children's safety," [online], Mapple Co., Ltd., [Retrieved August 14, 2023], Internet <URL: https: / / mapple.com / solution / pak-tsugakuro / >

[0004] However, in the system disclosed in Non-Patent Document 1, each user registers information about the safety of sidewalks on school routes based on their own subjective opinion, so the objectivity of the information is not fully guaranteed. Furthermore, each user manually registers the information into the system, which creates a burden for each user. Furthermore, if the number of users using the system is small, it is practically difficult to provide comprehensive information about the safety of sidewalks on school routes or to regularly update the information.

[0005] In view of the above, the present disclosure aims to provide an information processing system, an information processing method, and a non-transitory computer-readable medium that enable objective and efficient acquisition of information related to sidewalk safety. The present disclosure also aims to provide an information processing method and a non-transitory computer-readable medium that enable a clear understanding of sidewalk safety and dangerous areas on a map, as well as an information processing method and a non-transitory computer-readable medium that enable route search that takes sidewalk safety into consideration.

[0006] An information processing system according to one aspect of the present disclosure acquires, from a camera mounted on a vehicle, an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway, acquires vehicle position information indicating the position of the vehicle and shooting direction information indicating the shooting direction of the camera, determines sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the shooting direction information, determines a safety level of the sidewalk based on the image and an image recognition model, and generates sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0007] An information processing method according to one aspect of the present disclosure is executed by a computer and includes the steps of acquiring, from a camera mounted on a vehicle, an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway; acquiring vehicle position information indicating the position of the vehicle and shooting direction information indicating the shooting direction of the camera; determining sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the shooting direction information; determining a safety level of the sidewalk based on the image and an image recognition model; and generating sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0008] An information processing method according to one aspect of the present disclosure is executed by a computer and includes a step of visualizing a sidewalk safety level on a map based on sidewalk data, wherein the sidewalk data includes sidewalk safety level information indicating the sidewalk safety level and sidewalk position information indicating the position of the sidewalk.

[0009] An information processing method according to one aspect of the present disclosure is executed by a computer and includes a step of performing a route search that takes into account the safety level of a sidewalk based on sidewalk data and road network data. The sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, information about the position of the sidewalk, and identification information of a link associated with a road consisting of the sidewalk and a roadway adjacent to the sidewalk. The road network data is associated with the sidewalk data via the identification information of the link.

[0010] Also provided is a non-transitory computer readable medium bearing instructions that, when executed, cause a computing device to perform the information processing method.

[0011] According to the present disclosure, it is possible to provide an information processing system, an information processing method, and a non-transitory computer-readable medium that enable objective and efficient acquisition of information regarding sidewalk safety. Also, according to the present disclosure, it is possible to provide an information processing method and a non-transitory computer-readable medium that enable a clear understanding of sidewalk safety and dangerous areas on a map, as well as an information processing method and a non-transitory computer-readable medium that enable route search that takes sidewalk safety into consideration.

[0012] 8 is a diagram illustrating an example of an information processing system according to an embodiment of the present disclosure (hereinafter, referred to as the present embodiment). FIG. 8 is a diagram illustrating an example of the configuration of a vehicle according to the present embodiment. FIG. 8 is a diagram illustrating an example of the configuration of a server according to the present embodiment. FIG. 8 is a flowchart illustrating a series of processes for generating sidewalk data according to the present embodiment. FIG. 8 is a diagram illustrating an example of the data structure of sidewalk data. FIG. 8 is a diagram illustrating photographing area information. FIG. 8 is a diagram illustrating a process for determining sidewalk position information. (a) of FIG. 8 is an example of an image showing a sidewalk with safety level 4. (b) of FIG. 8 is an example of an image showing a sidewalk with safety level 3. (c) of FIG. 8 is an example of an image showing a sidewalk with safety level 2. (d) of FIG. 8 is an example of an image showing a sidewalk with safety level 1. FIG. 8 is a diagram illustrating a process for specifying the safety level of a sidewalk displayed in an image based on an image recognition model. FIG. 8 is a diagram illustrating road network data. FIG. 8 is a flowchart illustrating a process for visualizing the safety level of a sidewalk on a map. FIG. 8 is a schematic diagram illustrating an example of a map on which the safety level of a sidewalk is visualized. FIG. 8 is a diagram illustrating another example of a map on which the safety level of a sidewalk is visualized. FIG. 8 is a flowchart illustrating a route search process taking the safety level of a sidewalk into consideration.

[0013] (Outline of this embodiment) An outline of this embodiment will be described below.

[0014] An information processing system according to one aspect of the present disclosure acquires, from a camera mounted on a vehicle, an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway, acquires vehicle position information indicating the position of the vehicle and shooting direction information indicating the shooting direction of the camera, determines sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the shooting direction information, determines a safety level of the sidewalk based on the image and an image recognition model, and generates sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0015] According to the above configuration, it is possible to obtain sidewalk safety level information indicating the safety level of a sidewalk based on an image captured by a camera mounted on a vehicle and an image recognition model. In this way, it is possible to provide an information processing system that makes it possible to objectively and efficiently obtain information regarding sidewalk safety. For example, a user can clearly grasp the safety of a sidewalk and dangerous areas on a map through the sidewalk safety level visualized on a map. Furthermore, the user can search for a route to a destination taking sidewalk safety into consideration.

[0016] The information processing system may also visualize the safety level of the sidewalk on a map based on the sidewalk data.

[0017] According to the above configuration, the user can clearly understand the safety of the sidewalk and dangerous areas (especially the safety and dangerous areas of sidewalks on school routes) through the sidewalk safety level visualized on the map.

[0018] The information processing system may also determine the coloring of plot points corresponding to the sidewalk position information according to the sidewalk safety level information, and visualize the colored plot points on the map.

[0019] According to the above configuration, the user can clearly understand the safety of the sidewalk and dangerous areas by looking at the plot points displayed on the map.

[0020] The information processing system may also determine the safety level of the sidewalk from the perspective of whether or not there is a protective fence installed on the sidewalk, whether or not there is a step between the sidewalk and the roadway, and whether or not there is a boundary line between the sidewalk and the roadway.

[0021] According to the above configuration, it is possible to objectively determine the safety level of a sidewalk from the viewpoint of whether or not there is a protective fence installed on the sidewalk, whether or not there is a step between the sidewalk and the roadway, and whether or not there is a boundary line between the sidewalk and the roadway.

[0022] The information processing system may acquire attribute information of the safety fence based on the image. The sidewalk data may include the attribute information of the safety fence.

[0023] According to the above configuration, it is possible to evaluate the safety of the sidewalk in more detail using the attribute information of the protective fence.

[0024] The information processing system may acquire external environment information relating to an external environment around the road based on the image. The sidewalk data may include the external environment information.

[0025] According to the above configuration, it is possible to evaluate the safety of the sidewalk in more detail based on external environment information (for example, whether or not there is a shelter adjacent to the sidewalk).

[0026] The information processing system may also identify the sidewalk position information based on the vehicle position information, the shooting direction information, the camera's angle of view information, and map data showing the road.

[0027] According to the above configuration, it is possible to more accurately identify sidewalk position information, and therefore it is possible to acquire information regarding the safety of sidewalks with higher accuracy.

[0028] The information processing system may also specify identification information of links associated with the roads based on the vehicle position information. The links may constitute road network data for route search. The sidewalk data may include identification information of the links. The sidewalk data may be associated with the road network data via the identification information of the links.

[0029] According to the above configuration, since the sidewalk data is associated with the road network data via the link identification information, by utilizing both the sidewalk data and the road network data, it is possible to perform a route search that takes into account the safety level of the sidewalk.

[0030] The information processing system may determine intra-link sidewalk position information indicating a relative position of the sidewalk on the link based on information indicating a start point position and an end point position of the link and the sidewalk position information. The sidewalk data may include the intra-link sidewalk position information.

[0031] According to the above configuration, the sidewalk data includes link identification information and sidewalk position information within the link, so by utilizing both the sidewalk data and the road network data, it is possible to perform route searches that take into account the safety level of the sidewalk.

[0032] The information processing system may also perform a route search that takes into consideration the safety level of the sidewalk, based on the sidewalk data and the road network data.

[0033] According to the above configuration, it is possible to contribute to a reduction in the number of traffic accidents caused by the dangers of sidewalks by searching for a route that takes into account the safety level of the sidewalk.

[0034] An information processing method according to one aspect of the present disclosure is executed by a computer and includes the steps of acquiring, from a camera mounted on a vehicle, an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway; acquiring vehicle position information indicating the position of the vehicle and shooting direction information indicating the shooting direction of the camera; determining sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the shooting direction information; determining a safety level of the sidewalk based on the image and an image recognition model; and generating sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

[0035] According to the above configuration, it is possible to obtain sidewalk safety level information indicating the safety level of a sidewalk based on an image captured by a camera mounted on a vehicle and an image recognition model. In this way, an information processing method can be provided that enables objective and efficient acquisition of information regarding sidewalk safety. For example, a user can clearly grasp the safety of a sidewalk and dangerous areas on a map through the sidewalk safety level visualized on a map. Furthermore, the user can search for a route to a destination that takes sidewalk safety into consideration.

[0036] An information processing method according to one aspect of the present disclosure is executed by a computer and includes a step of visualizing a sidewalk safety level on a map based on sidewalk data, wherein the sidewalk data includes sidewalk safety level information indicating the sidewalk safety level and sidewalk position information indicating the position of the sidewalk.

[0037] According to the above method, the user can clearly understand the safety of the sidewalk and dangerous areas (especially the safety and dangerous areas of the sidewalk on the school route) through the sidewalk safety level visualized on the map.

[0038] An information processing method according to one aspect of the present disclosure is executed by a computer and includes a step of performing a route search that takes into account the safety level of a sidewalk based on sidewalk data and road network data. The sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, information about the position of the sidewalk, and identification information of a link associated with a road consisting of the sidewalk and a roadway adjacent to the sidewalk. The road network data is associated with the sidewalk data via the identification information of the link.

[0039] According to the above method, a user (e.g., a pedestrian or a vehicle driver) can search for a route to a destination that takes into account the safety of sidewalks. In particular, a pedestrian can search for a route to a destination that takes into account the safety of sidewalks on a map app installed on a mobile device. A vehicle driver can search for a route to a destination that takes into account the safety of sidewalks through a navigation system installed in the vehicle. Furthermore, an autonomous vehicle (autonomous robot) that autonomously drives from a departure point to a destination in autonomous driving mode can search for a route to a destination that takes into account the safety of sidewalks.

[0040] Also provided is a non-transitory computer readable medium that holds instructions (computer program) that, when executed, cause a computing device to perform the information processing method.

[0041] (Information Processing System According to the Present Embodiment) Hereinafter, an information processing system 1 according to the present embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of the information processing system 1 according to the present embodiment. As shown in FIG. 1, the information processing system 1 includes a vehicle 2, a server 3, a user terminal 4, and a vehicle 6. These are connected to a communication network 5. The vehicles 2 and 6 and the user terminal 4 are communicatively connected to the server 3 via the communication network 5. The communication network 5 is configured, for example, by the Internet or the like.

[0042] Next, the hardware configuration of the vehicle 2 will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the vehicle 2 according to this embodiment. As shown in Fig. 2, the vehicle 2 includes a vehicle control unit 20, a camera 21, a wireless communication unit 22, a GPS (Global Positioning System) receiver 23, an HMI (Human Machine Interface) 24, a storage device 25, a drive system 26, and a direction sensor 27.

[0043] The vehicle 2 may be a vehicle (automobile) capable of running in an autonomous driving mode. In this example, a four-wheeled vehicle is used as an example of a vehicle, but the number of wheels of the vehicle 2 is not particularly limited. The vehicle control unit 20 is configured to control various components provided in the vehicle 2, and is configured by, for example, at least one electronic control unit (ECU: Electronic Control Unit). The electronic control unit includes a computer system including one or more processors and one or more memories.

[0044] The camera 21 is configured to capture an image of the surrounding environment ahead of the vehicle 2. The camera 21 is disposed at a predetermined position on the vehicle 2 so as to capture an image of the surrounding environment ahead of the vehicle 2, for example, through the windshield of the vehicle 2. The image captured by the camera 21 may be a still image or a moving image. The frame rate of the moving image is not particularly limited. The image shows the road ahead of the vehicle 2. Here, the road is composed of a roadway and a sidewalk adjacent to the roadway. It should be noted that in this specification, the term "sidewalk" does not necessarily mean a sidewalk as defined in the Road Traffic Act of each country. The camera 21 may also be detachably mounted on the vehicle 2. In this regard, the camera 21 may be portable by the driver or the like.

[0045] The wireless communication unit 22 is configured to connect the vehicle 2 to the communication network 5, and includes a transmitting / receiving antenna and a wireless transmitting / receiving circuit. The wireless communication unit 22 may be a wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or may be a wireless communication module compatible with a fourth-generation mobile communication system or a fifth-generation mobile communication system such as LTE.

[0046] The GPS receiver 23 is configured to acquire information related to the current position of the vehicle 2. The HMI 24 is configured to include an input unit that accepts input operations from the driver and an output unit that outputs information related to the driving of the vehicle 2 to the driver. The storage device 25 is an external storage device such as a hard disk drive (HDD) or a solid state drive (SSD). Map data and vehicle control programs may be stored in the storage device 25. The drive system 26 is configured to control the driving state of the vehicle 2. For example, the drive system 26 is configured to control the accelerator, brakes, and steering of the vehicle 2.

[0047] The direction sensor 27 is configured to detect the traveling direction of the vehicle 2. In this example, the shooting direction of the camera 21 is assumed to match the traveling direction of the vehicle 2. Therefore, information about the traveling direction of the vehicle 2 detected by the direction sensor 27 becomes information about the shooting direction of the camera 21. The vehicle 2 transmits image data, vehicle position data, shooting direction data, and field of view information of the camera 21 to the server 3 via the communication network 5. The image data includes a plurality of images and a plurality of pieces of time information. Each of the plurality of images is associated with one of the plurality of pieces of time information. Each of the plurality of pieces of time information indicates the shooting time of the corresponding image (shooting date and time and shooting time).

[0048] The vehicle position data includes a plurality of pieces of vehicle position information and a plurality of pieces of time information. The vehicle position information is information indicating the position (longitude and latitude) of the vehicle 2. The longitude is displayed within the range of -180 degrees to +180 degrees. The latitude is displayed within the range of -90 degrees to +90 degrees. Each of the plurality of pieces of vehicle position information is associated with one of the plurality of pieces of time information. Each of the plurality of pieces of time information indicates the acquisition time of the corresponding vehicle position.

[0049] The shooting direction data includes information about the shooting directions of the multiple cameras 21 (the traveling direction of the vehicle 2) and multiple pieces of time information. The shooting direction of the camera 21 is indicated within a range from 0 degrees to 360 degrees. In this case, 0 degrees indicates the north direction, 90 degrees indicates the east direction, 180 degrees indicates the south direction, and 270 degrees indicates the west direction. Each piece of information about the multiple shooting directions is associated with one of multiple pieces of time information. Each piece of time information indicates the detection time of the corresponding shooting direction.

[0050] Next, the hardware configuration of the server 3 will be described below with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the server 3 according to this embodiment. As shown in Fig. 3, the server 3 includes a control unit 30, a storage device 31, an input / output interface 32, a communication unit 33, an input operation unit 34, and a display unit 35. These elements are connected to a communication bus 36.

[0051] The control unit 30 includes a memory and a processor. The memory is configured to store computer-readable instructions (programs). For example, the memory may include a ROM (Read Only Memory) storing various programs and a RAM (Random Access Memory) having multiple work areas for storing various programs executed by the processor. The processor may include at least one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The CPU may include multiple CPU cores. The GPU may include multiple GPU cores. The processor may be configured to load a specified program from various programs stored in the storage device 31 or the ROM onto the RAM and execute various processes in cooperation with the RAM. The memory may store an information processing program for causing the processor to execute a series of processes (information processing methods) executed by the server 3 in FIGS. 4, 11, and 14.

[0052] The storage device 31 is, for example, a storage device (storage) such as an HDD or SSD, and is configured to store programs and various data. The storage device 31 stores sidewalk data (described later) and map data. The map data includes road network data for route search, background data, annotation data, address data, and store information data. The memory and storage device 31 are examples of non-transitory computer-readable media.

[0053] As shown in FIG. 10 , the road network data is composed of multiple links (lines) and multiple nodes (points). A node is connected to multiple links. Each node is connected to other nodes via one link. Each link is assigned unique identification information (ID). Each link is associated with, for example, road attribute information and road regulation information (e.g., one-way streets, etc.). The road attribute information includes, for example, road type information, route cost information, number of lanes information, and width information. Each node is assigned unique identification information (ID). Each node is associated with, for example, lane information, direction guidance information, traffic light information, and intersection name information. In this way, the road network data has information related to links and information related to nodes.

[0054] The background data includes illustration data related to the visual background of the map (e.g., map shapes such as buildings, ocean, forests, roads, etc.). The annotation data includes text information to be displayed on the map (e.g., names of buildings and mountains, etc.). The address data includes address information associated with each building on the map. The store information data includes information related to stores on the map (e.g., opening and closing times, etc.).

[0055] 3 , the input / output interface 32 is an interface that enables connection between an external device and the server 3, and includes an interface conforming to a predetermined communication standard such as the USB standard or the HDMI (registered trademark) standard. The communication unit 33 may include various wired communication modules for communicating with external terminals on the communication network 5. The input operation unit 34 is, for example, a touch panel, a mouse, and / or a keyboard, and is configured to accept input operations by an operator and generate operation signals in response to the input operations by the operator. The display unit 35 is, for example, configured by a video display and a video display circuit.

[0056] Returning to Fig. 1 , the user terminal 4 is a terminal operated by the user U. The user terminal 4 is communicatively connected to the server 3 via a communication network 5, and receives map information from the server 3. The user terminal 4 includes a control unit having a processor and a memory, a storage device, a wireless communication unit, an input operation unit, a GPS receiver configured to acquire the current location of the user terminal 4, and a display unit. A map application or a web browser for displaying a map may be installed on the user terminal 4.

[0057] The vehicle 6 may be a vehicle capable of running in an autonomous driving mode. The vehicle 6 may include, for example, a vehicle control unit, a sensing device such as a camera, a millimeter wave radar, or a LiDAR unit, a wireless communication unit, a GPS receiver, an HMI, a storage device that stores map data and the like, and a drive system. The vehicle 6 may further include a car navigation system.

[0058] (Sidewalk Data Generation Process) Next, a series of processes for generating sidewalk data (an example of a data structure) according to this embodiment will be described below with reference to Fig. 4. Fig. 4 is a flowchart for describing a series of processes for generating sidewalk data according to this embodiment. As shown in Fig. 5, an example of information constituting the sidewalk data includes the image capture date and capture time (image capture time), an image ID, sidewalk position information (longitude, latitude), a link ID (link identification information), sidewalk position information within a link, image capture direction information, image capture area information (left-side image capture area or right-side image capture area), sidewalk safety level information, recognition score information, evacuation space information, and protective fence attribute information.

[0059] In step S1, the server 3 acquires image data, vehicle position data, shooting direction data, and camera angle of view information from the vehicle 2 via the communication network 5. As described above, the image data is data composed of multiple images and multiple pieces of time information linked to them. Each image shows the road ahead of the vehicle 2 (see, for example, Figure 6). The vehicle position data is data composed of multiple pieces of vehicle position information and multiple pieces of time information linked to them. The shooting direction data is data composed of multiple pieces of shooting direction information and multiple pieces of time information linked to them. The camera angle of view information is information indicating the angle of view of the camera 21 of the vehicle 2.

[0060] In step S2, the server 3 acquires an image, vehicle position information, and shooting direction information that are associated with each other from the image data, vehicle position data, and shooting direction data received from the vehicle 2. Here, because the image data, vehicle position data, and shooting direction data each contain time information, the server 3 associates the image, vehicle position information, and shooting direction information with each other through the common time information. For example, the server 3 associates an image Mt1 at time t1, a vehicle position Pt1 at time t1, and a shooting direction Dt1 at time t1 with each other. In this example, the server 3 extracts one image from the multiple images included in the image data, and then extracts vehicle position information and shooting direction information associated with the extracted one image from the vehicle position data and the shooting direction data.

[0061] In step S3, the server 3 acquires sidewalk position information indicating the position of the sidewalk based on the vehicle position information and the shooting direction information. More specifically, the server 3 determines the sidewalk position information corresponding to the vehicle position information based on the vehicle position information, the shooting direction information, the camera angle of view information, and map data indicating the road. In this regard, the process of determining the sidewalk position information will be described below with reference to Figure 7. Figure 7 is a diagram for explaining the process of determining the sidewalk position information.

[0062] As shown in Figure 7, assume that the position of a vehicle 2 traveling on a roadway 70 is P. In this case, position P1 of a left-side sidewalk 71 adjacent to the left side of the roadway 70 is defined as the intersection point of the boundary line between the roadway 70 and the left-side sidewalk 71 (more specifically, the boundary area between the outer roadway line 72 and the left-side sidewalk 71) and the left edge VL of the field of view V of the camera 21.

[0063] The server 3 acquires map information about the vicinity of the position P from the map data based on vehicle position information indicating the position P of the vehicle 2. Here, the map information about the vicinity of the position P includes position information about the roadway 70, the left sidewalk 71, and the outer lane line 72. Next, the server 3 determines the position of the angle of view V of the camera 21 on the map shown in FIG. 7 (in other words, the position of the imaging area of ​​the camera 21) based on the imaging direction information and the camera angle of view information. In the example shown in FIG. 7 , the vehicle 2 is traveling west, so the imaging direction of the camera 21 is west (270 degrees), and the angle of view V of the camera 21 extends westward from the vehicle 2. For example, in this example, when the vehicle 2 is traveling east, the imaging direction of the camera 21 is east (90 degrees), so the angle of view V of the camera 21 extends eastward from the vehicle 2. Thus, unless the imaging direction of the camera 21 is known, the position of the angle of view V cannot be determined, and therefore accurate sidewalk position information cannot be acquired.

[0064] Next, based on the positional information regarding the roadway 70, left sidewalk 71, and outer roadway line 72, and the positional relationship of the field of view V of the camera 21, the server 3 calculates the position (longitude, latitude) of the intersection point where the boundary line between the roadway 70 and left sidewalk 71 (more specifically, the boundary area between the outer roadway line 72 and the left sidewalk 71) intersects with the left edge VL of the field of view V of the camera 21. In this way, the server 3 determines the calculated positional information of the intersection point as sidewalk position information indicating the position of the left sidewalk 71.

[0065] Similarly, position P2 of the right-side sidewalk 73 adjacent to the right side of the roadway 70 is defined as the intersection point where the boundary line between the roadway 70 and the right-side sidewalk 73 (more specifically, the boundary area between the roadway outer line 74 and the right-side sidewalk 73) intersects with the right end VR of the field of view V of the camera 21.

[0066] The server 3 acquires position information regarding the roadway 70, the right sidewalk 73, and the outer roadway line 74 from the map data. Next, the server 3 calculates the position (longitude, latitude) of the intersection point where the boundary line between the roadway 70 and the right sidewalk 73 (more specifically, the boundary area between the outer roadway line 74 and the right sidewalk 73) intersects with the right edge VR of the field of view V of the camera 21, based on the position information regarding the roadway 70, the right sidewalk 73, and the outer roadway line 74 and the positional relationship of the field of view V of the camera 21. In this way, the server 3 determines the calculated position information of the intersection point as field of view information indicating the position of the right sidewalk 73.

[0067] In this way, in step S3, two pieces of sidewalk position information (sidewalk position information indicating the position of the left sidewalk 71 and sidewalk position information indicating the position of the right sidewalk 73) corresponding to the vehicle position information are acquired. Note that in this example, the intersection of the boundary line between the sidewalk and the roadway and the end of the camera's angle of view V is defined as the sidewalk position, but the method for determining the sidewalk position in this embodiment is not limited to this. Also, if the right sidewalk 73 is determined to be a median strip rather than a sidewalk based on the map data, sidewalk position information for the right sidewalk does not need to be acquired.

[0068] Next, in step S4, the server 3 identifies a link ID, which is identification information of a link associated with the road including the vehicle position information, based on the vehicle position information. Each link constituting the road network data shown in FIG. 10 has one or more pieces of position information. Specifically, each link has the position of a start point, the position of an end point, and the positions of one or more component points located between the start point and the end point. In this way, the server 3 identifies the link most relevant to the vehicle position information from the position information associated with each link. In particular, the server 3 identifies a link having position information that matches the vehicle position information or a link having position information closest to the vehicle position information, and then acquires the ID of the identified link. The link ID is, for example, an integer value.

[0069] Next, in step S5, the server 3 determines intra-link sidewalk position information indicating the relative position of the sidewalk on the link based on the start and end positions of the link identified in step S4 and the sidewalk position information. The intra-link sidewalk position information is displayed within a range from 0 to 1. At this point, the intra-link sidewalk position is calculated by dividing the distance between the start position and sidewalk position of the link by the distance between the start position and end position of the link (total length of the link). For example, if the sidewalk position is located at the midpoint of the entire link, the intra-link sidewalk position information is 0.5. If the sidewalk position coincides with the end position of the link, the intra-link sidewalk position information is 1.0.

[0070] Next, in step S6, the server 3 determines the safety level of the sidewalk based on the image and the image recognition model 130 (see FIG. 9). In particular, the server 3 may determine the safety level of the left sidewalk present in the image and also the safety level of the right sidewalk present in the image based on one image and the image recognition model. Specifically, as shown in FIG. 6, the image captured by the camera 21 of the vehicle 2 has two sidewalks (left sidewalk and right sidewalk) on either side of the roadway. For this reason, the server 3 determines the safety level of the left side photographed area S in one image. Ｌ and determine the safety level of the left sidewalk existing in the right photographing area S Ｒ In this respect, the server 3 determines the safety level of the right sidewalk present in each image. Ｌ , S Ｒ After extracting the image of each extracted photographing area S Ｌ , S Ｒ By inputting the image of each photographed area S into the input layer of the image recognition model, Ｌ , S Ｒ The safety level of the sidewalk in the image may be determined.

[0071] 9, pixel values ​​of an image showing a road are input to the input layer of the image recognition model 130. The image input to the input layer is a left sidewalk image of the left side of the road. Ｌ and the right sidewalk. ＲThe image may include only one of the left and right shooting areas S Ｌ and right side photography area S Ｒ The output layer of the image recognition model 130 outputs the probability of each sidewalk safety level. In this embodiment, the output layer of the image recognition model 130 outputs the probability of each of the sidewalk safety levels 4, 3, 2, 1, and 0.

[0072] At this point, the left photographing area S Ｌ Or right side shooting area S Ｒ When the image of the left sidewalk is input to the image recognition model 130, the server 3 determines the safety level of the left sidewalk or the right sidewalk. Ｌ and right side photography area S Ｒ is input to the image recognition model 130, the server 3 determines the safety level of both the left sidewalk and the right sidewalk.

[0073] The image recognition model 130 is a learning model constructed by machine learning. The image recognition model 130 is constructed from various learning images showing sidewalks with safety level 4, various learning images showing sidewalks with safety level 3, various learning images showing sidewalks with safety level 2, various learning images showing sidewalks with safety level 1, and various learning images showing safety level 0 (the classification of the sidewalk safety level cannot be determined) (for example, images that do not show a sidewalk such as a median strip). The image recognition model 130 is stored in the storage device 31 of the server 3, and the control unit 30 can classify the safety level of the sidewalk displayed in the image by using the image recognition model 130.

[0074] Each sidewalk safety level will be described below with reference to Figure 8. Figure 8(a) is an example of an image showing a sidewalk with safety level 4. Figure 8(b) is an example of an image showing a sidewalk with safety level 3. Figure 8(c) is an example of an image showing a sidewalk with safety level 2. Figure 8(d) is an example of an image showing a sidewalk with safety level 1. As shown in Figure 8, level 4 is the safest sidewalk safety level, and level 1 is the least safe. In other words, the safety of the sidewalk decreases as you progress from level 4 to level 1 (in other words, the danger of the sidewalk increases).

[0075] As shown in FIG. 8( a), at sidewalk safety level 4, there is a step between the sidewalk and the roadway, and a protective fence for pedestrian protection is installed on the sidewalk. An image showing such a sidewalk is recognized by the image recognition model 130 as having safety level 4 (high sidewalk safety). In other words, the probability of a unit with safety level 4 in the output layer of the image recognition model 130 is the highest. As shown in FIG. 8( b), at sidewalk safety level 3, there is a step between the sidewalk and the roadway, but there is no protective fence for pedestrian protection on the sidewalk. An image showing such a sidewalk is recognized by the image recognition model 130 as having safety level 3 (slightly high sidewalk safety). In other words, the probability of a unit with safety level 3 in the output layer of the image recognition model 130 is the highest.

[0076] As shown in FIG. 8( c), at sidewalk safety level 2, there are no steps or protective fences between the sidewalk and the roadway, but there is a roadway boundary line indicating the boundary of the roadway. An image showing such a sidewalk is recognized by the image recognition model 130 as having safety level 2 (the sidewalk is somewhat dangerous). That is, the probability of a unit with safety level 2 in the output layer of the image recognition model 130 is the highest. As shown in FIG. 8( d), at sidewalk safety level 1, there are no steps or protective fences between the sidewalk and the roadway, and there is no roadway boundary line indicating the boundary of the roadway. An image showing such a sidewalk is recognized by the image recognition model 130 as having safety level 1 (the sidewalk is highly dangerous). That is, the probability of a unit with safety level 1 in the output layer of the image recognition model 130 is the highest.

[0077] In addition, if the safety level of a sidewalk displayed in an image by the image recognition model 130 is not classified into any of safety levels 1 to 4, it is classified as safety level 0. In this case, the probability of the unit in the output layer associated with safety level 0 is highest. For example, if a road is not present in the image captured by the camera 21 mounted on the vehicle 2 (e.g., if the image shows a fence of a private house, etc.), the image recognition model 130 is unable to classify the safety level of the sidewalk. Similarly, if an object that is not a sidewalk, such as a median strip on a Nijo Road, is displayed in the image, the image recognition model 130 is unable to classify the safety level of the sidewalk. Taking such situations into consideration, units with safety level 0, which indicate that safety level classification is impossible, are provided in the output layer of the image recognition model 130 as indistinguishable units. In this way, by providing units with safety level 0 in the output layer of the image recognition model 130, it is possible to further improve the classification accuracy of each safety level.

[0078] In this embodiment, the server 3 determines the safety level of the sidewalk displayed on the image based on the image and the image recognition model 130, and then inputs the sidewalk safety level information and the recognition score information into the sidewalk data. Here, the recognition score is the probability of the determined safety level, more specifically, the probability value of the output layer unit associated with the determined safety level.

[0079] In this example, the safety or danger of the sidewalk is classified into four safety levels (five safety levels if safety level 0 is added), but the number of safety level classifications is not particularly limited. Also, in this example, the method of classifying the safety levels is to objectively evaluate the safety level of the sidewalk from the perspective of whether or not there is a protective fence installed on the sidewalk, whether or not there is a step between the sidewalk and the roadway, and whether or not there is a boundary line between the sidewalk and the roadway (outer roadway line), but the evaluation criteria are not limited to these items.

[0080] Returning to FIG. 4 , in step S7, the server 3 acquires external environment information about the external environment around the road based on the image and the external environment recognition model. Similarly, the external environment recognition model is constructed using various learning images depicting the external environment (e.g., various evacuation spaces). In particular, the server 3 determines, as the external environment information, whether or not an evacuation space (avoidance space) exists around the road where pedestrians can escape from a dangerous vehicle. In this regard, if a vacant lot, park, parking lot, etc., is adjacent to the road (sidewalk), the server 3 determines that an evacuation space exists around the road. On the other hand, if a block wall of a house is adjacent to the road (sidewalk), the server 3 determines that no evacuation space exists around the road. Thus, because the presence or absence of an evacuation space (an example of external environment information) is one factor contributing to the safety of the sidewalk, the external environment information may be included in the sidewalk data in this embodiment.

[0081] In step S8, the server 3 acquires attribute information of the pedestrian protection fence based on the image and the attribute information recognition model. In particular, the server 3 can estimate the material, strength, and height of the protection fence as attribute information of the protection fence. For example, the server 3 may classify the material of the protection fence as wood, steel, or concrete. Furthermore, the server 3 can estimate the overall strength (N / cm) of the protection fence based on the shape information of the protection fence and the material information of the protection fence. Furthermore, the server 3 can estimate the height of the protection fence (the height of the protection fence from the surface of the sidewalk) based on the vehicle position information and the protection fence displayed in the image. In this way, the server 3 can include the attribute information of the protection fence in the sidewalk data in this embodiment, because the attribute information of the protection fence is one of the factors that contribute to the safety of the sidewalk.

[0082] In step S9, the server 3 updates the sidewalk data shown in Fig. 5 based on the various information acquired in steps S1 to S8. As shown in Fig. 5, the sidewalk data includes information on the shooting time, image ID, sidewalk position, link ID, sidewalk position within the link, shooting direction, shooting area, sidewalk safety level, recognition score (probability), evacuation space, and protective fence. In this regard, the image ID may be generated by the hash value of the image. Also, when two sidewalks, a left sidewalk and a right sidewalk, are displayed in one image, the left side shooting area S Ｌ The above information is generated for the left sidewalk in the right photographing area S Ｒ The above information is generated for the right sidewalk in. That is, the sidewalk position, sidewalk position within the link, sidewalk safety level, and recognition score are generated for the left sidewalk, and the sidewalk position, sidewalk position within the link, sidewalk safety level, and recognition score are also generated for the right sidewalk.

[0083] According to this embodiment, it is possible to obtain sidewalk safety level information indicating the safety level of a sidewalk based on an image captured by the camera 21 mounted on the vehicle 2 and the image recognition model 130. In this way, it is possible to provide an information processing system 1 that makes it possible to objectively and efficiently obtain information about sidewalk safety. Furthermore, it is possible to evaluate the safety of a sidewalk in more detail based on information about the presence or absence of an evacuation space and attribute information of a protective fence. In this respect, if the sidewalk safety level is 1 and no evacuation space exists around the road, it is possible to objectively recognize that the sidewalk is highly dangerous. Furthermore, if the sidewalk safety level is 4, an evacuation space exists, and the height and strength of the protective fence are sufficiently high, it is possible to objectively recognize that the sidewalk is highly safe.

[0084] Furthermore, according to this embodiment, sidewalk position information corresponding to vehicle position information is determined based on vehicle position information, shooting direction information, camera angle of view information, and map data showing roads, making it possible to identify sidewalk positions more accurately. In this way, information related to sidewalk safety can be acquired with higher accuracy. In particular, as will be described later, improving the accuracy of sidewalk position information is useful when visualizing sidewalk safety levels as plotted points on a map.

[0085] (Process for Visualizing Sidewalk Safety Levels) Next, the process for visualizing sidewalk safety levels on a map will be described below with reference to FIG. 11 . FIG. 11 is a flowchart for explaining the process for visualizing sidewalk safety levels on a map. As a prerequisite for this process, it is assumed that the server 3 has the sidewalk data and map data shown in FIG. 5 stored in the storage device 31. In addition, in this example, it is assumed that a map on which the sidewalk safety levels are visualized is displayed on a user terminal 4 that is communicatively connected to the server 3 via the communication network 5 (see FIG. 1 ). It is assumed that a map display app and a browser are installed on the user terminal 4.

[0086] 11 , in step S10, in response to an input operation by the user U, the user terminal 4 transmits location information of the user terminal 4 to the server 3 via the communication network 5. The server 3 then receives the location information of the user terminal 4. In step S11, the server 3 extracts map information of the area around the current location of the user terminal 4 from the map data. In step S12, the server 3 extracts information related to sidewalks that exist within the map (coordinate area) of the area around the current location of the user terminal 4 from the sidewalk data. Here, the information related to the sidewalks is sidewalk position information and sidewalk safety level information.

[0087] In step S13, the server 3 determines the color of the plot point corresponding to the sidewalk position in accordance with the sidewalk safety level information. For example, if the sidewalk safety level associated with the sidewalk position P1 is 4, the server 3 determines the color of the plot point plotted at the sidewalk position P1 in accordance with the sidewalk safety level 4. The colors of the plot points may differ from one another depending on the sidewalk safety level. For example, the plot point may be colored a first color C1 for sidewalk safety level 1, a second color C2 for sidewalk safety level 2, a third color C3 for sidewalk safety level 3, and a fourth color C4 for sidewalk safety level 4. Here, the colors C1 to C4 may be different from one another. For example, the color C4 may be green, the color C3 may be yellow, the color C2 may be orange, and the color C1 may be red.

[0088] Next, in step S14, the server 3 generates map image data in which plot points corresponding to each sidewalk position are visualized, and then transmits the map image data to the user terminal 4 via the communication network 5. Thereafter, the map in which plot points corresponding to each sidewalk position are visualized is displayed on the display unit of the user terminal 4.

[0089] 12 is a schematic diagram showing an example of a map 140 in which the safety level of a sidewalk is visualized. As shown in FIG. 12, a large number of plot points K corresponding to the sidewalk positions are visualized along both ends of the road displayed on the map 140 (i.e., the left sidewalk and the right sidewalk). Furthermore, the color of the large number of plot points K is determined according to the sidewalk safety level, and thus the safety of the sidewalk is visualized on the map 140. In this way, by looking at the plot points K displayed on the map 140, the user U can intuitively grasp the safety of the sidewalk and dangerous areas (particularly the safety and dangerous areas of sidewalks on school routes).

[0090] 12, a defective area H of the plotted points may exist. For example, if the sidewalk safety level is 0 (in other words, if the sidewalk safety level cannot be classified), the plotted points associated with safety level 0 may not be displayed on the map 140. FIG. 13 is a diagram showing another example of a map 150 in which sidewalk safety levels are visualized. FIG. 13 shows a wide-area map 150 covering the southern area of ​​Tokyo. A large number of plotted points indicating sidewalk safety may be visualized in a specified area within the map 150.

[0091] The map on which the safety level of the sidewalk is visualized may be displayed on the display unit of the navigation system of the vehicle 6 (see FIG. 1). In this case, the sidewalk data and map data stored in the server 3 may be periodically distributed to the vehicle 6 via the communication network 5. When the sidewalk data and map data are stored in the storage device of the vehicle 6, the control unit of the navigation system mounted on the vehicle 6 may execute the processes shown in steps S10 to S15 in FIG. 11 to display the map on which the safety level of the sidewalk is visualized on the display unit of the navigation system.

[0092] In addition, in this example, plot points indicating the safety of the sidewalk are visualized on the map, but in addition to this, attribute information of the protective fence and / or external environmental information around the road (e.g., evacuation space information) may also be visualized on the map.

[0093] (Route Search Processing Taking Sidewalk Safety Levels into Account) Next, the route search processing taking sidewalk safety levels into account will be described below with reference to FIG. 14. FIG. 14 is a flowchart for explaining the route search processing taking sidewalk safety levels into account. As a prerequisite for this processing, it is assumed that the server 3 has the sidewalk data and map data shown in FIG. 5 stored in the storage device 31. In addition, in this example, it is assumed that the results of the route search processing are displayed on the user terminal 4 that is communicatively connected to the server 3 via the communication network 5. It is assumed that the user terminal 4 has an app and a browser for displaying maps installed.

[0094] As shown in FIG. 14 , in step S20, in response to an input operation by the user U, the user terminal 4 transmits current location information (location information of the current location) and destination information (location information of the destination) to the server 3 via the communication network 5. The server 3 then receives the current location information and the destination information. In step S21, the server 3 performs a route search process from the current location to the destination based on the road network data and sidewalk data of the map data, taking into account the route cost and sidewalk safety level of each link. Because the map data includes link IDs, the road network data consisting of multiple links and multiple nodes is associated with the sidewalk data via the link IDs. Furthermore, each link in the road network data has route cost information (e.g., distance cost). In the sidewalk data, each sidewalk safety level information is associated with each link. Therefore, by using both the road network data and the map data, it is possible to perform a route search process from the current location to the destination, taking into account both the route cost and the sidewalk safety level. Furthermore, if multiple sidewalk safety levels are associated with one link, the average value of the sidewalk safety levels belonging to that link may be considered in the route search process. For example, if 10 sidewalk safety levels are associated with a specific link and the average value of the 10 sidewalk safety levels belonging to the specific link is 3.5, the average sidewalk safety level of 3.5 may be considered as the sidewalk safety level of the specific link in the route search process.

[0095] The server 3 may extract only links associated with an average sidewalk safety level equal to or higher than a predetermined level, and then determine an optimal route from the perspective of route cost based on the extracted links. Alternatively, the server 3 may extract multiple candidate routes through a route search process from the perspective of the route cost of each link, and then select the route with the highest average sidewalk safety level from among the multiple candidate routes. In this way, a route search process is executed that takes both the route cost and the sidewalk safety level into consideration.

[0096] The server 3 may also calculate a corrected route cost for each link that takes into account the sidewalk safety level based on the route cost of each link and the average value of the sidewalk safety level, and then perform a route search from the departure point to the destination based on the corrected route cost of each link. The route search method may use a conventional route search algorithm such as the Dijkstra algorithm or the A-Star algorithm.

[0097] In step S22, the server 3 transmits the route search results to the user terminal 4 via the communication network 5. The route search results are then displayed on the display unit of the user terminal 4. According to this embodiment, since the sidewalk data is associated with the road network data via a link ID, it is possible to perform a route search that takes into account the safety level of the sidewalk by utilizing both the sidewalk data and the road network data. Therefore, the user U can search for a route to his or her destination that takes into account the safety of the sidewalk.

[0098] The route search results may be displayed on the display unit of the navigation system of the vehicle 6. In this case, the sidewalk data and map data stored in the server 3 may be periodically distributed to the vehicle 6 via the communication network 5. When the sidewalk data and map data are stored in the storage device of the vehicle 6, the control unit of the navigation system mounted on the vehicle 6 can perform a route search process from the current location to the destination taking into consideration the safety of the sidewalks by executing the processes shown in steps S20 to S23 in Fig. 14. The vehicle 6 may be an autonomous vehicle (autonomous robot) that autonomously drives from the departure point to the destination in an autonomous driving mode.

[0099] Although the embodiments of the present invention have been described above, the technical scope of the present invention should not be construed as being limited by the description of the present embodiments. The present embodiments are merely examples, and it will be understood by those skilled in the art that various modifications of the embodiments are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents.

[0100] This application appropriately incorporates by reference the contents disclosed in Japanese Patent Application No. 2024-007704 filed on January 22, 2024.

Claims

1. An information processing system that acquires an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle, acquires vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera, determines sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information, determines the safety level of the sidewalk based on the image and an image recognition model, and generates sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

2. The information processing system according to claim 1, which visualizes the safety level of the sidewalk on a map based on the sidewalk data.

3. The information processing system according to claim 2, which determines the coloring of plot points corresponding to the sidewalk position information according to the sidewalk safety level information and visualizes the colored plot points on the map.

4. The information processing system according to claim 1, which determines the safety level of the sidewalk from the viewpoints of the presence or absence of a protective fence installed on the sidewalk, the presence or absence of a step between the sidewalk and the roadway, and the presence or absence of a boundary line between the sidewalk and the roadway.

5. The information processing system according to claim 4, which acquires attribute information of the protective fence based on the image, and the sidewalk data includes the attribute information of the protective fence.

6. The information processing system according to claim 1, which acquires external environment information regarding the external environment around the road based on the image, and the sidewalk data includes the external environment information.

7. The information processing system according to claim 1, which specifies the sidewalk position information based on the vehicle position information, the imaging direction information, the field of view angle information of the camera, and map data showing the road.

8. The information processing system identifies identification information of a link associated with the road based on the vehicle position information, the link constitutes road network data for route search, the sidewalk data includes the identification information of the link, and the sidewalk data is associated with the road network data via the identification information of the link. The information processing system according to claim 1.

9. The information processing system determines sidewalk position information within the link indicating the relative position of the sidewalk on the link based on information indicating the start point position and the end point position of the link and the sidewalk position information, and the sidewalk data includes the sidewalk position information within the link. The information processing system according to claim 8.

10. The information processing system executes route search considering the safety level of the sidewalk based on the sidewalk data and the road network data. The information processing system according to claim 8.

11. A step of acquiring an image showing a road consisting of a roadway and a sidewalk adjacent to the roadway from a camera mounted on a vehicle, a step of acquiring vehicle position information indicating the position of the vehicle and imaging direction information indicating the imaging direction of the camera, a step of determining sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the imaging direction information, a step of determining the safety level of the sidewalk based on the image and an image recognition model, and a step of generating sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information. An information processing method executed by a computer.

12. An information processing method executed by a computer, including a step of visualizing the safety level of a sidewalk on a map based on sidewalk data, the sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and sidewalk position information indicating the position of the sidewalk.

13. An information processing method executed by a computer, comprising a step of performing route search considering the safety level of a sidewalk based on sidewalk data and road network data, wherein the sidewalk data includes sidewalk safety level information indicating the safety level of the sidewalk, information regarding the position of the sidewalk, and identification information of a link associated with a road composed of the sidewalk and a lane adjacent to the sidewalk, and the road network data is associated with the sidewalk data via the identification information of the link.

14. A non-transitory computer-readable medium holding instructions that, when executed by a computer device, cause the computer device to obtain an image showing a road composed of a lane and a sidewalk adjacent to the lane from a camera mounted on a vehicle, obtain vehicle position information indicating the position of the vehicle and shooting direction information indicating the shooting direction of the camera, determine sidewalk position information indicating the position of the sidewalk based on at least the vehicle position information and the shooting direction information, determine the safety level of the sidewalk based on the image and an image recognition model, and generate sidewalk data including sidewalk safety level information indicating the safety level of the sidewalk and the sidewalk position information.

Images