Pedestrian movement direction determination system and pedestrian movement direction determination software

Abstract

To provide a pedestrian travel direction determination system and pedestrian travel direction determination software which can determine an easy and safe direction for a pedestrian when he / she travels.SOLUTION: A pedestrian travel direction determination system 1 comprises a spectacle device 2 and a server 3. The spectacle device 2 comprises an imaging camera 4, an object detection unit 5, a travel direction detection unit 6 and a travel direction determination unit 7. The object detection unit 5 detects a crosswalk left boundary line segment and a crosswalk right boundary line segment extending in an inclined manner in a pickup image. The travel direction determination unit 7 determines that a direction of a pedestrian is in a normal state when the travel direction detection unit 6 detects that the crosswalk left boundary line segment and the crosswalk right boundary line segment do not cross an image center line and detects that the image center line passes through a position closer to an upper end of the crosswalk left boundary line segment and an upper end of the crosswalk right boundary line segment than a lower end of the crosswalk left boundary line segment and a lower end of the crosswalk right boundary line segment.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a pedestrian movement direction determination system and pedestrian movement direction determination software, and more particularly to a pedestrian movement direction determination system and pedestrian movement direction determination software for people with poor mobility, such as the elderly and the visually impaired. [Background technology]

[0002] For people with mobility issues, going outside and walking along the road to their destination is extremely difficult, no matter how simple the road may be.

[0003] To help those with mobility issues, especially those who are visually impaired, there are braille blocks, guide bells, braille signs, and white canes, and guide dogs and walking training by walking trainers are also available.

[0004] In addition, various techniques have been proposed to assist people with walking disabilities. For example, Patent Document 1 describes an invention in which a pedestrian terminal transmits a message including pedestrian information to a roadside unit from an ITS communication unit, and when the roadside unit receives the message sent from the pedestrian terminal with its ITS communication unit, it stores the pedestrian information included in the message in a memory unit as passage history information.Based on the passage history information in the memory unit, a control unit generates route guidance information (pedestrian support information) regarding evacuation routes in the event of a disaster, in particular detecting people in need of assistance such as elderly people and people who frequently engage in risky behavior, and generates route guidance information for these people.Then, a message including that route guidance information is transmitted to nearby pedestrian terminals. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-53371 A Summary of the Invention [Problem to be solved by the invention]

[0006] By the way, visually impaired people often worry, for example when crossing a pedestrian crossing, that they may walk in a direction that strays from the crosswalk, and even when walking on a regular sidewalk, that they may walk toward a curb or a step.

[0007] The invention described in Patent Document 1 can generate information to assist pedestrians based on information collected from pedestrian terminals, but this is insufficient to alleviate the above-mentioned anxieties of visually impaired people.

[0008] The present invention has been devised in consideration of the above points, and has an object to provide a pedestrian traveling direction determination system and pedestrian traveling direction determination software that can determine a safe and secure direction for pedestrians when proceeding. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the pedestrian travel direction determination system of the present invention comprises an imaging unit capable of imaging at least an area in front of a pedestrian; an object detection unit capable of detecting a boundary line of an installed object that is an installed object in an image obtained by imaging the image unit and that has an image center line that passes through a center point of the image and extends in the vertical direction of the image, the boundary line extending at an angle with respect to the vertical direction of the image in at least a lower region of the image; a travel direction detection unit capable of detecting whether the boundary line intersects with the image center line and detects whether the image center line passes through a position closer to an upper end of the boundary line than a lower end of the boundary line; and a travel direction determination unit capable of determining that the pedestrian's orientation is normal when the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line.

[0010] Here, the direction of travel detection unit can detect whether a boundary line intersects with an image center line, and can detect whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, and can detect the direction of a pedestrian relative to an installed object having an inclined boundary line in the captured image. Moreover, a "boundary line segment" means a straight line segment.

[0011] In addition, if the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, the travel direction determination unit, which can determine that the pedestrian's orientation is normal, can confirm whether the pedestrian's orientation relative to an installed object having an inclined boundary line is normal. In addition, in the present invention, the "normal state" means a state in which a pedestrian is facing a direction in which he or she can proceed safely and securely.

[0012] In addition, in the pedestrian travel direction determination system of the present invention, the boundary line segment can be configured as a left boundary line segment whose upper end x-coordinate value is greater than the lower end x-coordinate value in the coordinate system of the captured image, and a right boundary line segment whose upper end x-coordinate value is smaller than the lower end x-coordinate value in the coordinate system of the captured image.

[0013] Here, the "coordinate system of the captured image" is an image coordinate system in which, when the captured image is rectangular, the upper left corner is the origin (0,0), the x-axis runs horizontally to the right from the origin, and the y-axis runs vertically downward from the origin. In this case, when the travel direction detection unit detects that the image center line intersects with the left boundary line segment or the right boundary line segment, it is possible to determine whether the pedestrian is facing toward the left or right side of the installed object having the inclined boundary line segment, making it easy to correct the pedestrian's orientation.

[0014] Furthermore, the pedestrian travel direction determination system of the present invention includes a detected object information recording unit capable of recording installed object information, which is information relating to an installed object, and an object identification unit capable of identifying the installed object by comparing the information of the installed object detected by the object detection unit with the installed object information recorded by the detected object information recording unit. When the object identification unit identifies the installed object as a crosswalk, the travel direction detection unit can be further configured to detect whether the image center line intersects with a crosswalk arrival gate line segment that is the portion of a horizontal line segment having the smallest y coordinate value, which is the vertical coordinate value in the coordinate system of the captured image, of a crosswalk existence area rectangle corresponding to the entire crosswalk, which is displayed in the captured image when the object detection unit detects an installed object, that is, a y coordinate value that is the vertical coordinate value in the coordinate system of the captured image, that is greater than the x coordinate value of the intersection with the left boundary line segment or smaller than the x coordinate value of the intersection with the right boundary line segment.

[0015] In this case, it becomes easier to check whether the orientation of the pedestrian with respect to the crosswalk is normal.

[0016] Furthermore, in the pedestrian travel direction determination system of the present invention, the travel direction detection unit can be further configured to detect whether the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line is within a predetermined range when the object identification unit identifies an installed object other than the installed object identified by the object identification unit as a crosswalk as a pedestrian traffic light, and the travel direction determination unit can be further configured to determine that the direction of the pedestrian is normal when the travel direction detection unit detects that the crosswalk arrival entrance line intersects with the image center line and detects that the shortest distance between the coordinates of the pedestrian traffic light and the image center line is within a predetermined range.

[0017] Here, the "shortest distance" refers to the distance (difference) between x coordinate values ​​when the y coordinate values ​​are the same in the coordinate system of the captured image. In addition, the "predetermined range" here means a range greater than "0" and less than the shortest distance between the imaginary center line that runs through the center of the crosswalk and the coordinates of the pedestrian traffic light. In this case, it is possible to check with even greater accuracy whether the direction of the pedestrian with respect to the crosswalk is normal.

[0018] Furthermore, the pedestrian travel direction determination system of the present invention includes a notification unit capable of notifying the result of the determination made by the travel direction determination unit, wherein the travel direction determination unit is capable of determining that the pedestrian's orientation is in a first abnormal state when the travel direction detection unit detects that the image center line intersects with the left boundary line segment, and is capable of determining that the pedestrian's orientation is in a second abnormal state when the travel direction detection unit detects that the image center line intersects with the right boundary line segment, and the notification unit is capable of notifying information encouraging the pedestrian to turn right when the travel direction determination unit determines that the pedestrian is in the first abnormal state, and is capable of notifying information encouraging the pedestrian to turn left when the travel direction determination unit determines that the pedestrian is in the second abnormal state.

[0019] In this case, the direction of the pedestrian can be more easily corrected.

[0020] Furthermore, in the pedestrian travel direction determination system of the present invention, the notification unit can be a vibration notification unit having a main body and a plurality of vibrators, the main body having a longitudinal direction and capable of being detachably connected to one side in the longitudinal direction and the other side located opposite the one side to form a ring, the vibrators being attached in a line in the longitudinal direction of the main body on one side of the main body, and the vibration notification unit being capable of vibrating the vibrator located on one side of the ring-shaped main body when notifying information urging a turn to the right, and vibrating the vibrator on the other side located opposite the vibrator on one side when notifying information urging a turn to the left.

[0021] In this case, one side of the main body is connected to the other side to form the main body into a ring shape, and the main body can be attached to the pedestrian's body so that, for example, one side of the main body is positioned on the right side of the pedestrian's body, and the notification can be made more quickly than by voice.

[0022] In addition, the pedestrian travel direction determination system of the present invention includes a notification unit capable of notifying the result determined by the travel direction determination unit, and a route information setting unit capable of setting route information, which is information regarding the route between the starting point and the destination, based on information regarding the starting point and the destination, wherein the notification unit is capable of notifying of a change of direction based on the route information set by the route information setting unit, and the travel direction determination unit can be configured to determine that the pedestrian is changing direction if the travel direction detection unit detects that the boundary line segment and the image center line intersect within a predetermined time from when the notification unit notifies of the change of direction based on the route information.

[0023] In this case, when the notification unit notifies the pedestrian to change direction based on route information, for example, from the time when the notification unit notifies the pedestrian to turn right or left on the sidewalk until a predetermined time has elapsed, the travel direction determination unit determines that the pedestrian is in a state of changing direction rather than determining whether the pedestrian's direction is normal or not, so that the pedestrian can concentrate on changing direction.

[0024] Here, the "predetermined time" generally means the time it takes for a pedestrian to start and finish changing his / her direction while moving forward, and is, for example, 3 to 10 seconds.

[0025] In addition, in order to achieve the above-mentioned object, the pedestrian travel direction determination software of the present invention is pedestrian travel direction determination software for determining the direction of a pedestrian when traveling, and is software for causing an information processing device to function as a means including: an object detection unit capable of detecting a boundary line of an installed object that is an installed object in an captured image obtained by an imaging unit capable of imaging at least in front of the pedestrian, the boundary line having an image center line that passes through a center point of the captured image and extends in the vertical direction of the captured image, the boundary line extending at an angle with respect to the vertical direction of the captured image, in at least a lower region of the captured image; a travel direction detection unit capable of detecting whether the boundary line intersects with the image center line and detects whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line; and a travel direction determination unit capable of determining that the direction of the pedestrian is normal when the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line.

[0026] Here, the direction of travel detection unit can detect whether a boundary line intersects with an image center line, and can detect whether the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, and can detect the direction of a pedestrian relative to an installed object having an inclined boundary line in the captured image.

[0027] In addition, if the travel direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to the upper end of the boundary line than the lower end of the boundary line, the travel direction determination unit, which can determine that the pedestrian's orientation is normal, can confirm whether the pedestrian's orientation relative to an installed object having an inclined boundary line is normal. Effect of the Invention

[0028] The pedestrian travel direction determination system according to the present invention can determine a direction in which a pedestrian can travel in a safe and secure manner. The pedestrian travel direction determination software according to the present invention can determine a direction in which a pedestrian can travel in a safe and secure manner. [Brief description of the drawings]

[0029] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a pedestrian travel direction determination system to which the present invention is applied. [Diagram 2] 1A is a schematic diagram showing a first example of a captured image in a pedestrian travel direction determination system to which the present invention is applied, and FIG. 1B is a schematic diagram showing an example of an augmented reality captured image based on the captured image shown in FIG. 1A. [Diagram 3] FIG. 2(a) is a schematic diagram showing an example of an augmented reality captured image at the next stage of the augmented reality captured image shown in FIG. 2(b), and FIG. 2(b) is a schematic diagram showing an example of an augmented reality captured image including the result of a pedestrian's direction being determined by a direction determination unit of a pedestrian direction determination system to which the present invention is applied, based on the augmented reality captured image shown in FIG. 2(a). [Figure 4] 1 is a schematic diagram illustrating an example of a vibration notification unit included in a pedestrian travel direction determination system to which the present invention is applied. [Diagram 5] FIG. 1A is a schematic diagram showing a second example of an image captured in a pedestrian travel direction determination system to which the present invention is applied, and FIG. 1B is a schematic diagram showing an example of an augmented reality captured image including the result of a pedestrian's direction being determined by a travel direction determination unit of the pedestrian travel direction determination system to which the present invention is applied based on an augmented reality captured image of the captured image shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings for the purpose of understanding the present invention. FIG. 1 is a schematic diagram showing an example of the configuration of a pedestrian traveling direction determination system to which the present invention is applied.

[0031] Fig. 2(a) is a schematic diagram showing a first example of a captured image in a pedestrian travel direction determination system to which the present invention is applied, and Fig. 2(b) is a schematic diagram showing an example of an augmented reality captured image based on the captured image shown in Fig. 2(a).

[0032] Fig. 3(a) is a schematic diagram showing an example of an augmented reality captured image at a next stage of the augmented reality captured image shown in Fig. 2(b). Fig. 3(b) is a schematic diagram showing an example of an augmented reality captured image including a result of a pedestrian direction determined by a proceeding direction determination unit of a pedestrian proceeding direction determination system to which the present invention is applied, based on the augmented reality captured image shown in Fig. 3(a).

[0033] The pedestrian travel direction determination system 1 of the present invention shown in FIG. Here, the glasses device 2 has the shape of typical glasses, and the server 3 is capable of communicating with the glasses device 2 via the Internet 14. The glasses device 2 is worn by, for example, a visually impaired pedestrian. The glasses device 2 is a device having a general information processing function.

[0034] The glasses device 2 also includes an imaging camera 4 capable of capturing an image of at least the area ahead of the pedestrian. Here, the imaging camera 4 is an example of an imaging unit.

[0035] In addition, as long as the device has an imaging camera 4 capable of capturing an image at least in front of the pedestrian and is capable of communicating with the server 3 via the Internet 14, it does not necessarily have to be the glasses device 2; for example, a smartphone can be hung around the pedestrian's neck and placed at chest level to capture an image of the area in front of the pedestrian. In other words, the imaging direction of the imaging camera 4 is the direction in which the pedestrian is facing.

[0036] The glasses device 2 also includes a captured image transmitting unit 4A. Here, the captured image transmission unit 4A is capable of transmitting the captured images captured by the imaging camera 4 to the server 3 via the Internet .

[0037] The captured image also has an image centerline that passes through the center point of the captured image and extends in the vertical direction of the captured image.

[0038] On the other hand, the server 3 includes a captured image receiving unit 4B. Here, the captured image receiving section 4B is capable of receiving the captured image transmitted by the captured image transmitting section 4A.

[0039] The glasses device 2 also includes an object detection unit 5 . Here, the object detection unit 5 is capable of detecting an installed object, which is an object that has been installed, in the captured image.

[0040] Furthermore, the object detection unit 5 of the glasses device 2 can also detect the boundary line of the installed object. Moreover, this boundary line segment is a line segment based on the outer shape of the installation object, and is a straight line segment. Moreover, the boundary line segment detected by object detection unit 5 is a boundary line segment that extends at an angle with respect to the vertical direction of the captured image in at least the lower region of the captured image.

[0041] Here, "a boundary line segment that extends at an angle" means a straight line portion of a paved general sidewalk that appears to be at an angle in the captured image, a straight line portion of a crosswalk that appears to be at an angle, a straight line portion of a boundary block that appears to be at an angle, and a straight line portion of a curb that appears to be at an angle. Moreover, the "lower region of the captured image" refers to the lower half of the captured image.

[0042] Specific examples of installed objects include crosswalks, paved general sidewalks, boundary blocks lined up in the direction of a paved general sidewalk, pedestrian traffic lights, telegraph poles, and street lights. In addition, the boundary line of a general sidewalk and the boundary line of a boundary block are usually the same.

[0043] In addition, the center line of the captured image is displayed on the image. Furthermore, a rectangle is displayed at the location of the installed object detected in the captured image to indicate the presence of the installed object. In addition, a boundary line segment that is detected in the captured image and extends at an angle is also displayed at a corresponding location on the installed object that appears to be at an angle. An image in which the image center line, rectangles, inclined boundary lines, etc. are displayed is an "augmented reality captured image."

[0044] In this specification, an image in which an image center line, rectangle, inclined boundary line, etc. are displayed based on a captured image is referred to as an "augmented reality captured image," but an augmented reality captured image and a captured image are essentially the same. Therefore, the "image center line of the augmented reality captured image" and the "image center line of the captured image" are synonymous with each other.

[0045] As shown in FIG. 2(b), an image center line 22 that passes through a captured image center point 23, which is the center point of the captured image 20, and extends in the vertical direction of the captured image is displayed together with the captured image center point 23. Also, as shown in FIG. 2(b), crosswalk fragment rectangles 25A are displayed at locations of an installed object detected in the captured image 20 by the object detection unit 5, for example, a plurality of parallel white line areas of a crosswalk. Then, as shown in FIG. 3(a), the object detection unit 5 detects the entire area in which the multiple crosswalk fragment rectangles 25A are displayed as a single installed object, and a crosswalk existence area rectangle 25 large enough to encompass the multiple crosswalk fragment rectangles 25A is displayed in place of the crosswalk fragment rectangles 25A.

[0046] Furthermore, as shown in FIG. 2(b), a pedestrian traffic light presence area rectangle 29 is displayed at the location of an installed object, such as a pedestrian traffic light, detected in the captured image 20 by the object detection unit 5. Furthermore, as shown in FIG. 3(a), the inclined boundary lines of the crosswalk detected by the object detection unit 5 in the captured image 20, i.e., the left crosswalk boundary line 26 and the right crosswalk boundary line 27, are displayed at the inclined portion of the crosswalk.

[0047] Furthermore, the positions of rectangles, straight line segments, etc. are indicated in the coordinate system of the captured image. That is, in the coordinate system of the captured image, the upper left corner of the captured image is the origin of the image coordinate system, and the x coordinate value increases as you move from the origin in the horizontal direction of the image, i.e., toward the right of the image, and the y coordinate value increases as you move from the origin in the vertical direction of the image, i.e., toward the bottom of the image.

[0048] Here, the left crosswalk boundary line segment 26 is a boundary line segment whose upper end x-coordinate value in the coordinate system of the captured image 20 is greater than the lower end x-coordinate value. Further, the crosswalk right boundary line segment 27 is a boundary line segment whose upper end x-coordinate value in the coordinate system of the captured image 20 is smaller than the lower end x-coordinate value.

[0049] In this manner, the image in which the image center line 22, rectangles such as the crosswalk existence area rectangle 25, and inclined straight line segments such as the crosswalk left boundary line segment 26 are displayed is the augmented reality captured image 21.

[0050] Furthermore, the object detection unit 5 detects the left crosswalk boundary line segment 26, for example, in the following manner.

[0051] The image of the portion surrounded by the pedestrian crossing presence area rectangle 25 is copied and cut out, and the cut out image is binarized.

[0052] Then, pre-processing is performed to label a plurality of parallel white line regions. In other words, in the frame portion of the multiple parallel white line area, the three lines on the top and bottom sides are set to "0", and three pixels on the left and right sides are also set to "0", forcibly separating the inside from the frame portion. In addition, contraction and expansion processing is performed 0 to 2 times to remove areas consisting of only thin line segments of 1 to 2 pixels.

[0053] Next, by performing a labeling process on the binarized image that has been pre-processed with labeling, the separated parallel white line regions are labeled and a circumscribing rectangle that encloses the region can be obtained. From the obtained circumscribing rectangles, those whose length is equal to or greater than about 1 / 3 to half of the rectangular area width direction length and whose difference in the y direction is 2 or more are selected and extracted.

[0054] The selected and extracted rectangular data is expressed as follows: {(x1d[i],y1d[i])-(x2d[i],y2d[i])} Here, “i=0, 1, ...Nd-1” and “Nd” is the number of labels. In addition, these are assumed to have been subjected to sorting processing so that they are arranged in descending order of y1d[i] (from the bottom to the top of the image).

[0055] Then, the left boundary line is determined from the left edge data of the selected and extracted rectangular data. The data is expressed as follows: {(x1d[i],y1d[i])} Here, i = 0, 1, ...Nd-1.

[0056] In addition, the end point with the smallest x coordinate value is found, and the coordinate value at that point is set to (x0L, y0L). Next, the coordinate values ​​of a point whose x coordinate value is the same as "x0L" or is within a specified range from "x0L", whose y coordinate value is smaller than "y0L", and whose next point is further away from "x0L" than a specified value are defined as (x1L, y1L).

[0057] If the (x1L, y1L) point is the i1L-th point in the left boundary candidate point sequence, the direction angle p of the left boundary line connecting the i1L+1, Nd (number of labels)-th point (x1d[i], y1d[i]) and the (x1L, y1L) point is calculated using the following formula. p=angle(x1L,y1L,x1d[i],y1d[i])

[0058] Here, the direction angle is a clockwise map direction angle with north being 0 degrees. When the image center line of the captured image overlaps with the north azimuth, the azimuth angle is 0 degrees, and when it overlaps with the east azimuth, the azimuth angle is 90 degrees.

[0059] Then, it is determined whether the calculated direction angle is within the direction angle range [pLmin, pLmax] that is acceptable for points on the left boundary line segment, and for points that are within the range, the average direction angle p2L is calculated using the following formula. p2L=Σp「j」 / mL Here, "p"j" is an angle within the direction angle range [pLmin, pLmax] that is acceptable for a point on the left boundary line segment, and "mL" is the number of such angles.

[0060] The object detection unit 5 detects a line segment that satisfies this average direction angle p2L as the left boundary line segment 26 of the crosswalk.

[0061] Further, the object detection unit 5 detects the right crosswalk boundary line segment 27, for example, in the following manner.

[0062] The right boundary line is determined from the right edge data of the selected and extracted rectangular data. The data is expressed as follows: {(x1d[i],y1d[i])} Here, i = 0, 1, ...Nd-1.

[0063] In addition, the end point with the maximum x coordinate value is found, and the coordinate value at that point is set to (x0R, y0R). Next, the coordinate values ​​of a point whose x coordinate value is the same as "x0R" or is within a specified range from "x0R", whose y coordinate value is smaller than "y0R", and whose next point is further away from "x0R" than the specified value are defined as (x1R, y1R).

[0064] If the (x1R, y1R) point is the i1R-th point in the sequence of right boundary candidate points, the direction angle p of the right boundary line connecting the i1R+1, Nd (number of labels)-th point (x1d[i], y1d[i]) and the (x1R, y1R) point is calculated using the following formula. p=angle(x1R,y1R,x1d[i],y1d[i])

[0065] Here, the direction angle is a clockwise map direction angle with north being 0 degrees. When the image center line of the captured image overlaps with the north azimuth, the azimuth angle is 0 degrees, and when it overlaps with the west azimuth, the azimuth angle is 270 degrees.

[0066] Then, it is determined whether the calculated direction angle is within the direction angle range [pRmin, pRmax] that is acceptable for points on the right boundary line segment, and for points that are within the range, the average direction angle p2R is calculated using the following formula. p2R=Σp「j」 / mR Here, "p"j" is an angle within the direction angle range [pRmin, pRmax] that is acceptable for a point on the right boundary line segment, and "mR" is the number of such angles.

[0067] The object detection unit 5 detects a line segment that satisfies this average direction angle p2R as the right boundary line segment 27 of the pedestrian crossing.

[0068] The glasses device 2 also includes a traveling direction detection unit 6 . Here, the traveling direction detection unit 6 can detect whether or not the boundary lines, that is, the left pedestrian crossing boundary line 26 and the right pedestrian crossing boundary line 27, intersect with the image center line 22. Furthermore, the travel direction detection unit 6 can detect whether the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line segment 26 and the upper end of the right crosswalk boundary line segment 27 than to the lower end of the left crosswalk boundary line segment 26 and the lower end of the right crosswalk boundary line segment 27.

[0069] The glasses device 2 also includes a traveling direction determination unit 7 . Here, when the traveling direction detection unit 6 satisfies the following two requirements, the traveling direction determination unit 7 can determine that the direction of the pedestrian wearing the glasses device 2 is in the "normal state". (1) The traveling direction detection unit 6 detects that the left pedestrian crossing boundary line segment 26 and the right pedestrian crossing boundary line segment 27 do not intersect with the image center line 22; and (2) The direction of travel detection unit 6 detected that the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line portion 26 and the upper end of the right crosswalk boundary line portion 27 than to the lower end of the left crosswalk boundary line portion 26 and the lower end of the right crosswalk boundary line portion 27.

[0070] The glasses device 2 also includes a notification unit 8. Here, the notification unit 8 can notify the result of the determination made by the travel direction determination unit 7, for example, the determination result that the vehicle is in a "normal state." The notification method of the notification unit 8 may be, for example, at least one of notification by voice, notification by vibration, and notification by display on an image.

[0071] That is, as shown in FIG. 3(b), the image center line 22 does not intersect with the left crosswalk boundary line portion 26 and the right crosswalk boundary line portion 27, and further passes through a position closer to the upper end of the left crosswalk boundary line portion 26 and the upper end of the right crosswalk boundary line portion 27 than to the lower end of the left crosswalk boundary line portion 26 and the lower end of the right crosswalk boundary line portion 27. When the traveling direction detection unit 6 detects such a situation, the traveling direction determination unit 7 can determine that the orientation of the pedestrian wearing the eyewear device 2 is "normal." The notification unit 8 can notify the travel direction determination unit 7 of the “normal state” determination result by displaying a normal determination result symbol 24A, “◎”, on the augmented reality captured image 21, as shown in Figure 3(b).

[0072] At this time, the notification unit 8 can also notify the result of the determination made by the traveling direction determination unit 7 that the vehicle is in a "normal state" by, for example, a voice saying "You are moving in the correct direction."

[0073] The glasses device 2 also includes a detected object information transmitting section 5A. Here, the detected object information transmission unit 5A is capable of transmitting information regarding the installed object detected by the object detection unit 5, such as an augmented reality captured image 21 displaying an existence area rectangle surrounding at least a portion of the installed object, to the server 3 via the Internet 14.

[0074] On the other hand, server 3 has detected object information receiving unit 5B. Here, the detected object information receiving section 5B is capable of receiving information about the installed object transmitted from the glasses device 2. That is, the detected object information receiving unit 5B is capable of receiving the augmented reality captured image 21 that displays a presence area rectangle that surrounds at least a portion of the installed object.

[0075] Server 3 also has a detected object information database 9. Detected object information database 9 is an example of a detected object information recording section. Here, the detected object information database 9 is capable of recording installed object information, which is information relating to various objects that the object detection unit 5 can detect, that is, installed objects. Specifically, the installed object information is, for example, the name of the installed object and the external characteristics (shape, size, color, etc.) of the installed object.

[0076] The server 3 also includes an object identification unit 10. Here, the object identification unit 10 can identify the installed object by comparing the information of the installed object detected by the object detection unit 5 and received by the detected object information receiving unit 5B with the installed object information recorded in the detected object information database 9.

[0077] Specifically, the object identification unit 10 can identify an installed object by comparing the numerical values ​​of the rectangle of an installed object detected by the object detection unit 5, such as vertical and horizontal numerical values, with the installed object information recorded in the detected object information database 9, such as a vertical and horizontal numerical range. Here, the vertical value of a rectangle is the distance between the two horizontal lines of the rectangle, and the horizontal value of a rectangle is the distance between the two vertical lines of the rectangle, both of which are dimensional values ​​calculated by image analysis or the like.

[0078] Furthermore, when the object identification unit 10, for example, compares the numerical value of the crosswalk fragment rectangle 25A of the installed object detected by the object detection unit 5 with the numerical range recorded in the detected object information database 9 and identifies the installed object as a "white line of a crosswalk," it can identify the installed object in which the crosswalk existence area rectangle 25 is displayed as a "crosswalk" based on the fact that multiple crosswalk fragment rectangles 25A corresponding to the "white line of a crosswalk" are displayed.

[0079] The server 3 also includes a specific object information transmission unit 10A capable of transmitting information about the installation object identified by the object identification unit 10 to the glasses device 2 via the Internet 14.

[0080] On the other hand, the glasses device 2 includes a specific object information receiving unit 10B capable of receiving, from the server 3, information related to the installation object identified by the object identifying unit 10.

[0081] In addition, when the object identification unit 10 identifies the installed object as a “crosswalk,” the travel direction detection unit 6 can further detect whether the crosswalk arrival entrance line segment 28 intersects with the image center line 22. Here, the crosswalk arrival gate line segment 28 is the upper line segment of the crosswalk existence area rectangle 25, i.e., the horizontal line segment with the smallest y coordinate value of the crosswalk existence area rectangle 25 in the coordinate system of the captured image, whose x coordinate value is greater than the x coordinate value of the intersection of this horizontal line segment with the crosswalk left boundary line segment 26, or less than the x coordinate value of the intersection of this horizontal line segment with the crosswalk right boundary line segment 27.

[0082] In addition, when the object identification unit 10 identifies an installed object other than the installed object identified as a "crosswalk" as a "pedestrian traffic light," the travel direction detection unit 6 can further detect whether the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 in the coordinate system of the captured image is within a specified range.

[0083] Furthermore, the traveling direction determination unit 7 can further determine that the orientation of the pedestrian wearing the glasses device 2 is "further normal" when the traveling direction detection unit 6 satisfies the following two requirements. (1) The travel direction detection unit 6 detects that the crosswalk arrival entrance line segment 28 and the image center line 22 intersect; and (2) The travel direction detection unit 6 detects that the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 22 is within a predetermined range.

[0084] In addition, when the travel direction detection unit 6 detects that the image center line 22 intersects with the left crosswalk boundary line segment 26, the travel direction determination unit 7 can determine that the pedestrian's direction is in a "first abnormal state." In addition, when the travel direction detection unit 6 detects that the image center line 22 intersects with the crosswalk right boundary line segment 27, the travel direction determination unit 7 can determine that the pedestrian's direction is in a "second abnormal state."

[0085] Furthermore, when the traveling direction determination unit 7 determines that the first abnormal state exists, the notification unit 8 can provide information encouraging the driver to turn right. Furthermore, the notification unit 8 can notify information encouraging a left turn when the traveling direction determination unit 7 determines that the second abnormal state exists.

[0086] Furthermore, the pedestrian travel direction determination system 1 of the present invention can include a vibration notification unit 30 as shown in FIG. That is, FIG. 4 is a schematic diagram showing an example of a vibration notification unit included in a pedestrian traveling direction determination system to which the present invention is applied. Here, the vibration notification unit 30 is an example of the notification unit 8, and realizes a notification method by vibration.

[0087] The vibration notification unit 30 has a main body 31 and a plurality of vibrators 32. The main body 31 is configured in the shape of a belt.

[0088] That is, the main body 31 has a longitudinal direction, and one side in the longitudinal direction and the other side positioned opposite to the one side can be detachably connected to form a ring shape. Furthermore, the multiple vibrating bodies 32 are attached to one surface of the main body 31 in a line in the longitudinal direction of the main body 31 .

[0089] When the travel direction determination unit 7 determines that the first abnormal state exists, the notification unit 8 can provide information encouraging the driver to turn right. In this case, the vibration notification unit 30, which is an example of the notification unit 8, can vibrate a vibrating body 32 located on one side of the annular main body 31 (e.g., the upper side in Figure 4). In addition, when the travel direction determination unit 7 determines that a second abnormal state exists, the notification unit 8 can issue information encouraging the driver to turn left. In this case, the vibration notification unit 30 can vibrate the vibrator 32 on the other side (e.g., the lower side in Figure 4) of the annular main body 31, which is located opposite the vibrator 32 on one side of the main body 31.

[0090] The glasses device 2 also includes a sound transmitting unit 11 . Here, the voice transmitting unit 11 is capable of transmitting a voice uttered by a pedestrian wearing the glasses device 2 to the server 3 via the Internet 14.

[0091] The server 3 also includes a voice receiving unit 12 . Here, the sound receiving unit 12 is capable of receiving the sound transmitted from the glasses device 2 .

[0092] The server 3 also includes a route information setting unit 13 . Here, the route information setting unit 13 can set route information, which is information about a route between a departure point and a destination, based on the voice related to the departure point and the destination received by the voice receiving unit 12.

[0093] Furthermore, when setting route information, the route information setting unit 13 can of course set the route information based on map information or the like as appropriate. In this case, the server 3 has a map information recording unit (map information database) capable of recording map information, and can set route information based on the map information recorded in this map information database, or can set route information based on map information recorded in a map information database held by another server other than the server 3 while communicating with the other server via the Internet.

[0094] For example, based on a voice received by the voice receiving unit 12 saying "From XX Park to the sea side of Kokura Station," the route information setting unit 13 sets route information for a pedestrian to travel, with "XX Park" as the starting point and "the sea side of Kokura Station" as the destination.

[0095] Furthermore, the route information setting unit 13 does not necessarily have to set route information based on the voice received by the voice receiving unit 12 . If the departure point and destination are known, the route information setting unit 13 can set route information based on, for example, characters or images. In this case, the glasses device 2 may have a function for transmitting characters and images to the server 3, for example, an e-mail function.

[0096] The server 3 also includes a route information transmitting unit 13A. Here, the route information transmission unit 13A is capable of transmitting the route information set by the route information setting unit 13 to the glasses device 2 via the Internet .

[0097] On the other hand, the glasses device 2 includes a route information receiving unit 13B. Here, the route information receiving unit 13B is capable of receiving route information transmitted from the server 3.

[0098] Furthermore, the notification unit 8 can notify the route information received by the route information receiving unit 13B. The notification unit 8 can also notify a change of direction based on the route information set by the route information setting unit 13.

[0099] In addition, when the traveling direction detection unit 6 detects, for example, that the crosswalk right boundary line 27 and the image center line 22 intersect within a predetermined time, for example, 8 seconds, from the time when the notification unit 8 notifies of a change of direction based on the route information, the traveling direction determination unit 7 can determine that the pedestrian is changing direction.

[0100] The pedestrian travel direction determination system 1 of the present invention does not necessarily have to include a detected object information database 9, i.e., a detected object information recording unit, nor does it necessarily have to include an object identification unit 10. Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction detection unit 6 does not necessarily have to be able to detect whether the pedestrian crossing arrival gate line segment 28 and the image center line 22 intersect.

[0101] However, if the pedestrian travel direction determination system 1 of the present invention is equipped with a detected object information database 9 and an object identification unit 10, and is capable of identifying an installed object as a crosswalk and having the travel direction detection unit 6 detect whether the crosswalk arrival entrance line segment 28 intersects with the image center line 22, this is preferable, as it makes it easier to confirm whether the direction of the pedestrian relative to the crosswalk is normal.

[0102] Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction detection unit 6 does not necessarily have to be able to detect whether the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 22 is within a specified range. Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction determination unit 7 does not necessarily have to be able to determine that the pedestrian's direction is also normal when the travel direction detection unit 6 detects that the pedestrian crossing arrival gate line segment 28 and the image center line 22 intersect and detects that the shortest distance between the coordinates of a pedestrian traffic light and the image center line 22 is within a predetermined range.

[0103] However, if the travel direction detection unit 6 is capable of detecting whether the crosswalk arrival gate line segment 28 and the image center line 22 intersect, and in addition is capable of detecting whether the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 is within a specified range, and if the travel direction detection unit 6 detects that the crosswalk arrival gate line segment 28 and the image center line 22 intersect and that the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 is within a specified range, then it is preferable if the travel direction determination unit 7 is also capable of determining that the direction of the pedestrian is normal, because this allows for more accurate confirmation of whether the direction of the pedestrian, in particular with respect to the crosswalk, is normal.

[0104] Furthermore, in the pedestrian travel direction determination system 1 of the present invention, the travel direction determination unit 7 does not necessarily have to be able to determine that the pedestrian's orientation is in the "first abnormal state" when the travel direction detection unit 6 detects that the image center line 22 intersects with the left pedestrian crossing boundary line segment 26, and it does not necessarily have to be able to determine that the pedestrian's orientation is in the "second abnormal state" when the travel direction detection unit 6 detects that the image center line 22 intersects with the right pedestrian crossing boundary line segment 27. In the pedestrian travel direction determination system 1 of the present invention, the notification unit 8 does not necessarily have to be able to notify information encouraging a turn to the right when the travel direction determination unit 7 determines that the first abnormal state is occurring, and does not necessarily have to be able to notify information encouraging a turn to the left when the travel direction determination unit 7 determines that the second abnormal state is occurring.

[0105] However, if the travel direction determination unit 7 is capable of determining that a first abnormal state and a second abnormal state exist, and if the travel direction determination unit 7 determines that the first abnormal state exists, the notification unit 8 is capable of providing information encouraging the pedestrian to turn right, and if the travel direction determination unit 7 determines that the second abnormal state exists, the notification unit 8 is capable of providing information encouraging the pedestrian to turn left, this is preferable because it makes it easier to correct the pedestrian's direction than if the travel direction determination unit 7 is not configured in this way.

[0106] In addition, the pedestrian travel direction determination system 1 of the present invention does not necessarily have to be equipped with a route information setting unit 13, and the notification unit 8 does not necessarily have to be able to notify of a change of direction based on the route information set by the route information setting unit 13. Furthermore, the travel direction determination unit 7 does not necessarily have to be able to determine that the pedestrian is changing direction when the travel direction detection unit 6 detects that the left pedestrian crossing boundary line 26 or the right pedestrian crossing boundary line 27 has intersected with the image center line 22 within a predetermined time from when the notification unit 8 notifies of a change of direction based on the route information.

[0107] However, when the pedestrian travel direction determination system 1 of the present invention is equipped with a route information setting unit 13 and the travel direction detection unit 6 detects that the left pedestrian crossing boundary line 26 or the right pedestrian crossing boundary line 27 has intersected with the image center line 22 within a predetermined time from when the notification unit 8 notifies of a change of direction based on the route information, if the travel direction determination unit 7 can determine that the pedestrian is changing direction, the travel direction determination unit 7 determines that the pedestrian is changing direction rather than determining whether the pedestrian's direction is normal or not until the predetermined time has elapsed, which is preferable because it allows the pedestrian to concentrate on changing direction.

[0108] Furthermore, pedestrian travel direction determination software of the present invention for implementing such pedestrian travel direction determination system 1 is software for causing an information processing device to function as a glasses device 2 having the functions described above.

[0109] Moreover, the example shown in FIG. 1 is just an example, and the functions of the glasses device 2 and the server 3 are not limited to this. For example, the server 3 may have an object detection unit 5, a traveling direction detection unit 6, and a traveling direction determination unit 7. In this case, the pedestrian traveling direction determination software of the present invention for executing the pedestrian traveling direction determination system 1 can cause the information processing device to function as a server having the same functions as the glasses device 2.

[0110] In addition, if the server 3 has an object detection unit 5, a traveling direction detection unit 6, and a traveling direction determination unit 7, the object detection unit 5, the traveling direction detection unit 6, and the traveling direction determination unit 7 perform processing based on the captured image received by the captured image receiving unit 4B of the server 3.

[0111] Next, an example of a flow of determining a pedestrian's traveling direction using the pedestrian traveling direction determination system of the present invention will be described.

[0112] When a pedestrian starts walking on the sidewalk, the pedestrian transmits to the server 3 information required to set the route along which the pedestrian plans to walk. That is, when a pedestrian starts walking on the sidewalk, he or she speaks out loud information about the starting point, which is the location where the pedestrian is currently located, and information about the destination, and the voice transmission unit 11 of the glasses device 2 transmits the voice related to the starting point and destination to the server 3 via the Internet 14.

[0113] For example, if a pedestrian wants to walk to Kokura Station to return home after playing at "XX Park," the pedestrian will vocalize route information such as "From XX Park to the sea side of Kokura Station" to set the route he or she will take.

[0114] Then, the voice transmitting unit 11 of the glasses device 2 transmits the voice "From XX Park to the sea side of Kokura Station" to the server 3 via the Internet 14. On the other hand, the voice receiving unit 12 of the server 3 receives the voice "From XX Park to the sea side of Kokura Station" transmitted from the glasses device 2.

[0115] Then, based on the voice received by the voice receiving unit 12, "From XX Park to the sea side of Kokura Station," the route information setting unit 13 of the server 3 sets route information for the pedestrian to travel, with "XX Park" as the starting point and "the sea side of Kokura Station" as the destination.

[0116] In addition, the route information transmitting unit 13A of the server 3 transmits the route information set by the route information setting unit 13, "from XX Park to the sea side of Kokura Station," to the glasses device 2 via the Internet . On the other hand, the route information receiving unit 13B of the glasses device 2 receives the route information transmitted from the server 3.

[0117] Then, the notification unit 8 of the glasses device 2 notifies the pedestrian of the route information "from XX Park to the sea side of Kokura Station" received by the route information receiving unit 13B.

[0118] Also, for example, a pedestrian leaves XX Park and stands at the intersection in front of XX Park, relying on the tactile paving blocks. At this time, the imaging camera 4 of the glasses device 2 captures an image ahead of the pedestrian, and a first example of an image captured by the imaging camera 4 is shown in FIG. 2(a). In addition, an example of an augmented reality captured image 21 when the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 20 shown in Figure 2(a) is shown in Figures 2(b), 3(a) and 3(b).

[0119] In addition, the set route information includes crossing the crosswalk at the intersection in front of XX Park.

[0120] Furthermore, the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 20 . Furthermore, the object detection unit 5 of the glasses device 2 also detects the boundary line of the placed object that extends at an angle. When the object detection unit 5 detects an installed object or a boundary line segment, a rectangle or a boundary line segment is displayed at the location of the detected installed object or at a corresponding location of the installed object that appears tilted.

[0121] Furthermore, as shown in Figures 2(b), 3(a) and 3(b), the augmented reality captured image 21 displays an image center line 22 that passes through the captured image center point 23, which is the center point of the augmented reality captured image 21, and extends vertically through the captured image. Here, since the augmented reality captured image 21 is a display of lines and rectangles on the screen based on the captured image 20, the center point of the augmented reality captured image 21 and the center point of the captured image 20 are the same.

[0122] 2(b), a crosswalk fragment rectangle 25A is displayed in each of a plurality of parallel white line regions of the crosswalk, which are installation objects detected by the object detection unit 5. Then, the object detection unit 5 detects the entire area in which the multiple crosswalk fragment rectangles 25A are displayed as a single installed object, and as shown in Figure 3(a), a crosswalk existence area rectangle 25 large enough to encompass the multiple crosswalk fragment rectangles 25A is displayed in place of the crosswalk fragment rectangles 25A.

[0123] Furthermore, as shown in FIG. 2(b), a pedestrian traffic light presence area rectangle 29 is displayed on a pedestrian traffic light, which is an installed object detected by the object detection unit 5. Furthermore, as shown in FIG. 3(a), the inclined boundary segments of the crosswalk detected by the object detection unit 5, that is, the left crosswalk boundary segment 26 and the right crosswalk boundary segment 27, are displayed at an inclined portion of the crosswalk.

[0124] Then, the traveling direction detection unit 6 detects whether the left pedestrian crossing boundary line 26 and the right pedestrian crossing boundary line 27 intersect with the image center line 22 . The travel direction detection unit 6 also detects whether the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line portion 26 and the upper end of the right crosswalk boundary line portion 27 than to the lower end of the left crosswalk boundary line portion 26 and the lower end of the right crosswalk boundary line portion 27.

[0125] As shown in Figures 2(b), 3(a), and 3(b), the image center line 22 does not intersect with either the left crosswalk boundary line portion 26 or the right crosswalk boundary line portion 27, so the travel direction detection unit 6 detects that the image center line 22 does not intersect with either the left crosswalk boundary line portion 26 or the right crosswalk boundary line portion 27. Also, as shown in Figures 2(b), 3(a), and 3(b), the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line portion 26 and the upper end of the right crosswalk boundary line portion 27 than to the lower end of the left crosswalk boundary line portion 26 and the lower end of the right crosswalk boundary line portion 27, so the travel direction detection unit 6 detects that the image center line 22 passes through that position.

[0126] Then, the traveling direction determining unit 7 determines that the direction of the pedestrian wearing the glasses device 2 is "normal." Furthermore, the notification unit 8 notifies the pedestrian of the determination result "normal state" determined by the travel direction determination unit 7 by displaying a determination result symbol on the augmented reality captured image 21, or by a voice, for example, saying "You are moving in the correct direction."

[0127] In this way, the pedestrian travel direction determination system 1 of the present invention can determine the direction of a pedestrian without identifying an installed object in the captured image 20, as long as it can detect the obliquely extending boundary line of the installed object.

[0128] Separately from this process of determining the travel direction, the detected object information transmitter 5A transmits to the server 3 via the Internet 14 an augmented reality captured image 21 that displays a presence area rectangle that surrounds at least a portion of the installed object.

[0129] Meanwhile, in the server 3 that receives such an augmented reality captured image 21, the object identification unit 10 compares the information of the installed object detected by the object detection unit 5 with the installed object information recorded in the detected object information database 9 to identify the installed object.

[0130] Furthermore, the object identification unit 10 identifies the installed object in which the crosswalk existence area rectangle 25 is displayed as a “crosswalk”, and the identified object information transmission unit 10A transmits information on the identified installed object to the glasses device 2. Therefore, when the notification unit 8 notifies by voice the result of the determination made by the traveling direction determination unit 7 that the vehicle is in a "normal state," the notification unit 8 also notifies the content of the identified installed object, for example, "You are proceeding in the correct direction on the crosswalk."

[0131] In addition, since the object identification unit 10 has identified the installed object as a "pedestrian crossing," the travel direction detection unit 6 also detects whether the pedestrian crossing arrival gate line segment 28 and the image center line 22 intersect.

[0132] In addition, the object identification unit 10 identifies an installed object in which the pedestrian traffic light existence area rectangle 29 is displayed, that is, an installed object other than the installed object that the object identification unit 10 identified as a "crosswalk," as a "pedestrian traffic light," and the identified object information transmission unit 10A transmits information regarding the identified installed object to the glasses device 2. In addition, the direction of travel detection unit 6 further detects whether the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image, i.e., the coordinates of the detected installed object identified as a "pedestrian traffic light", and the image center line 22 is within a predetermined range, i.e., greater than "0" and less than or equal to the shortest distance between a virtual center line passing through the center of the crosswalk and the coordinates of the pedestrian traffic light.

[0133] As shown in Figures 2(b), 3(a), and 3(b), the image center line 22 intersects with the crosswalk arrival gate line segment 28, and since the image center line 22 passes near the center of the crosswalk, the travel direction detection unit 6 detects that the crosswalk arrival gate line segment 28 intersects with the image center line 22, and that the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 in the coordinate system of the captured image is within a specified range.

[0134] Therefore, in this case, the traveling direction determining unit 7 determines that the direction of the pedestrian wearing the glasses device 2 is "still in a normal state." Then, the notification unit 8 notifies the pedestrian of the determination result determined by the travel direction determination unit 7 that the state is "still normal" by displaying a normal determination result symbol 24 such as "◎" on the augmented reality captured image 21, as shown in Figure 3(b), or by a voice message, for example, "You are moving sufficiently in the correct direction."

[0135] Furthermore, for example, if the image center line 22 shown in Figures 2(b), 3(a), and 3(b) is located a little further to the right and intersects with the pedestrian traffic light existence area rectangle 29, the image center line 22 intersects with the crosswalk arrival entrance line segment 28, but is not within the specified range because the shortest distance between the coordinates of the pedestrian traffic light and the image center line 22 is "0". Therefore, in this case, the travel direction determination unit 7 does not determine that the state is "still normal," so the notification unit 8 notifies the user by displaying a determination result symbol such as "O" on the augmented reality captured image 21, or by providing a voice message saying, "You are moving in the correct direction to some extent."

[0136] On the other hand, if route information is set for a pedestrian starting from "XX Park" and aiming at "the sea side of Kokura Station," when the pedestrian leaves XX Park and stands at the intersection in front of XX Park, relying on the tactile paving blocks, it is also anticipated that they will stand facing diagonally to the crosswalk.

[0137] FIG. 5(a) is a schematic diagram showing a second example of an image captured in the pedestrian traveling direction determination system to which the present invention is applied. FIG. 5(b) is a schematic diagram showing an example of an augmented reality captured image including the result of a pedestrian direction determination unit of a pedestrian direction determination system to which the present invention is applied determining the direction of a pedestrian based on the augmented reality captured image of the captured image shown in FIG. 5(a).

[0138] FIG. 5(a) shows a second example of an image captured by the imaging camera 4 of the glasses device 2 when a pedestrian stands facing diagonally toward the crosswalk. FIG. 5(b) shows an example of an augmented reality captured image 41 when the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 40 shown in FIG. 5(a).

[0139] In addition, the set route information includes crossing the crosswalk at the intersection in front of XX Park.

[0140] Furthermore, the object detection unit 5 of the glasses device 2 detects an installed object in the captured image 40 . Furthermore, the object detection unit 5 of the glasses device 2 also detects the boundary line of the placed object that extends at an angle. When the object detection unit 5 detects an installed object or a boundary line segment, a rectangle or a boundary line segment is displayed at the location of the detected installed object or at a corresponding location of the installed object that appears tilted.

[0141] Furthermore, as shown in FIG. 5(b), the augmented reality captured image 41 displays an image center line 42 that passes through a captured image center point 43, which is the center point of the augmented reality captured image 41, and extends in the vertical direction of the captured image. Here, the center point of the augmented reality captured image 41 and the center point of the captured image 40 are the same.

[0142] In addition, the object detection unit 5 detects the entire area in which multiple crosswalk fragment rectangles (not shown) are displayed as a single installed object, and a crosswalk existence area rectangle 45 large enough to encompass the multiple crosswalk fragment rectangles (not shown) is displayed in place of the crosswalk fragment rectangles (not shown), as shown in FIG. 5(b).

[0143] Furthermore, as shown in FIG. 5(b), a pedestrian traffic light presence area rectangle 49 is displayed on a pedestrian traffic light, which is an installed object detected by the object detection unit 5. Furthermore, as shown in FIG. 5(b), a pedestrian crossing right boundary line segment 47, which is a boundary line segment of the pedestrian crossing that extends at an angle and that has been detected by object detection unit 5, is displayed at an inclined portion of the pedestrian crossing. Here, the object detection unit 5 has not detected the left boundary line of the pedestrian crossing.

[0144] Then, the traveling direction detection unit 6 detects whether or not the crosswalk right boundary line segment 47 and the image center line 42 intersect. The traveling direction detection unit 6 also detects whether the image center line 42 passes through a position closer to the upper end of the right crosswalk boundary line segment 47 than to the lower end of the right crosswalk boundary line segment 47 .

[0145] As shown in FIG. 5(b), the image center line 42 intersects with the crosswalk right boundary line segment 47, so the travel direction detection unit 6 detects that the crosswalk right boundary line segment 47 and the image center line 42 intersect.

[0146] Also, as shown in FIG. 5(b), the image center line 42 passes through a position closer to the bottom end of the crosswalk right boundary line segment 47 than to the top end of the crosswalk right boundary line segment 47; in other words, the image center line 42 does not pass through a position closer to the top end of the crosswalk right boundary line segment 47 than to the bottom end of the crosswalk right boundary line segment 47. Therefore, the traveling direction detection unit 6 detects that the image center line 42 does not pass through a position closer to the upper end of the right crosswalk boundary line segment 47 than to the lower end of the right crosswalk boundary line segment 47 .

[0147] In this case, the travel direction determination unit 7 determines that the direction of the pedestrian wearing the eyeglass device 2 is "not in a normal state," but since the travel direction detection unit 6 detects that the image center line 42 intersects with the crosswalk right boundary line segment 47, the travel direction determination unit 7 determines that the direction of the pedestrian is "in a second abnormal state."

[0148] In addition, the notification unit 8 notifies the pedestrian of the judgment result determined by the travel direction judgment unit 7 that "a second abnormal state exists" by displaying an abnormality judgment result symbol 44 such as an "X" on the augmented reality captured image 41, as shown in Figure 5(b). Furthermore, if the travel direction determination unit 7 determines that the pedestrian's direction is in the "second abnormal state," the notification unit 8 further notifies the pedestrian by providing information encouraging the pedestrian to turn left, for example, by a voice message saying, "You are moving to the right. Please turn a little further to the left." Furthermore, in this case, a vibration notification unit 30 having a ring-shaped main body 31 worn like a belt around the pedestrian's waist vibrates a vibrator 32 located on one side of the ring-shaped main body 31, i.e., on the pedestrian's left flank, to notify the pedestrian of information encouraging them to turn left.

[0149] On the other hand, when the traveling direction detection unit 6 detects that the image center line intersects with the left boundary line of the crosswalk, the traveling direction determination unit 7 determines that the direction of the pedestrian is in the "first abnormal state."

[0150] In this case, the notification unit 8 notifies the driver by issuing information encouraging the driver to turn right, for example, by a voice saying, "You are heading left. Please turn a little further to the right." Furthermore, in this case, a vibration notification unit 30 having a ring-shaped main body 31 worn like a belt around the pedestrian's waist vibrates a vibrator 32 located on the opposite side of the ring-shaped main body 31, i.e., on the pedestrian's right side, to notify the pedestrian of information encouraging them to turn right.

[0151] Separately from this process of determining the travel direction, detected object information transmitter 5A transmits to server 3 via Internet 14 an augmented reality captured image 41 that displays a presence area rectangle that surrounds at least a portion of the installed object.

[0152] In addition, the object identification unit 10 identifies the installed object in which the crosswalk existence area rectangle 45 is displayed as a “crosswalk”, and the identified object information transmission unit 10A transmits information about the identified installed object to the glasses device 2. Therefore, when the notification unit 8 notifies by voice the result of the determination made by the travel direction determination unit 7 that "this is a second abnormal state", the notification unit 8 also notifies the content of the identified installed object, for example, "You are proceeding to the right on the crosswalk."

[0153] In addition, since the object identification unit 10 has identified the installed object as a "pedestrian crossing," the travel direction detection unit 6 also detects whether the pedestrian crossing arrival gate line segment 48 and the image center line 42 intersect.

[0154] In addition, the object identification unit 10 identifies an installed object in which the pedestrian traffic light existence area rectangle 49 is displayed, that is, an installed object other than the installed object that the object identification unit 10 identified as a "crosswalk," as a "pedestrian traffic light," and the identified object information transmission unit 10A transmits information regarding the identified installed object to the glasses device 2. Furthermore, the traveling direction detection unit 6 further detects whether or not the shortest distance between the coordinates of the pedestrian traffic light in the coordinate system of the captured image and the image center line 42 is within a predetermined range.

[0155] As shown in FIG. 5(b), image center line 42 does not intersect with crosswalk arrival gate line segment 48, and image center line 42 intersects with crosswalk right boundary line segment 47, and passes through a position closer to the lower end of crosswalk right boundary line segment 47 than to the upper end of crosswalk right boundary line segment 47. Therefore, the travel direction detection unit 6 detects that the crosswalk arrival entrance line segment 48 and the image center line 42 do not intersect, and detects that the shortest distance between the coordinates of the pedestrian traffic light and the image center line 42 is not within a specified range.

[0156] On the other hand, a variant is also envisaged in which a pedestrian vocalizes route information such as "from X△ Park to △△ Building" to set the route they will take, and the route information setting unit 13 of the server 3 sets route information with "X△ Park" as the starting point and "△△ Building" as the destination. The route information that has been set is to proceed to the right without crossing the crosswalk at the intersection in front of XX Park.

[0157] In this case, immediately after the pedestrian leaves the park, he or she faces in roughly the same direction as the crosswalk, as shown in Figure 2(a). Then, the notification unit 8 notifies pedestrians of a rightward turn based on the route information set by the route information setting unit 13.

[0158] When a pedestrian receives a notice of a rightward turn, he or she begins to move forward by turning their body to the right in order to proceed to the right. At this time, the captured image in front of the pedestrian changes from that shown in FIG. 2(a) to that shown in FIG. 5(a).

[0159] Furthermore, the object detection unit 5 of the glasses device 2 detects the left boundary line of the crosswalk and the right boundary line of the crosswalk. Further, the traveling direction detection unit 6 detects whether or not the left boundary line of the crosswalk and the right boundary line of the crosswalk intersect with the image center line.

[0160] Therefore, when a pedestrian turns his / her body to the right and starts moving forward, the image center line intersects with the right boundary line of the crosswalk, and the moving direction detection unit 6 detects that the right boundary line of the crosswalk and the image center line intersect.

[0161] Then, when the traveling direction detection unit 6 detects that a boundary line, for example, the right boundary line of the crosswalk, intersects with the image center line within a predetermined time, for example, 8 seconds, from when the notification unit 8 notifies of a rightward turn based on the route information, the traveling direction determination unit 7 determines that the pedestrian is changing direction.

[0162] For example, from when a pedestrian is informed to turn right until 8 seconds have elapsed, even if the image center line intersects with a boundary line such as the right boundary line of the pedestrian crossing, the traveling direction determination unit determines that the pedestrian is in a state of changing direction rather than determining whether the pedestrian's direction is normal or not, so that the pedestrian can concentrate on changing direction.

[0163] After turning right just before the crosswalk, pedestrians follow the route information and proceed along the general sidewalk alongside the roadway. Then, the object detection unit 5 of the glasses device 2 detects the boundary line segments of the installed object that extend at an angle, i.e., the straight line segments of the curb that appear to be inclined between the roadway and the general sidewalk, and the straight line segments of the boundary blocks that appear to be inclined between the sidewalk and the property, as the left boundary line segment of the general sidewalk and the right boundary line segment of the general sidewalk.

[0164] Furthermore, the traveling direction detection unit 6 detects whether or not the left boundary line segment of the general sidewalk and the right boundary line segment of the general sidewalk intersect with the image center line. In addition, the travel direction detection unit 6 detects whether the image center line passes through a position closer to the upper end of the general sidewalk left boundary line portion and the upper end of the general sidewalk right boundary line portion than the lower end of the general sidewalk left boundary line portion and the lower end of the general sidewalk right boundary line portion.

[0165] Then, when the travel direction detection unit 6 detects that the image center line does not intersect with the general sidewalk left boundary line segment and the general sidewalk right boundary line segment, and further detects that the image center line passes through a position closer to the upper end of the general sidewalk left boundary line segment and the upper end of the general sidewalk right boundary line segment than the lower end of the general sidewalk left boundary line segment and the lower end of the general sidewalk right boundary line segment, the travel direction determination unit 7 determines that the pedestrian's direction is "normal."

[0166] As described above, the pedestrian travel direction determination system 1 of the present invention is equipped with a travel direction detection unit 6, which can detect whether the boundary segments, i.e., the left crosswalk boundary segment 26 and the right crosswalk boundary segment 27, intersect with the image center line 22, and further can detect whether the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary segment 26 and the upper end of the right crosswalk boundary segment 27 than to the lower end of the left crosswalk boundary segment 26 and the lower end of the right crosswalk boundary segment 27.Therefore, it is possible to detect the orientation of a pedestrian relative to an installed object having an inclined boundary segment in the captured image.

[0167] Furthermore, the pedestrian travel direction determination system 1 of the present invention is equipped with a travel direction determination unit 7, which can determine that the pedestrian's orientation is "normal" when the travel direction detection unit 6 detects that the left crosswalk boundary line segment 26 and the right crosswalk boundary line segment 27 do not intersect with the image center line 22 and when the travel direction detection unit 6 detects that the image center line 22 passes through a position closer to the upper end of the left crosswalk boundary line segment 26 and the upper end of the right crosswalk boundary line segment 27 than to the lower ends of the left crosswalk boundary line segment 26 and the right crosswalk boundary line segment 27. Therefore, it is possible to confirm whether the orientation of the pedestrian with respect to an installed object having an inclined boundary line segment is normal or not.

[0168] Therefore, the pedestrian travel direction determination system 1 of the present invention can determine a direction in which a pedestrian can travel in a safe and secure manner.

[0169] In addition, the pedestrian travel direction determination software of the present invention can cause an information processing device to function as a glasses device 2 having functions similar to the travel direction detection unit 6 and travel direction determination unit 7 provided in the pedestrian travel direction determination system 1 of the present invention.

[0170] Therefore, the pedestrian travel direction determination software of the present invention can determine a direction in which a pedestrian can travel in a safe and secure manner. [Explanation of symbols]

[0171] 1. Pedestrian direction determination system 2 Eyeglasses device 3 Server 4. Imaging camera 4A Image capture transmission unit 4B Image receiving unit 5. Object detection section 5A Detected object information transmitter 5B Detected object information receiving unit 6 Direction detection unit 7 Direction of travel determination section 8 Announcement Department 9. Detected object information database 10 Object identification part 10A Specific object information transmission unit 10B Specific object information receiving unit 11 Audio transmitter 12 Audio receiving section 13 Route information setting section 13A Route information transmission unit 13A Route information receiver 14. Internet 20 Captured images 21 Augmented reality images 22 Image centerline 23 Center point of captured image 24 Normal Judgment Result Code 25 Pedestrian crossing area rectangle 25A Crosswalk Rectangular Fragment 26 Left boundary line of pedestrian crossing 27 Right boundary line of pedestrian crossing 28 Pedestrian crossing arrival gate line segment 29 Pedestrian traffic light area rectangle 30 Vibration notification section 31 Main unit 32 Vibration Body 40 Captured images 41 Augmented reality images 42 Image centerline 43 Image center point 44 Abnormality Judgment Result Code 45 Pedestrian crossing area rectangle 47 Right boundary line of pedestrian crossing 48 Crosswalk Arrival Line 49 Pedestrian traffic light area rectangle

Claims

1. An imaging unit capable of imaging at least an area in front of a pedestrian; an object detection unit capable of detecting a boundary line of an installed object, which is an object installed in an image captured by the imaging unit and has an image center line passing through a center point of the captured image and extending in a vertical direction of the captured image, the boundary line extending at an angle with respect to the vertical direction of the captured image in at least a lower region of the captured image; a travel direction detection unit capable of detecting whether the boundary line segment intersects with the image center line and detecting whether the image center line passes through a position closer to an upper end of the boundary line segment than to a lower end of the boundary line segment; and a traveling direction determination unit that can determine that the pedestrian is facing normally when the traveling direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to an upper end of the boundary line than to a lower end of the boundary line. Pedestrian direction determination system.

2. The boundary line segment is a left boundary line segment whose upper end x coordinate value in the coordinate system of the captured image is greater than the lower end x coordinate value, and a right boundary line segment whose upper end x coordinate value in the coordinate system of the captured image is smaller than the lower end x coordinate value. The pedestrian travel direction determination system according to claim 1 .

3. a detection object information recording unit capable of recording installation object information which is information regarding the installation object; an object identification unit capable of identifying an installed object by comparing information on the installed object detected by the object detection unit with the installed object information recorded by the detected object information recording unit; When the object identification unit identifies the installed object as a pedestrian crossing, the travel direction detection unit is further capable of detecting whether the image center line intersects with a pedestrian crossing arrival gate line segment that is a portion of a horizontal line segment having a vertical coordinate value in a coordinate system of the captured image of a pedestrian crossing existence area rectangle corresponding to the entire pedestrian crossing, the y coordinate value of which increases toward the bottom of the image and has the smallest y coordinate value, which is greater than the x coordinate value of the intersection with the left boundary line segment or smaller than the x coordinate value of the intersection with the right boundary line segment, and that is displayed in the captured image when the object detection unit detects the installed object. The pedestrian travel direction determination system according to claim 2 .

4. the travel direction detection unit is further capable of detecting, when an installed object other than an installed object identified by the object identification unit as a pedestrian crossing is identified as a pedestrian traffic light by the object identification unit, whether or not a shortest distance between the coordinates of the pedestrian traffic light in a coordinate system of the captured image and the image center line is within a predetermined range; The travel direction determination unit can further determine that the pedestrian's direction is normal when the travel direction detection unit detects that the crosswalk arrival entrance line segment and the image center line intersect and detects that the shortest distance between the coordinates of the pedestrian traffic light and the image center line is within a predetermined range. The pedestrian travel direction determination system according to claim 3 .

5. A notification unit capable of notifying the result determined by the traveling direction determination unit, the traveling direction determination unit is capable of determining that the orientation of the pedestrian is in a first abnormal state when the traveling direction detection unit detects that the image center line intersects with the left boundary line segment, and is capable of determining that the orientation of the pedestrian is in a second abnormal state when the traveling direction detection unit detects that the image center line intersects with the right boundary line segment, The notification unit is capable of notifying information encouraging a turn to the right when the traveling direction determination unit determines that a first abnormal state exists, and is capable of notifying information encouraging a turn to the left when the traveling direction determination unit determines that a second abnormal state exists. The pedestrian travel direction determination system according to claim 2 .

6. The notification unit is a vibration notification unit having a main body and a plurality of vibrators, the main body having a longitudinal direction, and capable of forming a ring shape by detachably connecting one side of the longitudinal direction to the other side located opposite to the one side, the vibrators being attached in a line in the longitudinal direction of the main body on one side of the main body, the vibration notification unit being capable of vibrating the vibrator located on one side of the ring-shaped main body when notifying information urging a right turn, and being capable of vibrating the vibrator on the other side located opposite to the vibrator on one side when notifying information urging a left turn. The pedestrian travel direction determination system according to claim 5 .

7. a notification unit capable of notifying a result determined by the traveling direction determination unit; A route information setting unit capable of setting route information, which is information relating to a route between the departure point and the destination, based on information relating to the departure point and the destination; The notification unit is capable of notifying a change of direction based on the route information set by the route information setting unit, The traveling direction determination unit is capable of determining that the pedestrian is in a turning state when the traveling direction detection unit detects that the boundary line segment intersects with the image center line within a predetermined time from when the notification unit notifies the pedestrian of a turn based on the route information. The pedestrian travel direction determination system according to claim 1 .

8. A pedestrian movement direction determination software for determining a direction of a pedestrian when moving, comprising: Information processing equipment, an object detection unit capable of detecting a boundary line of an installed object, the boundary line extending at an angle with respect to the vertical direction of the captured image, in at least a lower region of the captured image, in a captured image obtained by capturing an image of at least an area in front of a pedestrian, the captured image having an image center line passing through a center point of the captured image and extending in a vertical direction of the captured image; a travel direction detection unit capable of detecting whether the boundary line segment intersects with the image center line and detecting whether the image center line passes through a position closer to an upper end of the boundary line segment than to a lower end of the boundary line segment; a traveling direction determination unit capable of determining that the pedestrian is facing normally when the traveling direction detection unit detects that the boundary line does not intersect with the image center line and that the image center line passes through a position closer to an upper end of the boundary line than to a lower end of the boundary line; Pedestrian direction determination software to function as a means including

Images