Driving support method, driving support device, computer program, and driving support system

Abstract

To provide a traveling support method capable of providing an operator who performs at least one of remote monitoring and remote operation of a moving body with information required for the remote monitoring or remote operation of the moving body.SOLUTION: A traveling support method for supporting traveling of a moving body capable of performing autonomous traveling includes the steps of: detecting a person existing in an image on the basis of the image photographed by a camera mounted on the moving body; calculating distance to the person from the moving body on the basis of the detection result of the person; calculating the height of the person on the basis of the detection result of the person and the distance to the person; and imparting a degree of risk regarding the person to the person on the basis of the height.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a driving assistance method, a driving assistance device, a computer program, and a driving assistance system. [Background technology]

[0002] Non-patent document 1 discloses a system that detects other moving objects, people, or fallen objects on the road around an autonomously moving robot from images captured by a camera mounted on the robot, and notifies the detection results to an operator who performs at least one of remotely monitoring and remotely operating the robot.

[0003] Non-Patent Document 2 discloses a system in which an image captured by a camera mounted on an autonomous vehicle is transmitted to a server in a control center, and the server detects objects from the image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Panasonic obtains road use permit for fully remote delivery robot”, [online], April 18, 2022, IoTNEWS, R-Gene Co., Ltd. IoTNEWS Division, [Retrieved November 13, 2023], Internet<URL: https: / / iotnews.jp / smart-city / 200568 / > [Non-patent document 2] “NEC Develops Learning-Based Media Transmission Control Technology to Enhance Remote Monitoring of Cars and Other Devices Using AI”, [online], January 12, 2021, DIGITAL SHIFT TIMES, Digital Holdings Co., Ltd., [Retrieved November 13, 2023], Internet<URL: https: / / digital-shift.jp / flash_news / FN210112_8> Summary of the Invention [Problem to be solved by the invention]

[0005] However, although the system in Non-Patent Document 1 can notify an operator of detection results, it is difficult for the operator to instantly determine which object in the video the notified detection result refers to. This makes it difficult for one operator to monitor multiple moving objects in parallel.

[0006] In the system of Non-Patent Document 2, when the communication bandwidth of the transmission path between the autonomous vehicle and the server becomes narrow, the video must be transmitted from the autonomous vehicle to the server after processing such as reducing the video quality. As a result, the server must perform object detection from low-quality video, which may result in missed or erroneous detection of objects.

[0007] The present disclosure has been made in consideration of the above circumstances, and aims to provide a driving assistance method, a driving assistance device, a computer program, and a driving assistance system that can provide an operator who performs at least one of the tasks of remotely monitoring and remotely operating a moving body with information necessary for remotely monitoring or remotely operating the moving body. [Means for solving the problem]

[0008] A driving assistance method according to one aspect of the present disclosure is a driving assistance method for assisting a moving body capable of autonomous driving to drive, and includes the steps of: detecting a person present in an image taken by a camera mounted on the moving body based on the image; calculating the distance from the moving body to the person based on the person detection result; calculating the height of the person based on the person detection result and the distance to the person; and assigning a higher level of danger to the person the shorter the height of the person.

[0009] The present invention can be realized not only as a driving assistance method having such characteristic processing units, but also as a driving assistance device having processing units corresponding to such characteristic steps, or as a computer program for causing a computer to function as the driving assistance device, or as a semiconductor integrated circuit that realizes part or all of the driving assistance device, or as a driving assistance system including the driving assistance device. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an operator who performs at least one of remote monitoring and remote operation of a mobile object with information necessary for remote monitoring or remote operation of the mobile object. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a driving assistance system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of a driving assistance device according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram for explaining the attribute determination process performed by the attribute determination unit. [Figure 4] FIG. 4 is a diagram illustrating an example of risk level table data. [Figure 5] FIG. 5 is a diagram showing a display example of the display screen. [Figure 6] FIG. 6 is a diagram showing a display example of the display screen. [Figure 7] FIG. 7 is a diagram showing a display example of the display screen. [Figure 8] FIG. 8 is a diagram showing a display example of the display screen. [Figure 9] FIG. 9 is a flowchart showing an example of the operation of the driving support device. DETAILED DESCRIPTION OF THE INVENTION

[0012] [Summary of the embodiments of the present disclosure] First, an outline of the embodiments of the present disclosure will be listed and described. (1) A driving assistance method according to one embodiment of the present disclosure is a driving assistance method for assisting a moving body capable of autonomous driving to drive, and includes the steps of: detecting a person present in an image captured by a camera mounted on the moving body based on the image; calculating a distance from the moving body to the person based on the person detection result; calculating a height of the person based on the person detection result and the distance to the person; and assigning a risk level to the person based on the height.

[0013] With this configuration, a risk level can be assigned to a person according to their height. For example, a shorter person is assigned a higher risk level because the impact of a moving object coming into contact with the person is greater. This allows the operator to remotely monitor moving objects that are capturing images of high-risk people intensively, change the moving object's travel route to avoid coming into contact with high-risk people, or temporarily suspend the moving object if the risk level is particularly high. This allows the operator to be provided with information necessary for remote monitoring or remote operation of moving objects.

[0014] (2) In the above (1), the step of assigning to the person may include a step of classifying the person into a first class based on the height, and a step of assigning the risk level corresponding to the first class to the person classified into the first class.

[0015] According to this configuration, for example, a person whose height is below a predetermined threshold can be classified into a class of preschoolers or lower elementary school students, and the person can be assigned a risk level corresponding to that class.

[0016] (3) In the above (1) or (2), the step of assigning to the person may include a step of classifying a plurality of people who are crowded together into a second class based on the heights of the plurality of people and the detection results, and a step of assigning the risk level corresponding to the second class to each of the plurality of people classified into the second class and at least one of the group of the plurality of people classified into the second class.

[0017] With this configuration, for example, when multiple people, such as elementary or junior high school students, whose heights are within a specified range, are crowded together, the multiple people can be classified into the same class, and each person or group of people can be assigned a risk level corresponding to that class.

[0018] (4) In any of (1) to (3) above, the driving assistance method may further include a step of calculating the speed of the person based on the result of detecting the person, and a step of calculating a predicted time until the moving object collides with the person based on the distance to the person and the speed of the person, and the step of assigning the risk level to the person may further assign the risk level to the person based on the predicted time.

[0019] For example, assuming that a moving object will collide with a person, the shorter the predicted time until the moving object collides with the person, the greater the urgency of the need to intervene in the operation of the moving object. With this configuration, the shorter the predicted time for a person, the higher the risk level assigned to that person. This allows the operator to make an emergency stop of the moving object before it collides with the person. Therefore, the operator can be provided with information necessary for remote monitoring or remote operation of the moving object.

[0020] (5) In any of (1) to (4) above, the method may further include a step of notifying an operator who performs at least one of remote monitoring and remote operation of the moving object based on the level of danger of the person present in the image.

[0021] According to this configuration, the operator can appropriately remotely monitor or remotely control the moving object in response to the notification based on the risk level.

[0022] (6) A driving assistance device according to another embodiment of the present disclosure is a driving assistance device for assisting an autonomously driven mobile body in driving, and includes a detection unit that detects a person present in an image captured by a camera mounted on the mobile body based on the image; a distance calculation unit that calculates the distance from the mobile body to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; and a risk level assignment unit that assigns a risk level to the person based on the height.

[0023] This configuration includes processing units corresponding to the characteristic steps in the driving support method described above, and therefore provides the same functions and effects as the driving support method described above.

[0024] (7) A computer program according to another embodiment of the present disclosure is a computer program for causing a computer to function as a driving assistance device for assisting the driving of an autonomously driving mobile body, and causes the computer to function as a detection unit that detects a person present in an image taken by a camera mounted on the mobile body based on the image, a distance calculation unit that calculates the distance from the mobile body to the person based on the detection result of the person, a height calculation unit that calculates the height of the person based on the detection result of the person and the distance to the person, and a risk level assignment unit that assigns a risk level to the person based on the height.

[0025] According to this configuration, the computer can function as the driving assistance device described above, thereby achieving the same effects and advantages as the driving assistance device described above.

[0026] (8) A driving assistance system according to another embodiment of the present disclosure is a driving assistance system for assisting a vehicle capable of autonomous driving to drive, comprising: a driving assistance device; and a display control device connected to the driving assistance device via a network. The driving assistance device includes: a detection unit that detects a person present in an image captured by a camera mounted on the vehicle based on the image; a distance calculation unit that calculates the distance from the vehicle to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; a risk level assignment unit that assigns a risk level to the person based on the height; and a risk level information transmission unit that transmits risk level information based on the risk level assigned to the person to the display control device. The display control device displays a screen based on the risk level information received from the driving assistance device.

[0027] With this configuration, a risk level can be assigned to a person according to their height. For example, a shorter person is assigned a higher risk level because the impact of contact with a moving object is greater. The display control device displays a screen based on risk level information based on the risk level assigned to the person. This allows an operator to remotely monitor, intensively, moving objects capturing images of high-risk people, change the moving object's route to avoid contact with high-risk people, or temporarily suspend the moving object if the risk level is particularly high. This allows the operator to be provided with information necessary for remote monitoring or remote operation of moving objects.

[0028] [Details of the embodiments of the present disclosure] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are examples and do not limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims are components that can be added arbitrarily. Furthermore, each figure is a schematic diagram and is not necessarily a precise illustration.

[0029] The same components are denoted by the same reference numerals, and their functions and names are also the same, so their explanations will be omitted where appropriate.

[0030] [Overall configuration of driving assistance system] FIG. 1 is a block diagram showing a hardware configuration of a driving assistance system according to an embodiment of the present disclosure. The driving assistance system 100 includes driving assistance devices 1A, 1B, and 1C and a driving control device 2.

[0031] The driving support devices 1A, 1B, 1C and the driving control device 2 are connected to each other via a network 3. The driving assistance devices 1A, 1B, and 1C are devices that assist the driving of a moving body. The moving body includes, for example, a mobile robot that can autonomously drive on a driving path such as a scheduled road, or an automobile that can autonomously drive.

[0032] The driving assistance devices 1A, 1B, and 1C are mounted on different moving bodies and calculate the degree of danger to a person present around the moving body. The driving assistance devices 1A, 1B, and 1C each transmit danger level information indicating the calculated degree of danger to the driving control device 2 via the network 3. Here, the degree of danger to a person indicates, for example, the magnitude of the impact that the moving body will have on the person when the moving body collides with the person. The impact on the person includes, for example, physical or mental pain, such as injury to the person due to the collision. Furthermore, the degree of danger to a person indicates the length of time until the moving body will affect the person. For example, the shorter the predicted time until the moving body collides with the person, the greater the urgency of intervention in the driving of the moving body, and therefore the greater the degree of danger is assigned.

[0033] The driving control device 2 is installed in a monitoring center or the like that monitors the driving of the mobile body, and is operated by an operator that monitors the driving of the mobile body. The driving control device 2 receives risk level information from the driving support devices 1A, 1B, and 1C via the network 3. The driving control device 2 displays an image based on the received risk level information on a display screen such as a display (not shown) connected to itself.

[0034] 1 shows three driving assistance devices 1A, 1B, and 1C, but the number of driving assistance devices 1 is not limited to three. Hereinafter, when there is no need to distinguish between the driving assistance devices 1A, 1B, and 1C, they will be referred to as driving assistance devices 1. The driving control device 2 displays an image based on the risk level information received from the multiple driving assistance devices 1 on the display screen of one or more displays.

[0035] An operator monitoring the driving of the mobile object looks at the display screen and intervenes in the driving of the mobile object equipped with the driving assistance device 1 that transmitted the risk level information indicating a high risk level. For example, the operator transmits a control signal to the mobile object via the driving control device 2 to control the braking or steering of the mobile object. Upon receiving the control signal, the mobile object performs control based on the control signal, prioritizing autonomous driving.

[0036] [Configuration of driving assistance device 1] FIG. 2 is a block diagram showing a functional configuration of the driving assistance device 1 according to the embodiment of the present disclosure.

[0037] The driving assistance device 1 is connected by wire or wirelessly to a camera 4 mounted on the moving body. The camera 4 is an example of a sensor mounted on the moving body and outputs detected data. The camera 4, for example, captures an area ahead of the moving body in the traveling direction, and outputs video data of that area (hereinafter simply referred to as "video"). However, the area captured by the moving body is not limited to the area ahead of the moving body, but may also be an area to the side or behind the moving body. The video is composed of multiple image data (hereinafter simply referred to as "images") in a time series.

[0038] The driving assistance device 1 includes an image acquisition unit 11, a person area detection unit 12, a distance calculation unit 13, a height calculation unit 14, an attribute determination unit 15, a person area tracking unit 16, a speed calculation unit 17, a risk calculation unit 18, and a risk information transmission unit 19.

[0039] The image acquisition unit 11 acquires an image of the area around the moving object from the camera 4. Here, it is assumed that the image acquisition unit 11 acquires an image of the area in front of the moving object.

[0040] Based on the video acquired by the video acquisition unit 11, the person area detection unit 12 detects people present in each image included in the video. That is, the person area detection unit 12 detects the area of ​​the person included in the image. For example, the person area detection unit 12 inputs the image into a learning model and acquires information on the position and size of the circumscribing rectangle of the person from the learning model. The position of the circumscribing rectangle is, for example, the coordinates of the upper left corner of the circumscribing rectangle, and the size of the circumscribing rectangle is, for example, the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the circumscribing rectangle. The learning model is, for example, a convolution neural network (CNN) or a transformer. Using images containing various types of people as training data, learning of the position and size of people progresses using a machine learning method such as deep learning, and the parameters of the learning model are determined.

[0041] Distance calculation unit 13 calculates the distance from the moving object to the person based on the detection result of the person detected by person area detection unit 12. That is, information on the position and size of the circumscribing rectangle of the person in the image is obtained from person area detection unit 12. Distance calculation unit 13 calculates the distance d from the moving object (camera 4) to the person based on the following equation 1. d=H / tan(V_FOV / 2) / (2y / NpxV-1) …(Formula 1)

[0042] Here, H is the height from the ground to the mounting position of the camera 4. V_FOV is the vertical angle of view of the camera 4. y is the number of pixels from the bottom of the circumscribing rectangle of the person to the top of the screen. NpxV is the total number of pixels in the vertical direction of the image.

[0043] Height calculation unit 14 calculates the height of a person based on the distance to the person and the person detection result. That is, height calculation unit 14 acquires distance d to the person from distance calculation unit 13, and acquires information on the size of a rectangular area including the person from person area detection unit 12. Height calculation unit 14 calculates height h of the person based on the distance d and the information on the size of the rectangular area. For example, height calculation unit 14 calculates height h by substituting distance d and size sy into a relational expression that indicates the relationship between height h, distance d, and the size of the rectangular area in the y-axis direction (hereinafter referred to as "size sy").

[0044] Even for the same person, the size sy decreases as the distance d increases. Also, even for the same distance d, the size sy increases as the person becomes taller. Therefore, the relational expression indicates that, for example, height h is inversely proportional to distance d and proportional to size sy.

[0045] The height calculation unit 14 may calculate the height h from the distance d and the size sy by referring to height table data that indicates in advance the relationship between the distance d, the size sy, and the height h. The height table data is stored, for example, in a memory (not shown) of the driving assistance device 1.

[0046] The attribute determining unit 15 determines the attribute of the person based on the person area detected by the person area detecting unit 12 and the height of the person calculated by the height calculating unit 14.

[0047] For example, the attribute determining unit 15 assigns attribute 1 "preschooler / lower grade elementary school student" (hereinafter referred to as "attribute 1") to a person whose height is 120 cm or less.

[0048] Furthermore, the attribute determination unit 15 detects people whose height is greater than 120 cm and less than 150 cm. When there are multiple detected people, the attribute determination unit 15 groups people who are close to each other into a group. The grouping method will be described later. The attribute determination unit 15 assigns attribute 2 "group of elementary and junior high school students" (hereinafter referred to as "attribute 2") to multiple people who have been grouped into the same group.

[0049] FIG. 3 is a diagram illustrating the attribute determination process performed by the attribute determination unit 15. FIG. 3 shows an example of an image captured on a road on which a moving object is traveling. The image includes persons 51 to 53 walking on the road. Note that in FIG. 3 and in FIGS. 5 to 8 described below, each person is shown by a schematic figure. The areas of persons 51 to 53 are shown by circumscribing rectangles 61 to 63, respectively. It is assumed that person 51 is 120 cm or shorter in height. Therefore, the attribute determination unit 15 assigns attribute 1 to person 51.

[0050] Assume that the heights of person 52 and person 53 are greater than 120 cm and equal to or less than 150 cm. Therefore, the attribute determination unit 15 determines whether person 52 and person 53 are located close to each other. As an example, the horizontal distance in real space (the x-axis direction when projected onto the image) between the centers of the circumscribing rectangle 62 of person 52 and the circumscribing rectangle 63 of person 53 is defined as dx. The attribute determination unit 15 estimates the direction in which person 52 exists from the center position of the circumscribing rectangle 62 in the image. The attribute determination unit 15 also acquires the distance d to person 52 from the distance calculation unit 13, and estimates the position of person 52 in real space based on the estimated direction in which person 52 exists and the distance d. The attribute determination unit 15 also estimates the position of person 53 in real space in the same way as person 52. The attribute determination unit 15 calculates the distance dx from the estimated positions of people 52 and 53. The distance in the depth direction between person 52 and person 53 is defined as dy. The attribute determination unit 15 calculates the absolute value of the difference between the distance d to person 52 and the distance d to person 53 calculated by the distance calculation unit 13 as the distance dy. If the distance dx is equal to or less than a predetermined first threshold and the distance dy is equal to or less than a predetermined second threshold, the attribute determination unit 15 groups the person 51 and the person 52 into the same group. In addition, the attribute determination unit 15 assigns attribute 2 to the person 51 and the person 52.

[0051] When there are three or more people who are greater than 120 cm and less than 150 cm, the attribute determination unit 15 performs the above process of determining for each person whether they are in close proximity to other people, and groups people who are in close proximity to any person in the group into the same group.

[0052] Referring back to FIG. 2 , the person region tracking unit 16 tracks the person region detected by the person region detection unit 12. For example, the person region tracking unit 16 assigns the same label to circumscribing rectangles that are close to each other between two temporally consecutive images. Specifically, the person region tracking unit 16 calculates the distance between the centers of the circumscribing rectangles between the two images, and if the calculated distance is equal to or less than a predetermined third threshold, determines that the two circumscribing rectangles are close to each other and assigns the same label to the two circumscribing rectangles. Note that, when a label has already been assigned to the first circumscribing rectangle between a first circumscribing rectangle and a second circumscribing rectangle to which the same label is to be assigned, the person region tracking unit 16 assigns the same label to the second circumscribing rectangle as the label assigned to the first circumscribing rectangle. Furthermore, when labels have not been assigned to both the first circumscribing rectangle and the second circumscribing rectangle to which the same label is to be assigned, the person region tracking unit 16 assigns the same new label to the two circumscribing rectangles. The person region tracking unit 16 sequentially labels such circumscribing rectangles between two temporally consecutive images that make up the video. As a result, people included in circumscribing rectangles that have the same label are determined to be the same person and are tracked. The person region tracking unit 16 may also use a tracking model such as BoT-SORT.

[0053] The speed calculation unit 17 calculates the speed of each person based on the tracking result of the person area by the person area tracking unit 16 and the distance from the moving object to the person calculated by the distance calculation unit 13. In other words, the speed calculation unit 17 calculates the amount of change in distance between the image of the frame to be processed and the image of the same person (a person included in a circumscribing rectangle with the same label) between the image of the frame being processed and the image a predetermined number of frames before that frame. The speed calculation unit 17 calculates the relative speed v of the person with respect to the moving object by dividing the calculated amount of change in distance by the time corresponding to the predetermined number of frames.

[0054] The danger level calculation unit 18 calculates the danger level for each person based on the distance from the moving body to the person calculated by the distance calculation unit 13, the relative speed of the person calculated by the speed calculation unit 17, and the attributes of the person calculated by the attribute determination unit 15.

[0055] Specifically, the risk calculation unit 18 calculates, for each person, a predicted time until the moving object collides with the person (hereinafter referred to as "predicted collision time"), assuming that the moving object collides with the person. The predicted collision time is the time until the distance between the moving object and the person becomes 0. In other words, the risk calculation unit 18 calculates the predicted collision time by dividing the distance d from the moving object to the person by the relative speed v of the person. Next, the risk calculation unit 18 calculates the risk based on the predicted collision time and attributes of each person by referring to predetermined risk table data. The risk table data is stored, for example, in a memory (not shown) of the driving assistance device 1.

[0056] FIG. 4 is a diagram showing an example of risk level table data. The risk level table data shows the relationship between the predicted collision time and attributes and the risk level. The risk level is, for example, an integer between 0 and 3, with larger values ​​indicating higher risk levels. For example, if the predicted collision time is less than 10 seconds and a person has attribute 1, the risk level is the highest, 3. Also, if the predicted collision time is 30 seconds or more but less than 60 seconds and a person has attribute 2, the risk level is the lowest, 0.

[0057] The danger level information transmission unit 19 generates danger level information including the danger level for the person calculated by the danger level calculation unit 18 and the position coordinates of the person, and transmits the generated danger level information to the driving control device 2. Here, the position coordinates of the person are, for example, the position and size of a circumscribing rectangle of the person. This allows the danger level information transmission unit 19 to notify an operator who performs at least one of remote monitoring and remote operation of a mobile object based on the danger level of a person present in an image.

[0058] The risk level information transmitting unit 19 transmits the risk level information together with the compressed video obtained by compressing the video acquired by the video acquiring unit 11. The risk level information transmitting unit 19 may transmit the compressed video together with the risk level information, or may transmit the risk level information as data separate from the compressed video.

[0059] The driving control device 2 receives the compressed video and the risk level information from the driving assistance device 1. The driving control device 2 displays the risk level information on the display screen of the display together with the video obtained by decompressing the compressed video.

[0060] 5 to 8 are diagrams showing examples of the display screen. The display screen includes images of the area ahead of the moving object captured by the camera 4. Each image includes a person 51, and a circumscribing rectangle 61 of the person 51 is displayed superimposed. The driving control device 2 also displays the degree of danger and attributes of the person 51 in the upper left corner of the image.

[0061] The driving control device 2 varies the display mode of the danger level display area (for example, the color or pattern of the display area) so that the operator can easily visually recognize the danger level.

[0062] [Operation procedure of driving assistance device 1] FIG. 9 is a flowchart showing an example of the operation of the driving support device 1. The driving assistance device 1 acquires an image of the area ahead of the moving object from the camera 4 (step S1).

[0063] The driving support device 1 detects a human region present in each image included in the acquired video (step S2).

[0064] The driving assistance device 1 calculates the distance d from the moving object to the person based on the detected person area (step S3).

[0065] The driving assistance device 1 calculates the height h of the person from the distance d and the size of the person area in the y-axis direction (step S4).

[0066] The driving assistance device 1 determines the attributes of the person based on the detected person area and the height h of the person (step S5). The driving assistance device 1 tracks the detected person area (step S6).

[0067] The driving assistance device 1 calculates the relative speed v of the person based on the tracking result of the person area and the distance d (step S7).

[0068] The driving assistance device 1 calculates the degree of danger to the person based on the distance d, the relative speed v, and the attributes of the person (step S8).

[0069] The driving assistance device 1 creates risk level information including the risk level and the position and size of the person area, and transmits it to the driving control device 2 (step S9).

[0070] The driving assistance device 1 performs, for example, the processes from step S1 to step S9 for one image, and repeatedly executes these processes every time the image is updated.

[0071] As described above, according to this embodiment, a risk level can be assigned to a person according to their height. A higher risk level is assigned to shorter people because the impact of contact with a moving object is greater. This allows the operator to remotely monitor moving objects that are capturing images of high-risk people intensively, change the moving object's travel route to avoid contact with high-risk people, or temporarily suspend the moving object if the risk level is particularly high. Therefore, the operator can be provided with information necessary for remote monitoring or remote operation of moving objects.

[0072] Furthermore, assuming that a moving object will collide with a person, the shorter the predicted time to collision, the greater the urgency with which the moving object must intervene. With this configuration, the shorter the predicted time to collision with a person, the higher the danger level assigned to that person. This allows the operator to bring the moving object to an emergency stop before it collides with the person.

[0073] <Variation 1> In the above embodiment, the distance calculation unit 13 calculates the distance d to the person using the camera parameters, but the method for calculating the distance d is not limited to this.

[0074] For example, the distance calculation unit 13 can calculate the distance d to the person using a software library capable of performing distance estimation from the video captured by the camera 4. Such a software library uses, for example, a learning model determined by machine learning of a large number of two-dimensional images. The learning model can accept a two-dimensional image as input and output a depth map in which each pixel indicates the distance to the object.

[0075] <Variation 2> Furthermore, the distance calculation unit 13 may use a sensor other than the camera 4 to calculate the distance to the person. For example, the distance calculation unit 13 may use millimeter wave radar to detect the distance from the moving body to the person, or may use LiDAR (Light Detection and Ranging) to detect the distance. The millimeter wave radar or LiDAR is installed at a position on the moving body where it can emit millimeter waves in the direction of travel of the moving body.

[0076] The distance calculation unit 13 acquires the position of the circumscribing rectangle of the person from the person area detection unit 12. The distance calculation unit 13 calculates the direction in which the person exists from the position of the circumscribing rectangle, and acquires the distance from the millimeter wave radar or LiDAR to the person located in that direction.

[0077] <Variation 3> In the above-described embodiment, the attribute determination unit 15 groups multiple people into the same group based on the distance between the people. Alternatively, the attribute determination unit 15 may group multiple people into the same group based on the distance d from the moving object to the people and the relative speed v of the people.

[0078] For example, the attribute determination unit 15 compares the distance dA and relative speed vA to person A with the distance dB and relative speed vB to person B. If the state in which |dA-dB| is equal to or less than a predetermined fourth threshold and |vA-vB| is equal to or less than a predetermined fifth threshold continues for a predetermined time or more, the attribute determination unit 15 groups person A and person B into the same group.

[0079] <Variation 4> The attribute determining unit 15 may group a plurality of people using an image processing technique. For example, the attribute determination unit 15 estimates the skeleton of a person by processing the image. The attribute determination unit 15 determines whether people are holding hands from the position of the skeleton, and groups people who are determined to be holding hands into the same group.

[0080] <Variation 5> Furthermore, the attribute determination unit 15 may group people who are having a conversation into the same group. For example, the attribute determination unit 15 recognizes the line of sight and mouth movements of each person by processing the image. If, based on the recognition result, people are looking at each other and their mouths are moving alternately, the attribute determination unit 15 determines that the people are having a conversation and groups them into the same group.

[0081] The attribute determination unit 15 may also determine whether people are having a conversation based on the distance between the rectangular areas and the movement of their mouths. For example, if the distance dx shown in Fig. 3 is equal to or less than a predetermined sixth threshold and the mouths of the people are moving alternately, the attribute determination unit 15 determines that the people are having a conversation and groups them into the same group.

[0082] [Note] Each process (each function) in the above-described embodiments is realized by a processing circuit including one or more processors. The processing circuit may be configured as an integrated circuit or the like that combines one or more memories, various analog circuits, and various digital circuits in addition to the one or more processors. The one or more memories store programs (instructions) that cause the one or more processors to execute each of the processes. The one or more processors may execute each of the processes according to the programs read from the one or more memories, or according to logic circuits pre-designed to execute each of the processes. The processor may be various processors suitable for computer control, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). Note that the physically separated processors may cooperate with each other to execute each of the processes. For example, the processors mounted on a plurality of physically separated computers may cooperate with each other to execute the processes via a network such as a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, etc. The program may be installed into the memory from an external server device or the like via the network, or may be distributed in a state stored in a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a semiconductor memory, and installed into the memory from the recording medium. Furthermore, at least some of the above-described embodiments and modifications may be combined in any manner.

[0083] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0084] 1. Driving support device 1A Driving assistance device 1B Driving support device 1C Driving support device 2. Driving control device 3 Network 4. Camera 11 Video acquisition unit 12 Person area detection unit (detection unit) 13 Distance calculation unit 14 Height calculation section 15 Attribute determination unit (risk level assignment unit) 16 Person area tracking unit 17 Speed ​​calculation section 18 Risk calculation unit (risk assignment unit) 19 Risk Information Transmission Unit 51 People 52 People 53 People 61 circumscribed rectangle 62 circumscribed rectangle 63 circumscribed rectangle 100 Driving Assistance System

Claims

1. A driving assistance method for assisting driving of a moving body capable of autonomous driving, comprising: detecting a person present in an image based on the image captured by a camera mounted on the moving object; calculating a distance from the moving object to the person based on the result of the person detection; calculating a height of the person based on the detection result of the person and the distance to the person; and assigning to the person a degree of danger related to the person that indicates the magnitude of the impact that the moving body will have on the person if the moving body collides with the person, based on the person's height.

2. A driving assistance method for assisting driving of an autonomously driving mobile body, comprising: detecting a person present in an image based on the image captured by a camera mounted on the moving object; calculating a distance from the moving object to the person based on the result of the person detection; calculating a height of the person based on the detection result of the person and the distance to the person; assigning a risk level to the person based on the height; The driving assistance method includes: calculating a speed of the person based on the result of the person detection; and calculating a predicted time until the moving object collides with the person based on the distance to the person and the speed of the person, The step of assigning a risk level to the person further assigns the risk level to the person, the risk level indicating the length of time until the moving object will affect the person, based on the predicted time.

3. The step of assigning to the person includes: classifying the person into a first class based on the height; 3. The driving assistance method according to claim 1, further comprising the step of: assigning the degree of danger corresponding to the first class to the person classified into the first class.

4. The step of assigning to the person includes: classifying the people who are crowded together into a second class based on the heights of the people and the detection results; and assigning the degree of danger corresponding to the second class to at least one of each of the plurality of people classified into the second class and a group of the plurality of people classified into the second class.

5. 3. The driving assistance method according to claim 1, further comprising the step of notifying an operator who performs at least one of remote monitoring and remote operation of the moving object based on the degree of danger of the person present in the image.

6. A driving assistance device for assisting driving of an autonomously driving mobile body, a detection unit that detects a person present in an image captured by a camera mounted on the moving object, based on the image; a distance calculation unit that calculates a distance from the moving object to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; and a danger level assigning unit that assigns a danger level to the person based on the person's height, the danger level indicating the magnitude of the impact that the moving body will have on the person if the moving body collides with the person.

7. A driving assistance device for assisting driving of an autonomously driving mobile body, comprising: a detection unit that detects a person present in an image captured by a camera mounted on the moving object, based on the image; a distance calculation unit that calculates a distance from the moving object to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; a risk level assigning unit that assigns a risk level regarding the person to the person based on the height, The driving assistance device a speed calculation unit that calculates the speed of the person based on the result of the person detection; a predicted time calculation unit that calculates a predicted time until the moving object collides with the person based on the distance to the person and the speed of the person, The danger level assigning unit further assigns the danger level to the person, indicating the length of time until the moving object will affect the person, based on the predicted time.

8. A computer program for causing a computer to function as a driving assistance device for assisting driving of an autonomously driven mobile body, The computer a detection unit that detects a person present in an image captured by a camera mounted on the moving object, based on the image; a distance calculation unit that calculates a distance from the moving object to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; A computer program that functions as a danger level assigning unit that assigns a danger level to a person that indicates the magnitude of the impact that the moving object will have on the person if the moving object collides with the person, based on the person's height.

9. A computer program for causing a computer to function as a driving assistance device for assisting the driving of an autonomously driven mobile body, comprising: The computer a detection unit that detects a person present in an image captured by a camera mounted on the moving object, based on the image; a distance calculation unit that calculates a distance from the moving object to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; a risk level assigning unit that assigns a risk level to the person based on the height; The computer a speed calculation unit that calculates the speed of the person based on the result of the person detection; further functioning as a predicted time calculation unit that calculates a predicted time until the moving object collides with the person based on the distance to the person and the speed of the person; The computer program further includes a computer program for causing the person to be at risk, the computer program further comprising: a computer program for causing the person to be at risk; a computer program for causing the person to be at risk;

10. A driving assistance system for assisting a vehicle capable of autonomous driving, A driving assistance device; a display control device connected to the driving assistance device via a network, The driving assistance device a detection unit that detects a person present in an image captured by a camera mounted on the moving object, based on the image; a distance calculation unit that calculates a distance from the moving object to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; a risk level assigning unit that assigns a risk level to the person based on the height, the risk level indicating the magnitude of the impact that the moving object will have on the person when the moving object collides with the person; a risk level information transmission unit that transmits risk level information based on the risk level assigned to the person to the display control device, the display control device performs a screen display based on the risk level information received from the driving assistance device. Driving assistance system.

11. A driving assistance system for assisting driving of an autonomously driving mobile body, comprising: A driving assistance device; a display control device connected to the driving assistance device via a network, The driving assistance device a detection unit that detects a person present in an image captured by a camera mounted on the moving object, based on the image; a distance calculation unit that calculates a distance from the moving object to the person based on the person detection result; a height calculation unit that calculates the height of the person based on the person detection result and the distance to the person; a risk level assigning unit that assigns a risk level to the person based on the height; a risk level information transmission unit that transmits risk level information based on the risk level assigned to the person to the display control device, the display control device performs a screen display based on the risk level information received from the driving support device, The driving assistance device a speed calculation unit that calculates the speed of the person based on the result of the person detection; a predicted time calculation unit that calculates a predicted time until the moving object collides with the person based on the distance to the person and the speed of the person, The risk level assigning unit further assigns the risk level to the person, indicating the length of time until the moving object will affect the person, based on the predicted time. Driving assistance system.

Images