Occupancy status notification method, occupancy status notification device, computer program, and driving assistance system

Abstract

To provide an occupation situation notification method capable of providing an operator who performs at least one of remote monitoring and remote operation of a moving body with information required for the remote monitoring or remote operation of the moving body.SOLUTION: An occupation situation notification method for notifying an operator who performs at least one of remote monitoring and remote operation of a moving body of the situation of the moving body that performs autonomous traveling on a scheduled travel path includes the steps of: acquiring detection data by a sensor mounted on the moving body; detecting an interference object that is an object possibly interfering with the traveling of the moving body from the detection data; calculating an occupancy rate of the travel path of the interference object; and notifying the operator of occupation situation information of the interference object on the basis of the occupancy rate.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to an occupancy status notification method, an occupancy status notification device, a computer program, and a driving assistance system. [Background technology]

[0002] Non-patent document 1 discloses a system that detects other moving objects, people, or fallen objects on the road around an autonomously moving robot from images captured by a camera mounted on the robot, and notifies the detection results to an operator who performs at least one of remotely monitoring and remotely operating the robot.

[0003] Non-Patent Document 2 discloses a system in which an image captured by a camera mounted on an autonomous vehicle is transmitted to a server in a control center, and the server detects objects from the image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Panasonic obtains road use permit for fully remote delivery robot”, [online], April 18, 2022, IoTNEWS, R-Gene Co., Ltd. IoTNEWS Division, [Retrieved November 13, 2023], Internet<URL: https: / / iotnews.jp / smart-city / 200568 / > [Non-patent document 2] “NEC Develops Learning-Based Media Transmission Control Technology to Enhance Remote Monitoring of Cars and Other Devices Using AI”, [online], January 12, 2021, DIGITAL SHIFT TIMES, Digital Holdings Co., Ltd., [Retrieved November 13, 2023], Internet<URL: https: / / digital-shift.jp / flash_news / FN210112_8> Summary of the Invention [Problem to be solved by the invention]

[0005] However, although the system in Non-Patent Document 1 can notify an operator of detection results, it is difficult for the operator to instantly determine which object in the video the notified detection result refers to. This makes it difficult for one operator to monitor multiple moving objects in parallel.

[0006] In the system of Non-Patent Document 2, when the communication bandwidth of the transmission path between the autonomous vehicle and the server becomes low, the video must be transmitted from the autonomous vehicle to the server after processing such as reducing the image quality. As a result, the server must perform object detection from low-quality video, which may result in missed or erroneous detection of objects.

[0007] The present disclosure has been made in consideration of the above circumstances, and aims to provide an occupancy status notification method, an occupancy status notification device, a computer program, and a driving assistance system that can provide an operator who performs at least one of the tasks of remotely monitoring and remotely operating a mobile body with the information necessary for remotely monitoring or remotely operating the mobile body. [Means for solving the problem]

[0008] An occupancy status notification method according to one aspect of the present disclosure is a method for notifying an operator who performs at least one of remote monitoring and remote operation of a mobile body traveling autonomously along a planned travel path of the status of the mobile body, and includes the steps of acquiring detection data by a sensor mounted on the mobile body, detecting an obstructing object from the detection data, which is an object that may obstruct the travel of the mobile body, calculating the occupancy rate of the obstructing object on the travel path, and notifying the operator of occupancy status information of the obstructing object based on the occupancy rate.

[0009] The present invention can be realized not only as an occupancy status notification method having such a characteristic processing unit, but also as an occupancy status notification device having processing units corresponding to such characteristic steps, or as a computer program for causing a computer to function as the occupancy status notification device, or as a semiconductor integrated circuit that realizes part or all of the occupancy status notification device, or as a driving assistance system including the occupancy status notification device. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an operator who performs at least one of remote monitoring and remote operation of a mobile object with information necessary for remote monitoring or remote operation of the mobile object. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a driving assistance system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of a driving assistance device according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram showing an example of an obstructing person region before integration. [Figure 4] FIG. 4 is a diagram showing an example of the obstructing person region after integration. [Figure 5] FIG. 5 is a diagram showing an example of the positional relationship between an obstructing person and the lens and image sensor of a camera installed on a moving object. [Figure 6] FIG. 6 is a diagram showing a display example of the display screen. [Figure 7] FIG. 7 is a diagram showing a display example of the display screen. [Figure 8] FIG. 8 is a flowchart showing an example of the operation of the driving support device. [Figure 9] FIG. 9 is a flowchart showing an example of the operation of the driving support device. [Figure 10] FIG. 10 is a diagram showing a display example of the display screen. DETAILED DESCRIPTION OF THE INVENTION

[0012] [Summary of the embodiments of the present disclosure] First, an outline of the embodiments of the present disclosure will be listed and described. (1) An occupancy status notification method according to one embodiment of the present disclosure is a method for notifying an operator who performs at least one of remote monitoring and remote operation of a mobile body, of the status of the mobile body that is autonomously traveling on a planned traveling path, and includes the steps of acquiring detection data by a sensor mounted on the mobile body, detecting an obstructing object, which is an object that may obstruct the traveling of the mobile body, from the detection data, calculating the occupancy rate of the obstructing object on the traveling path, and notifying the operator of occupancy status information of the obstructing object based on the occupancy rate.

[0013] According to this configuration, it is possible to notify an operator of occupancy status information based on the occupancy rate of obstructing objects that may obstruct the travel of a moving object. This makes it possible to provide the operator with information necessary for remote monitoring or remote operation of the moving object. For example, when the occupancy rate is high, the operator can slow down the traveling speed of the moving object or output a sound from the moving object to alert the obstructing object to the presence of the moving object.

[0014] (2) In the above (1), in the step of detecting the obstructing object, the obstructing object may be detected based on at least one of the distance from the moving body to the object, the speed of the object, and the predicted collision time between the moving body and the object.

[0015] According to this configuration, objects that are close to the moving body, objects that move at a high speed, or objects that have a short predicted collision time when it is assumed that the moving body and the object will collide can be considered as obstructing objects, and occupancy status information based on the occupancy rate of the obstructing objects can be notified to the operator.

[0016] (3) In the above (1) or (2), the step of notifying the operator may notify the occupancy status information in a notification mode according to the strength of the occupancy rate.

[0017] With this configuration, for example, when the intensity of the occupancy rate is high, such as the value of the occupancy rate or the duration of the occupancy rate above a certain threshold, the occupancy status information can be notified in a manner that makes it easy for the operator to notice. As a result, when the intensity of the occupancy rate is high, the operator can slow down the traveling speed of the mobile object or output a sound from the mobile object to alert the target object to the presence of the mobile object.

[0018] (4) In any one of (1) to (3) above, the occupancy status information may include information on the occupancy rate over time.

[0019] With this configuration, for example, the operator can know the duration of a high occupancy rate. If the high occupancy rate continues for a long time, the operator can sound an alarm from the mobile unit to alert others around it. This can lead to a decrease in the occupancy rate and create an environment where mobile units can travel without being obstructed by objects.

[0020] (5) In any one of (1) to (4) above, the occupancy status information may include information prompting an operation to stop the moving object.

[0021] With this configuration, the operator can receive notification of information that prompts the operator to stop the moving body when the occupancy rate is high, for example, and by stopping the moving body, the operator can avoid the moving body colliding with an obstructing object.

[0022] (6) An occupancy status notification device according to another embodiment of the present disclosure is an occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile body, of the status of the mobile body traveling autonomously along a planned travel path, and includes an acquisition unit that acquires detection data from a sensor mounted on the mobile body, a detection unit that detects, from the detection data, an obstruction object, which is an object that may obstruct the travel of the mobile body, a calculation unit that calculates the occupancy rate of the obstruction object on the travel path, and a notification unit that notifies the operator of occupancy status information of the obstruction object based on the occupancy rate.

[0023] This configuration includes processing units corresponding to the characteristic steps in the above-described occupancy status notification method, and therefore provides the same functions and effects as the above-described occupancy status notification method.

[0024] (7) A computer program according to another embodiment of the present disclosure is a computer program for causing a computer to function as an occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile body, of the status of the mobile body traveling autonomously along a planned travel path, and causes the computer to function as an acquisition unit that acquires detection data from a sensor mounted on the mobile body, a detection unit that detects, from the detection data, obstruction objects that are objects that may obstruct the travel of the mobile body, a calculation unit that calculates the occupancy rate of the obstruction objects on the travel path, and a notification unit that notifies the operator of occupancy status information of the obstruction objects based on the occupancy rate.

[0025] According to this configuration, the computer can function as the above-described occupancy status notification device, thereby achieving the same effects and advantages as the above-described occupancy status notification device.

[0026] (8) A driving assistance system according to another embodiment of the present disclosure is a driving assistance system that assists a moving body in driving, and includes an occupancy status notification device that notifies the status of the moving body that is autonomously driving on a planned driving path, and a display device connected to the occupancy status notification device via a network, wherein the occupancy status notification device includes an acquisition unit that acquires detection data by a sensor mounted on the moving body, a detection unit that detects, from the detection data, an obstruction object that is an object that may obstruct the driving of the moving body, a calculation unit that calculates the occupancy rate of the obstruction object on the driving path, and a transmission unit that transmits occupancy status information of the obstruction object to the display device based on the occupancy rate, and the display device displays a screen based on the occupancy status information received from the occupancy status notification device.

[0027] According to this configuration, occupancy status information based on the occupancy rate of obstructing objects that may obstruct the travel of the mobile unit can be transmitted to the display device and displayed on the screen. The operator can be provided with information necessary for remote monitoring or remote operation of the mobile unit. For example, when the occupancy rate is high, the operator can slow down the travel speed of the mobile unit or output a sound from the mobile unit to alert the obstructing objects to the presence of the mobile unit.

[0028] [Details of the embodiments of the present disclosure] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are examples and do not limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims are components that can be added arbitrarily. Furthermore, each figure is a schematic diagram and is not necessarily a precise illustration.

[0029] The same components are denoted by the same reference numerals, and their functions and names are also the same, so their explanations will be omitted where appropriate.

[0030] [Overall configuration of driving assistance system] FIG. 1 is a block diagram showing a hardware configuration of a driving assistance system according to an embodiment of the present disclosure. The driving assistance system 100 includes driving assistance devices 1A, 1B, and 1C and a driving control device 2.

[0031] The driving support devices 1A, 1B, 1C and the driving control device 2 are connected to each other via a network 3.

[0032] The driving assistance devices 1A, 1B, and 1C are devices that assist the driving of a moving body. The moving body includes, for example, a mobile robot that can autonomously drive on a driving path such as a scheduled road, or an automobile that can autonomously drive.

[0033] The driving assistance devices 1A, 1B, and 1C are mounted on different moving bodies and calculate the occupancy rate of people around the moving bodies relative to the moving body's driving path. The driving assistance devices 1A, 1B, and 1C each transmit occupancy status information indicating the calculated occupancy rate to the driving control device 2 via the network 3. Here, the occupancy rate of people relative to the driving path indicates, for example, the ratio of the width of the driving path that people occupy to the width of the driving path along which the moving body is scheduled to travel. For the same driving path width, the higher the occupancy rate, the more difficult it is for the moving body to avoid a collision with a person. In other words, the higher the occupancy rate, the more difficult it is for the moving body to drive autonomously, and the more likely it is that some kind of action, such as an operator operating the moving body, is required.

[0034] The driving control device 2 is installed in a monitoring center or the like that monitors the operation of the mobile body, and is operated by an operator that monitors the operation of the mobile body. The driving control device 2 receives occupancy status information from the driving support devices 1A, 1B, and 1C via the network 3. The driving control device 2 displays an image based on the received occupancy status information on a display screen such as a display (not shown) connected to itself.

[0035] 1 shows three driving assistance devices 1A, 1B, and 1C, but the number of driving assistance devices 1 is not limited to three. Hereinafter, when there is no need to distinguish between the driving assistance devices 1A, 1B, and 1C, they will be referred to as driving assistance devices 1. The driving control device 2 displays an image based on occupancy status information received from the multiple driving assistance devices 1 on the display screen of one or more displays.

[0036] An operator monitoring the operation of the mobile object looks at the display screen and intervenes in the operation of the mobile object equipped with the driving assistance device 1 that transmitted the occupancy status information indicating a high occupancy rate. For example, the operator transmits a control signal to the mobile object via the driving control device 2 to control the braking or steering of the mobile object. Upon receiving the control signal, the mobile object performs control based on the control signal, prioritizing autonomous driving.

[0037] [Configuration of driving assistance device 1] FIG. 2 is a block diagram showing a functional configuration of the driving assistance device 1 according to the embodiment of the present disclosure.

[0038] The driving assistance device 1 is connected by wire or wirelessly to a camera 4 mounted on the moving body. The camera 4 is an example of a sensor mounted on the moving body and outputs detected data. The camera 4, for example, captures an area ahead of the moving body in the traveling direction, and outputs video data of that area (hereinafter simply referred to as "video"). However, the area captured by the moving body is not limited to the area ahead of the moving body, but may also be an area to the side or behind the moving body. The video is composed of multiple image data (hereinafter simply referred to as "images") in a time series.

[0039] The driving assistance device 1 includes an image acquisition unit 11, a person area detection unit 12, a distance calculation unit 13, a person area tracking unit 14, a speed calculation unit 15, an obstructing person area detection unit 16, an obstructing person area integration unit 17, a road width acquisition unit 18, an occupancy status determination unit 19, and an occupancy status information transmission unit 20.

[0040] The image acquisition unit 11 acquires an image of the area around the moving object from the camera 4. Here, it is assumed that the image acquisition unit 11 acquires an image of the area in front of the moving object.

[0041] Based on the video acquired by the video acquisition unit 11, the person area detection unit 12 detects people present in each image included in the video. That is, the person area detection unit 12 detects the area of ​​the person included in the image. For example, the person area detection unit 12 inputs the image into a learning model and acquires information on the position and size of the circumscribing rectangle of the person from the learning model. The position of the circumscribing rectangle is, for example, the coordinates of the upper left corner of the circumscribing rectangle, and the size of the circumscribing rectangle is, for example, the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the circumscribing rectangle. The learning model is, for example, a convolution neural network (CNN) or a transformer. Using images containing various types of people as training data, learning of the position and size of people progresses using a machine learning method such as deep learning, and the parameters of the learning model are determined.

[0042] Distance calculation unit 13 calculates the distance from the moving object to the person based on the detection result of the person detected by person area detection unit 12. That is, information on the position and size of the circumscribing rectangle of the person in the image is obtained from person area detection unit 12. Distance calculation unit 13 calculates the distance d from the moving object (camera 4) to the person based on the following equation 1. d=H / tan(V_FOV / 2) / (2y / NpxV-1) …(Formula 1)

[0043] Here, H is the height from the ground to the mounting position of the camera 4. V_FOV is the vertical angle of view of the camera 4. y is the number of pixels from the bottom of the circumscribing rectangle of the person to the top of the screen. NpxV is the total number of pixels in the vertical direction of the image.

[0044] The person region tracking unit 14 tracks the person region detected by the person region detection unit 12. For example, the person region tracking unit 14 assigns the same label to circumscribing rectangles that are close to each other between two temporally consecutive images. Specifically, the person region tracking unit 14 calculates the distance between the centers of the circumscribing rectangles between the two images, and if the calculated distance is equal to or less than a predetermined first distance threshold, determines that the two circumscribing rectangles are close to each other and assigns the same label to the two circumscribing rectangles. Note that, when a label has already been assigned to the first circumscribing rectangle between a first circumscribing rectangle and a second circumscribing rectangle to which the same label is to be assigned, the person region tracking unit 14 assigns the same label to the second circumscribing rectangle as the label assigned to the first circumscribing rectangle. Furthermore, when labels have not been assigned to both the first circumscribing rectangle and the second circumscribing rectangle to which the same label is to be assigned, the person region tracking unit 14 assigns the same new label to the two circumscribing rectangles. The person region tracking unit 14 sequentially labels such circumscribing rectangles between two temporally consecutive images that make up the video. As a result, people included in circumscribing rectangles with the same label are determined to be the same person and tracked. The person region tracking unit 14 may also use a tracking model such as BoT-SORT.

[0045] The speed calculation unit 15 calculates the speed of each person based on the tracking result of the person area by the person area tracking unit 14 and the distance from the moving object to the person calculated by the distance calculation unit 13. In other words, the speed calculation unit 15 calculates the amount of change in distance to the same person (a person included in a circumscribing rectangle with the same label) between the image of the frame to be processed and the image of the frame preceding the specified number of frames. The speed calculation unit 15 calculates the relative speed v of the person with respect to the moving object by dividing the calculated amount of change in distance by the time corresponding to the specified number of frames.

[0046] The obstructing person area detection unit 16 detects an obstructing object that may obstruct the travel of the moving body. Specifically, the obstructing person area detection unit 16 detects the area of ​​the obstructing person that may obstruct the travel of the moving body, based on the distance d from the moving body to the person calculated by the distance calculation unit 13 and the relative speed v of the person calculated by the speed calculation unit 15.

[0047] For example, the obstructing person area detection unit 16 determines that a person whose distance d is less than a predetermined second distance threshold or whose relative speed v is equal to or greater than a predetermined first speed threshold is an obstructing person, and detects the person's area as an obstructing person area.

[0048] Furthermore, the obstructing person area detection unit 16 may detect an obstructing person area based on a predicted time until a moving object collides with a person (hereinafter referred to as "predicted collision time"), assuming that the moving object will collide with the person. For example, the obstructing person area detection unit 16 calculates the predicted collision time for each person by dividing the distance d by the relative speed v. The obstructing person area detection unit 16 determines that a person whose predicted collision time is equal to or shorter than a predetermined first time threshold is an obstructing person, and detects the person area of ​​that person as an obstructing person area.

[0049] The obstructing person area integrating unit 17 integrates the obstructing person areas of obstructing persons located close to each other.

[0050] FIG. 3 is a diagram showing an example of an obstructing person region before integration. The image includes obstructing persons 51 and 52. The obstructing person region detection unit 16 detects a circumscribing rectangle including the obstructing person 51 as an obstructing person region 61, and detects a circumscribing rectangle including the obstructing person 52 as an obstructing person region 62. The obstructing person region integration unit 17 determines whether the obstructing person 51 and the obstructing person 52 are located in close proximity. As an example, the horizontal distance in real space (the x-axis direction when projected onto the image) between the centers of the obstructing person regions 61 and 62 is defined as dx. The obstructing person region integration unit 17 estimates the direction in which the obstructing person 51 exists from the center position of the obstructing person region 61 in the image. The obstructing person region integration unit 17 also obtains the distance d to the obstructing person 51 from the distance calculation unit 13, and estimates the position of the obstructing person 51 in real space based on the estimated direction in which the obstructing person 51 exists and the distance d. The obstructing person region integrating unit 17 estimates the position of the obstructing person 52 in real space in the same way as the obstructing person 51. The obstructing person region integrating unit 17 calculates the distance dx from the estimated positions of the obstructing persons 51 and 52.

[0051] The distance in the depth direction between the obstructing person 51 and the obstructing person 52 is defined as dy. The obstructing person region integrating unit 17 calculates the absolute value of the difference between the distance d to the obstructing person 51 and the distance d to the obstructing person 52 calculated by the distance calculation unit 13 as the distance dy.

[0052] If the distance dx is less than or equal to a predetermined third distance threshold and the distance dy is less than or equal to a predetermined fourth distance threshold, the obstructing person area integration unit 17 determines that the obstructing persons 51 and 52 are a group and integrates the obstructing person area 61 and the obstructing person area 62.

[0053] Fig. 4 is a diagram showing an example of an obstructing person area after integration. The obstructing person area integration unit 17 integrates the obstructing person area 61 and the obstructing person area 62 shown in Fig. 3 to generate an obstructing person area 71. The obstructing person area 71 is a circumscribed rectangle of the obstructing person 51 and the obstructing person 52 included in the obstructing person area 61 and the obstructing person area 62, respectively.

[0054] 2 again, the road width acquisition unit 18 acquires the width of the road along which the moving object is scheduled to autonomously travel, at a position where an obstructing person will pass. Here, it is assumed that map data including the road along which the moving object is scheduled to autonomously travel is stored in a memory (not shown) of the driving assistance device 1. It is also assumed that the map data includes information indicating the width of the road.

[0055] The travel lane width acquisition unit 18 acquires the distance d from the moving object to the obstructing person calculated by the distance calculation unit 13 for each of the obstructing person areas detected by the obstructing person area detection unit 16 and the obstructing person areas integrated by the obstructing person area integration unit 17. The travel lane width acquisition unit 18 acquires the travel lane width W at the position where the obstructing person is located from the map data based on the current position, traveling direction, and distance d of the moving object.

[0056] The occupancy status determining unit 19 calculates the occupancy rate of the obstructing person on the travel path as the occupancy status of the obstructing person on the travel path.

[0057] Specifically, the occupancy status determination unit 19 calculates the width D of the obstructing person area in real space for each of the obstructing person area detected by the obstructing person area detection unit 16 and the obstructing person area integrated by the obstructing person area integration unit 17.

[0058] Figure 5 is a diagram showing an example of the positional relationship between an obstructing person and the lens and image sensor of a camera installed on a moving object. The distance (focal length) between the lens and the image sensor is defined as f. The distance to the obstructing person calculated by distance calculation unit 13 is defined as distance d. The width of the obstructing person region in the x-axis direction in the image is defined as n. Width n is the number of pixels in the x-axis direction that the obstructing person region occupies in the image.

[0059] The occupancy status determining unit 19 calculates the width D of the obstructing person area in real space based on the following equation 2. D=d×n×pixel_size / f…(Formula 2) Here, pixel_size is the size of one pixel in the image sensor. The occupancy state determining unit 19 also acquires the width W of the road at the position where the obstructing person is present.

[0060] The occupancy status determination unit 19 calculates the occupancy rate OC of the obstructing person by dividing the calculated width D of the obstructing person in real space by the width W of the road at the position where the obstructing person is present, which is acquired by the road width acquisition unit 18. The larger the occupancy rate OC, the higher the degree to which the obstructing person occupies the road.

[0061] The occupancy status determining unit 19 calculates the occupancy rate OC of the obstructing person regions after they have been integrated by the obstructing person region integrating unit 17 in the following manner.

[0062] Referring to FIG. 4, the width of the obstructing person region 71 in real space is defined as D, and the width of the road at the position where the obstructing persons 51 and 52 included in the obstructing person region 71 exist is defined as W.

[0063] The occupancy status determination unit 19 calculates the distance d in the above formula 2 based on the distance d1 to the obstructing individual 51 and the distance d2 to the obstructing individual 52. For example, the occupancy status determination unit 19 may calculate the average value of the distances d1 and d2 as the distance d, or may calculate the minimum value of the distances d1 and d2 as the distance d. The occupancy status determination unit 19 calculates the width D of the obstructing individual region 71 in real space by substituting the calculated distance d into formula 2.

[0064] The occupancy status determination unit 19 calculates the travel path width W based on the travel path width W1 at the position where the obstructing person 51 is present and the travel path width W2 at the position where the obstructing person 52 is present. For example, the occupancy status determination unit 19 may calculate the travel path width W as the average value of the travel path width W1 and the travel path width W2, or may calculate the travel path width W as the minimum value of the travel path width W1 and the travel path width W2.

[0065] The occupancy status determining unit 19 calculates the occupancy rate OC of the obstructing persons 51, 52 included in the obstructing person area 71 by dividing the calculated width D by the travel lane width W.

[0066] The occupancy status information transmitting unit 20 notifies the operator of the occupancy status information by transmitting the occupancy status information, including the occupancy status of the obstructing person on the driving path determined by the occupancy status determining unit 19, to the driving control device 2 via the network 3.

[0067] For example, the occupancy status information transmitting unit 20 compares the occupancy rate OC calculated by the occupancy status determining unit 19 with a predetermined occupancy rate threshold Tho, and if the occupancy rate OC is equal to or greater than the occupancy rate threshold Tho, transmits occupancy status information including the occupancy rate OC to the driving control device 2. Note that the occupancy status information transmitting unit 20 may include information about the obstructing person area in the occupancy status information to be transmitted.

[0068] Furthermore, the occupancy status information transmission unit 20 calculates the duration of the state in which the occupancy rate OC is equal to or greater than the occupancy rate threshold Tho (hereinafter referred to as "high occupancy state duration") DT, and compares the high occupancy state duration DT with a predetermined duration threshold Thp. If the high occupancy state duration DT is equal to or greater than the duration threshold Thp, the occupancy status information transmission unit 20 further includes the high occupancy state duration DT in the occupancy status information and transmits it to the driving control device 2. The high occupancy state duration DT is an example of information indicating the passage of occupancy rate over time.

[0069] The occupancy status information transmitting unit 20 transmits the occupancy status information together with the compressed video obtained by compressing the video acquired by the video acquiring unit 11. The occupancy status information transmitting unit 20 may transmit the occupancy status information together with the compressed video, or may transmit the occupancy status information as data separate from the compressed video.

[0070] The driving control device 2 receives the compressed video and occupancy status information from the driving assistance device 1. The driving control device 2 displays the occupancy status information on the display screen of the display together with the video obtained by decompressing the compressed video.

[0071] 6 and 7 are diagrams showing examples of the display screen. 6, the driving control device 2 displays images of the area ahead of the moving object captured by the camera 4 on the display screen. Each image includes obstructing persons 51 and 52. The driving control device 2 superimposes an obstructing person area 71 including the obstructing persons 51 and 52. The driving control device 2 also displays the occupancy rate and the duration of the high occupancy state in a display area 81 in the upper left corner of the image.

[0072] The driving control device 2 may display the occupancy status information in a notification mode that corresponds to the intensity of the occupancy status. In other words, the driving control device 2 may display the occupancy status information in a notification mode that corresponds to the intensity of the occupancy rate or the intensity of the duration of the high occupancy state.

[0073] 7, for example, when the occupancy rate is equal to or greater than a predetermined occupancy rate threshold Tho2, or when the duration of the high occupancy state is equal to or greater than a predetermined duration threshold Thp2, the driving control device 2 may highlight the occupancy rate and the duration of the high occupancy state by hatching the display area 82, provided that Tho2>Tho and Thp2>Thp.

[0074] [Operation procedure of driving assistance device 1] 8 and 9 are flowcharts showing an example of the operation of the driving assistance device 1. FIG. The driving assistance device 1 acquires an image of the area ahead of the moving object from the camera 4 (step S1).

[0075] The driving support device 1 detects a human region present in each image included in the acquired video (step S2).

[0076] The driving assistance device 1 calculates the distance d from the moving object to the person based on the detected person area (step S3). The driving assistance device 1 tracks the detected person area (step S4).

[0077] The driving assistance device 1 calculates the relative speed v of the person based on the tracking result of the person area and the distance d (step S5). The driving assistance device 1 detects the obstructing person area based on the distance d and the relative speed v (step S6). The driving assistance device 1 integrates the obstructing person regions of the obstructing people who are located close to each other (step S7).

[0078] The travel assistance device 1 acquires the width W of the travel path along which the moving body is to travel autonomously, at the position where the obstructing person will pass (step S8).

[0079] The driving assistance device 1 calculates the width D of the obstructing person region in real space based on the distance d and the width of the obstructing person region in the image (step S9).

[0080] The driving assistance device 1 calculates the occupancy rate OC of the obstructing person based on the width W of the road and the width D of the obstructing person area (step S10). The driving assistance device 1 compares the occupancy rate OC with the occupancy rate threshold Tho (step S11).

[0081] If the occupancy rate OC is equal to or greater than the occupancy rate threshold value Tho (YES in step S11), the driving assistance device 1 measures the high occupancy state duration DT (step S12). The driving assistance device 1 compares the high occupancy state duration DT with the duration threshold Thp (step S13).

[0082] If the high occupancy state duration DT is equal to or greater than the duration threshold Thp (YES in step S13), the driving assistance device 1 notifies the operator of the occupancy rate OC and the high occupancy state duration DT by transmitting occupancy status information including the occupancy rate OC and the high occupancy state duration DT to the driving control device 2 (step S14).

[0083] If the high occupancy state duration DT is less than the duration threshold Thp (NO in step S13), the driving assistance device 1 notifies the operator of the occupancy rate OC by transmitting occupancy status information including the occupancy rate OC to the driving control device 2 (step S15).

[0084] The driving support device 1 performs, for example, the processes from step S1 to step S15 for one image, and repeatedly executes these processes every time the image is updated.

[0085] As described above, according to this embodiment, it is possible to notify an operator of occupancy status information based on the occupancy rate of obstructing persons who may obstruct the travel of a moving object. This makes it possible to provide the operator with information necessary for remote monitoring or remote operation of the moving object. For example, when the occupancy rate is high, the operator can slow down the traveling speed of the moving object or output a sound from the moving object to alert the obstructing object to the presence of the moving object.

[0086] The driving assistance device 1 detects an obstructing person based on at least one of the distance from the moving object to the person, the speed of the person, and the predicted collision time between the moving object and the person. Therefore, a person who is close to the moving object, a person who is moving at a high speed, or a person who has a short predicted collision time when the moving object and the person collide can be determined to be an obstructing person, and occupancy status information based on the occupancy rate of the obstructing person can be notified to the operator.

[0087] Furthermore, the driving control device 2 notifies the operator of the occupancy status information by displaying the occupancy status information on the screen in a notification mode according to the strength of the occupancy rate. This allows the operator to slow down the traveling speed of the moving object or output a sound from the moving object when the strength of the occupancy rate is high, to alert the target object to the presence of the moving object.

[0088] <Variation 1> In the above embodiment, the driving assistance device 1 transmits occupancy status information to the driving control device 2, and the driving control device 2 displays an image based on the occupancy status information. The occupancy status information may include information prompting an operation to stop the moving object.

[0089] For example, when the occupancy rate OC calculated by the occupancy status determination unit 19 is equal to or greater than a predetermined stop judgment threshold Ths, the occupancy status information transmission unit 20 of the driving assistance device 1 transmits message information to the driving control device 2 as occupancy status information, encouraging the driver to perform an operation to stop the moving body.

[0090] This allows the driving control device 2 to display the occupancy status information including the message information on the display screen of the display, so the operator can immediately stop the moving object to avoid the moving object colliding with the obstructing object.

[0091] <Variation 2> 6 and 7 show examples of the display screen of the driving control device 2. The display examples are not limited to these.

[0092] FIG. 10 is a diagram showing a display example of the display screen. The driving control device 2 displays an image similar to that shown in Fig. 6 on the display screen. However, the driving control device 2 displays a graph of the occupancy rate along with the occupancy rate in the display area 83 at the top left of the image. The horizontal axis of the graph indicates time (for example, the time when the occupancy rate was calculated), and the vertical axis indicates the occupancy rate. The graph of the occupancy rate is an example of information showing the occupancy rate over time.

[0093] According to this modification, for example, the operator can know the duration of a high occupancy rate. If the high occupancy rate continues for a long time, the operator can sound an alarm from the moving object to alert people around the moving object. This can lead to a decrease in the occupancy rate and create an environment where moving objects can travel without being obstructed by obstructive persons.

[0094] <Variation 3> In the above embodiment, the distance calculation unit 13 calculates the distance d to the person using the camera parameters, but the method for calculating the distance d is not limited to this.

[0095] For example, the distance calculation unit 13 can calculate the distance d to the person using a software library capable of performing distance estimation from the video captured by the camera 4. Such a software library uses, for example, a learning model determined by machine learning of a large number of two-dimensional images. The learning model can accept a two-dimensional image as input and output a depth map in which each pixel indicates the distance to the object.

[0096] <Variation 4> Furthermore, the distance calculation unit 13 may use a sensor other than the camera 4 to calculate the distance to the person. For example, the distance calculation unit 13 may use millimeter wave radar to detect the distance from the moving body to the person, or may use LiDAR (Light Detection and Ranging) to detect the distance. The millimeter wave radar or LiDAR is installed at a position on the moving body where it can emit millimeter waves in the direction of travel of the moving body.

[0097] The distance calculation unit 13 acquires the position of the circumscribing rectangle of the person from the person area detection unit 12. The distance calculation unit 13 calculates the direction in which the person exists from the position of the circumscribing rectangle, and acquires the distance from the millimeter wave radar or LiDAR to the person located in that direction.

[0098] <Variation 5> In the above embodiment, the obstructing person area integrating unit 17 integrates multiple obstructing person areas based on the distance between the obstructing persons. Alternatively, the obstructing person area integrating unit 17 may integrate multiple obstructing person areas based on the distance d from the moving object to the obstructing person and the relative speed v of the obstructing person.

[0099] For example, the obstructing person area integrating unit 17 compares the distance dA and relative speed vA to obstructing person A with the distance dB and relative speed vB to obstructing person B. When a state in which |dA-dB| is equal to or less than a predetermined fifth distance threshold and |vA-vB| is equal to or less than a predetermined second speed threshold continues for a predetermined time or more, the obstructing person area integrating unit 17 integrates the obstructing person areas of obstructing person A and obstructing person B.

[0100] <Variation 6> The obstructing person region integrating unit 17 may integrate a plurality of obstructing person regions using an image processing technique.

[0101] For example, the obstructing person area integrating unit 17 estimates the skeleton of the obstructing person by processing the image. The obstructing person area integrating unit 17 determines whether the obstructing person is holding hands from the position of the skeleton, and integrates the obstructing person areas of multiple obstructing people who are determined to be holding hands.

[0102] <Variation 7> Furthermore, the obstructing person area integrating unit 17 may integrate the obstructing person areas of multiple obstructing persons who are having a conversation. For example, the obstructing person area integrating unit 17 recognizes the line of sight and mouth movements of each obstructing person by processing the image. If the obstructing person area integrating unit 17 determines from the recognition result that the obstructing persons are looking at each other and their mouths are moving alternately, it determines that the obstructing persons are having a conversation and integrates the obstructing person areas of the multiple obstructing persons who are having a conversation.

[0103] Alternatively, the disruptive person area integrating unit 17 may determine whether the disruptive persons are talking based on the distance between the rectangular areas and the movement of their mouths. For example, if the distance dx shown in Fig. 3 is equal to or less than a predetermined sixth distance threshold and the mouths of the disruptive persons are moving alternately, the disruptive person area integrating unit 17 determines that the disruptive persons are talking to each other and integrates the disruptive person areas of the multiple disruptive persons who are talking.

[0104] <Variation 8> In the above embodiment, a person such as a pedestrian is considered as an object that may obstruct the travel of the moving body, but the object is not limited to a person. For example, the object may be a fallen object or a placed object that is on the travel path of the moving body.

[0105] [Note] Each process (each function) in the above-described embodiments is realized by a processing circuit including one or more processors. The processing circuit may be configured as an integrated circuit or the like that combines one or more memories, various analog circuits, and various digital circuits in addition to the one or more processors. The one or more memories store programs (instructions) that cause the one or more processors to execute each of the processes. The one or more processors may execute each of the processes according to the programs read from the one or more memories, or according to logic circuits pre-designed to execute each of the processes. The processor may be various processors suitable for computer control, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). Note that the physically separated processors may cooperate with each other to execute each of the processes. For example, the processors mounted on a plurality of physically separated computers may cooperate with each other to execute the processes via a network such as a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, etc. The program may be installed into the memory from an external server device or the like via the network, or may be distributed in a state stored in a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a semiconductor memory, and installed into the memory from the recording medium. Furthermore, at least some of the above-described embodiments and modifications may be combined in any manner.

[0106] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0107] 1 Driving support device (occupancy status notification device) 1A Driving support device (occupancy status notification device) 1B Driving support device (occupancy status notification device) 1C Driving support device (occupancy status notification device) 2. Driving control device 3 Network 4. Camera 11 Video acquisition unit (acquisition unit) 12 Person area detection unit 13 Distance calculation unit (detection unit) 14 Person area tracking unit 15 Speed ​​calculation unit (detection unit) 16. Obstructing person area detection unit (detection unit) 17 Obstruction Person Area Integration Department 18 Road width acquisition unit 19 Occupancy status determination unit (calculation unit) 20 Occupancy status information transmission unit (notification unit, transmission unit) 51 Obstructive person 52 Obstructive person 61 Obstruction person area 62 Obstruction person area 71 Obstruction person area 81 Display area 82 Display area 83 Display area 100 Driving Assistance System

Claims

1. An occupancy status notification method for notifying an operator who performs at least one of remote monitoring and remote operation of a mobile object that is autonomously traveling along a scheduled travel route, the method comprising: the travel path indicates the entire road on which the moving object travels, acquiring detection data by a sensor mounted on the moving object; detecting an obstructing object that may obstruct the travel of the mobile body from the detection data; calculating an occupancy rate of the obstacle object relative to the travel path; calculating a high occupancy state duration, which is a duration of a state in which the occupancy rate is equal to or greater than an occupancy rate threshold; an occupancy status notification method including a step of notifying the operator of the duration of the high occupancy status as occupancy status information of the obstructing object when the duration of the high occupancy status is equal to or greater than a duration threshold.

2. An occupancy status notification method for notifying an operator who performs at least one of remote monitoring and remote operation of a mobile object that is autonomously traveling along a scheduled travel route of the status of the mobile object, comprising: the travel path indicates the entire road on which the moving object travels, acquiring detection data by a sensor mounted on the moving object; detecting a first obstruction object and a second obstruction object that may obstruct the travel of the moving body from the detection data; a step of determining whether to integrate the first obstruction object and the second obstruction object based on at least one of a distance between the first obstruction object and the second obstruction object, a relative speed between the first obstruction object and the second obstruction object, a determination result as to whether the first obstruction object and the second obstruction object, which are people, are holding hands, and a determination result as to whether the first obstruction object and the second obstruction object, which are people, are talking; calculating an occupancy rate of the area obtained by integrating the first obstruction object and the second obstruction object relative to the travel path when it is determined that the first obstruction object and the second obstruction object should be integrated; and notifying the operator of the integrated occupancy status information of the first obstruction object and the second obstruction object based on the occupancy rate.

3. An occupancy status notification method for notifying an operator who performs at least one of remote monitoring and remote operation of a mobile object that is autonomously traveling along a scheduled travel route, the method comprising: the travel path indicates the entire road on which the moving object travels, acquiring detection data by a sensor mounted on the moving object; detecting an obstructing object that may obstruct the travel of the mobile body from the detection data; calculating an occupancy rate of the obstacle object relative to the travel path; notifying the operator of occupancy status information of the obstruction object based on the occupancy rate; In the step of detecting an obstructing object, the obstructing object is detected based on a predicted collision time between the mobile body and an object.

4. 3. The occupancy status notifying method according to claim 1, wherein in the step of notifying an operator, the occupancy status information is notified in a notification format corresponding to the strength of the occupancy rate.

5. The occupancy status notification method according to claim 1 or 2, wherein the occupancy status information includes information prompting an operation to stop the moving object.

6. An occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a moving object that is autonomously traveling on a scheduled traveling route, the occupancy status notification device comprising: the travel path indicates the entire road on which the moving object travels, an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects an obstructing object that is an object that may obstruct the travel of the moving body from the detection data; a calculation unit that calculates an occupancy rate of the obstacle object relative to the travel path; An occupancy status notification device comprising: a notification unit that calculates a high occupancy state duration, which is the duration of a state in which the occupancy rate is equal to or greater than an occupancy rate threshold, and notifies the operator of the high occupancy state duration as occupancy status information of the obstructing object when the high occupancy state duration is equal to or greater than the duration threshold.

7. An occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile object that is autonomously traveling along a scheduled travel route, the device comprising: the travel path indicates the entire road on which the moving object travels, an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects, from the detection data, a first obstruction object and a second obstruction object that are objects that may obstruct the travel of the moving body; an integration unit that determines whether to integrate the first obstruction object and the second obstruction object based on at least one of a distance between the first obstruction object and the second obstruction object, a relative speed between the first obstruction object and the second obstruction object, a determination result as to whether the first obstruction object and the second obstruction object, which are people, are holding hands, and a determination result as to whether the first obstruction object and the second obstruction object, which are people, are talking; a calculation unit that calculates an occupancy rate of the area obtained by integrating the first obstruction object and the second obstruction object relative to the travel path when it is determined that the first obstruction object and the second obstruction object should be integrated; an occupancy status notification device comprising: a notification unit that notifies the operator of the integrated occupancy status information of the first obstruction object and the second obstruction object based on the occupancy rate.

8. An occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile object that is autonomously traveling along a scheduled travel route, the device comprising: the travel path indicates the entire road on which the moving object travels, an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects an obstructing object that is an object that may obstruct the travel of the moving body from the detection data; a calculation unit that calculates an occupancy rate of the obstacle object relative to the travel path; a notification unit that notifies the operator of occupancy status information of the obstruction object based on the occupancy rate, The detection unit detects the obstructing object based on a predicted collision time between the moving body and the object.

9. A computer program for causing a computer to function as an occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile object that is autonomously traveling on a scheduled traveling route, the computer program comprising: the travel path indicates the entire road on which the moving object travels, The computer an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects an obstructing object that is an object that may obstruct the travel of the moving body from the detection data; a calculation unit that calculates an occupancy rate of the obstacle object relative to the travel path; A computer program that calculates a high occupancy state duration, which is the duration of a state in which the occupancy rate is equal to or greater than an occupancy rate threshold, and functions as a notification unit that notifies the operator of the high occupancy state duration as occupancy status information of the obstructing object when the high occupancy state duration is equal to or greater than the duration threshold.

10. A computer program for causing a computer to function as an occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile object traveling autonomously along a scheduled travel route of the status of the mobile object, comprising: the travel path indicates the entire road on which the moving object travels, The computer an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects, from the detection data, a first obstruction object and a second obstruction object that are objects that may obstruct the travel of the moving body; an integration unit that determines whether to integrate the first obstruction object and the second obstruction object based on at least one of a distance between the first obstruction object and the second obstruction object, a relative speed between the first obstruction object and the second obstruction object, a determination result as to whether the first obstruction object and the second obstruction object, which are people, are holding hands, and a determination result as to whether the first obstruction object and the second obstruction object, which are people, are talking; a calculation unit that calculates an occupancy rate of the area obtained by integrating the first obstruction object and the second obstruction object relative to the travel path when it is determined that the first obstruction object and the second obstruction object should be integrated; A computer program for causing the computer to function as a notification unit that notifies the operator of the integrated occupancy status information of the first obstruction object and the second obstruction object based on the occupancy rate.

11. A computer program for causing a computer to function as an occupancy status notification device that notifies an operator who performs at least one of remote monitoring and remote operation of a mobile object traveling autonomously along a scheduled travel route of the status of the mobile object, comprising: the travel path indicates the entire road on which the moving object travels, The computer an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects an obstructing object that is an object that may obstruct the travel of the moving body from the detection data; a calculation unit that calculates an occupancy rate of the obstacle object relative to the travel path; functioning as a notification unit that notifies the operator of occupancy status information of the obstruction object based on the occupancy rate; The detection unit detects the interfering object based on a predicted collision time between the moving body and the object.

12. A driving assistance system that assists driving of a moving object, an occupancy status notification device that notifies the status of the moving body autonomously traveling along a scheduled traveling route; a display device connected to the occupancy status notification device via a network, the travel path indicates the entire road on which the moving object travels, The occupancy status notification device an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects an obstructing object that is an object that may obstruct the travel of the moving body from the detection data; a calculation unit that calculates an occupancy rate of the obstacle object relative to the travel path; a transmitter that calculates a high occupancy state duration, which is a duration of a state in which the occupancy rate is equal to or greater than an occupancy rate threshold, and transmits the high occupancy state duration to the display device as occupancy status information of the obstruction object when the high occupancy state duration is equal to or greater than the duration threshold; the display device performs a screen display based on the occupancy status information received from the occupancy status notification device. Driving assistance system.

13. A driving assistance system that assists driving of a moving object, an occupancy status notification device that notifies the status of the moving body autonomously traveling along a scheduled traveling route; a display device connected to the occupancy status notification device via a network, the travel path indicates the entire road on which the moving object travels, The occupancy status notification device an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects, from the detection data, a first obstruction object and a second obstruction object that are objects that may obstruct the travel of the moving body; an integration unit that determines whether to integrate the first obstruction object and the second obstruction object based on at least one of a distance between the first obstruction object and the second obstruction object, a relative speed between the first obstruction object and the second obstruction object, a determination result as to whether the first obstruction object and the second obstruction object, which are people, are holding hands, and a determination result as to whether the first obstruction object and the second obstruction object, which are people, are talking; a calculation unit that calculates an occupancy rate of the area obtained by integrating the first obstruction object and the second obstruction object relative to the travel path when it is determined that the first obstruction object and the second obstruction object should be integrated; a transmitter that transmits, to the display device, integrated occupancy status information of the first obstruction object and the second obstruction object based on the occupancy rate, the display device performs a screen display based on the occupancy status information received from the occupancy status notification device. Driving assistance system.

14. A driving assistance system that assists driving of a moving object, comprising: an occupancy status notification device that notifies the status of the moving body autonomously traveling along a scheduled traveling route; a display device connected to the occupancy status notification device via a network, the travel path indicates the entire road on which the moving object travels, The occupancy status notification device an acquisition unit that acquires detection data by a sensor mounted on the moving body; a detection unit that detects an obstructing object that is an object that may obstruct the travel of the moving body from the detection data; a calculation unit that calculates an occupancy rate of the obstacle object relative to the travel path; a transmitting unit that transmits occupancy status information of the obstruction object to the display device based on the occupancy rate, the detection unit detects the obstructing object based on a predicted collision time between the moving body and an object; the display device performs a screen display based on the occupancy status information received from the occupancy status notification device. Driving assistance system.

Images