Information processing device, server, sensor data transmission method and program

Abstract

[Problem] To improve performance of inspecting an object under inspection by using a mobile object. [Solution] This information processing device comprises: a first acquisition means that acquires a timing at which a mobile object, which is capable of switching between at least two driving modes including an autonomous driving mode for movement by autonomous driving and a non-autonomous driving mode, has switched from the autonomous driving mode to the non-autonomous driving mode; a second acquisition means that acquires sensor information from a sensor that determines the state of a road and is mounted on the mobile object; and a transmission means that transmits, to a predetermined host device, the sensor information and mode change information indicating the timing at which the mobile object has switched from the autonomous driving mode to the non-autonomous driving mode.

Description

Information processing device, server, inspection system, sensor data transmission method, and program recording medium

[0001] The present invention relates to an information processing device, a server, an inspection system, a sensor data transmission method, and a program recording medium.

[0002] Patent Document 1 discloses a degradation diagnosis system that uses a camera attached to a mobile object to diagnose the degree of deterioration of public facilities. According to the document, the degradation diagnosis system includes a degradation level analysis unit that analyzes the degree of deterioration of an inspection object captured in images taken by the camera. The degradation diagnosis system also includes a priority calculation unit that calculates a priority of the inspection object based on the degree of deterioration of the same inspection object captured in multiple images and information about the driving status of the mobile object. For example, if a sudden steering or braking operation is performed due to deterioration of the inspection object, the degradation diagnosis system increases the priority of the inspection object.

[0003] International Publication No. 2020 / 022042

[0004] The deterioration diagnosis system described above uses images taken by a camera attached to a mobile object as well as information on the vehicle's driving conditions to calculate the degree of deterioration and priority of the object being inspected. However, because this method relies on images and driving conditions, there is a possibility that the detection of minor deterioration of the object being inspected may be delayed.

[0005] The present invention aims to provide an information processing device, a server, an inspection system, a sensor data transmission method, and a program recording medium that can contribute to improving the performance of inspection of inspection objects using mobile objects.

[0006] According to a first aspect, there is provided an information processing device comprising: a first acquisition means for acquiring the timing at which a mobile body capable of switching between at least two or more driving modes, including an autonomous driving mode for moving by autonomous driving and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; a second acquisition means for acquiring sensor information from a sensor for determining the state of an object to be inspected that is mounted on the mobile body; and a transmission means for transmitting mode change information indicating the timing at which the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a specified higher-level device.

[0007] According to a second aspect, a server is provided that includes a receiving means for receiving the mode change information and the sensor information from the above-mentioned information processing device, and a determining means for determining the state of an object to be inspected using the mode change information and the sensor information.

[0008] According to a third aspect, there is provided an inspection system including the above-mentioned information processing device and the above-mentioned server.

[0009] According to a fourth aspect, there is provided a sensor data transmission method which acquires the timing at which a mobile body capable of switching between at least two or more driving modes, including an autonomous driving mode that moves autonomously and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode, acquires sensor information from a sensor for determining the state of an object to be inspected that is mounted on the mobile body, and transmits mode change information indicating the timing at which the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a specified higher-level device.

[0010] According to a fifth aspect, there is provided a program recording medium having recorded thereon a program that causes a computer to execute the following processes: a process of acquiring the timing at which a mobile body capable of switching between at least two or more driving modes, including an autonomous driving mode for moving by autonomous driving and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; a process of acquiring sensor information from a sensor for determining the state of an object to be inspected that is mounted on the mobile body; and a process of transmitting mode change information indicating the timing at which the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a specified higher-level device.

[0011] According to the present invention, an information processing device, a server, an inspection system, a sensor data transmission method, and a program recording medium are provided that can contribute to improving the performance of inspection of an object to be inspected using a mobile object.

[0012] 1 is a diagram illustrating a configuration of an embodiment of the present invention. FIG. 1 is a diagram for explaining operation of an embodiment of the present invention. FIG. 1 is a diagram for explaining an image provided by an information processing device of an embodiment of the present invention. FIG. 2 is a diagram for explaining an image provided by an information processing device shown as a comparative example. FIG. 2 is another diagram for explaining an image provided by an information processing device of an embodiment of the present invention. FIG. 1 is a diagram illustrating a configuration of a first embodiment of the present invention. FIG. 2 is a functional block diagram illustrating the configuration of a control center device of the first embodiment of the present invention. FIG. 3 is a flowchart illustrating operation of the control center device of the first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of an image (without mode change information) provided by the control center device to an analysis server. FIG. 4 is a diagram illustrating an example of an image provided by the control center device of the first embodiment of the present invention to a server. FIG. 4 is a diagram illustrating a configuration of a second embodiment of the present invention. FIG. 5 is a functional block diagram illustrating the configuration of the control center device of the second embodiment of the present invention. FIG. 5 is a flowchart illustrating operation of the control center device of the second embodiment of the present invention. FIG. 6 is a diagram illustrating an example of data provided by the control center device of the second embodiment of the present invention to a server. FIG. 7 is a diagram illustrating a configuration of a third embodiment of the present invention. FIG. 7 is a functional block diagram illustrating the configuration of the control center device of the third embodiment of the present invention. FIG. 8 is a flowchart illustrating operation of the control center device of the third embodiment of the present invention. FIG. 9 is a diagram illustrating an example of data provided by the control center device of the third embodiment of the present invention to a server. FIG. 1 is a diagram showing the configuration of a fourth embodiment of the present invention. FIG. 2 is a functional block diagram showing the configuration of a control center device of the fourth embodiment of the present invention. FIG. 3 is a flowchart showing the operation of the control center device of the fourth embodiment of the present invention. FIG. 4 is a diagram showing an example of data provided to a server by the control center device of the fourth embodiment of the present invention. FIG. 5 is a diagram showing the configuration of a fifth embodiment of the present invention. FIG. 6 is a functional block diagram showing the configuration of an in-vehicle terminal of the fifth embodiment of the present invention. FIG. 7 is a flowchart showing the operation of the in-vehicle terminal of the fifth embodiment of the present invention. FIG. 8 is a diagram showing the configuration of a computer that can function as an information processing device of the present invention.

[0013] First, an overview of one embodiment of the present invention will be described with reference to the drawings. Note that the reference numerals in this overview are used for convenience to identify each element as an example to facilitate understanding, and are not intended to limit the present invention to the illustrated form. Furthermore, connecting lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. Furthermore, this computer device is configured to be able to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless. Furthermore, although ports or interfaces exist at the input / output connection points of each block in the drawings, they are not shown.

[0014] In one embodiment, the present invention can be realized in a configuration including an information processing device 10 capable of communicating with a mobile body V1 that can switch between at least two driving modes, including an autonomous driving mode in which the mobile body moves autonomously and a non-autonomous driving mode, as shown in Figure 1, and a higher-level device 20 that receives sensor information from the information processing device 10.

[0015] More specifically, the information processing device 10 includes a first acquisition means 11, a second acquisition means 12, and a transmission means 13. The first acquisition means 11 acquires the timing at which the mobile object V1 switches from an autonomous driving mode to a non-autonomous driving mode. The second acquisition means 12 acquires sensor information from a sensor mounted on the mobile object for determining the state of an inspection target. The transmission means 13 transmits mode change information indicating the timing at which the mobile object V1 switches from the autonomous driving mode to the non-autonomous driving mode and the sensor information to a predetermined host device.

[0016] For example, as shown in Figure 2, an example will be described in which a pothole PH is found on the road surface ahead of a moving object V1, and this pothole PH is to be detected as an abnormality. The dotted line extending from a sensor mounted on the vehicle in Figure 2 indicates the sensing range of the sensor. Point P1 indicates the point at which the driver of the moving object V1 or a person performing remote control (operator) visually recognizes the pothole PH and switches from autonomous driving mode to non-autonomous driving mode in order to improve safety, etc. Note that the non-autonomous driving mode may be a direct driving mode (manual driving mode) in which the driver manually drives the vehicle, or a remote driving mode in which the remote control side remotely drives the vehicle, etc.

[0017] As shown in Fig. 3, when the information processing device 10 of this embodiment switches from the autonomous driving mode to the non-autonomous driving mode, it transmits mode change information and sensor information to the host device 20. This allows the host device 20 to perform a detailed analysis of the sensor information from just before the pothole PH, as shown in Fig. 3. As a result, it becomes possible to quickly discover cracks in the road surface just before the pothole PH.

[0018] Figure 4 is an illustration of analyzing sensor information using the method described in the Background Art. As shown in Figure 4, the degree of road surface deterioration can be determined from the sensor information, but information regarding the timing of switching from autonomous driving mode to non-autonomous driving mode cannot be obtained. There is a possibility that cracks in the road surface just before the pothole PH may be overlooked.

[0019] Note that the information processing device 10 of this embodiment may also be configured to promptly transmit mode change information and sensor information to the host device 20 when switching from non-autonomous driving mode to autonomous driving mode. For example, point P2 in FIG. 5 indicates the point where the non-autonomous driving mode was switched to autonomous driving mode. In this case, when the information processing device 10 switches from autonomous driving mode to non-autonomous driving mode, it promptly transmits mode change information indicating that the driving mode has been returned to autonomous driving mode and sensor information to the host device 20. In this manner, the host device 20 can complete detailed analysis of the sensor information. This makes it possible to reduce the cost of transmitting sensor information and the calculation cost of the host device 20.

[0020] As described above, this embodiment makes it possible to improve the performance of inspecting an object to be inspected using the mobile unit V1. While the above example illustrates an example in which the driver of the mobile unit V1 or the remote control operator switches from the autonomous driving mode to the non-autonomous driving mode, the entity responsible for switching from the autonomous driving mode to the non-autonomous driving mode is not limited to these individuals. For example, the mobile unit V1 may switch to the non-autonomous driving mode if it determines that autonomous driving is not possible based on information sensed by the mobile unit V1 itself. In this case, the mobile unit V1 notifies the driver or the remote control operator and then transmits mode change information and sensor information to the host device 20. Possible cases in which the mobile unit V1 itself determines that autonomous driving is not possible include when the mobile unit V1 detects an abnormality in the road or surrounding conditions, or when it detects an erroneous recognition of the road or surrounding conditions. Autonomous driving may also include autonomous driving in which the mobile unit V1 autonomously determines whether or not to drive based on information sensed by the mobile unit V1.

[0021] [First embodiment] Next, a first embodiment in which the functions of the information processing device of the present invention are arranged in a control center device 100 that controls a moving object will be described in detail with reference to the drawings. In the following description, an example in which the moving object is a bus equipped with an automatic driving function will be described.

[0022] Fig. 6 is a diagram showing the configuration of the first embodiment of the present invention, which shows the configuration of a road inspection system including a control center device 100, a bus 300, a base station 500 of a mobile communication network that realizes communication between the bus 300 and the control center device 100, and an analysis server 400.

[0023] The bus 300 is capable of switching between an automatic driving mode, in which the bus 300 is driven automatically based on information sensed by the bus itself, and a remote driving mode, in which the bus is operated under remote control from the control center device 100. The bus 300 is also equipped with a camera 310 as a sensor, capable of capturing images of the area ahead of the bus 300. Switching between the automatic driving mode and the remote driving mode may be performed by the bus 300 itself or by the control center device 100 itself. In the following description, it is assumed that an operator OP at the control center device 100 switches between the automatic driving mode and the remote driving mode by referring to the image captured by the camera 310. The bus 300 may also have, in addition to the automatic driving mode and the remote driving mode, a manual driving mode, for example. In this case, switching between the driving modes may also be performed by the bus 300 itself or by the control center device 100 itself.

[0024] 7 is a functional block diagram showing the configuration of the control center device 100 according to the first embodiment of the present invention. Referring to FIG. 7, the configuration of the control center device 100 includes an operation mode information acquisition unit 101, an image acquisition unit 102, a transmission unit 103, and a remote control unit 104.

[0025] The remote control means 104 uses a display device (not shown) or the like to provide the operator OP with information on the instruments mounted on the bus and images from the camera 310. The operator OP operates the bus 300 based on the information provided by the remote control means 104. Specifically, the remote control means 104 receives steering operations for the bus 300 and commands to be given to the bus 300 from the information operator OP, and remotely controls the bus 300.

[0026] The driving mode information acquisition means 101 acquires driving mode information of the bus 300 from the remote control means 104. When a change occurs in the driving mode of the bus 300, the driving mode information acquisition means 101 creates mode change information indicating the timing at which the driving mode was switched. Therefore, the driving mode information acquisition means 101 functions as a first acquisition means 11 that acquires the timing at which the driving mode was switched from the autonomous driving mode to the non-autonomous driving mode. This mode change information can include, for example, information on the driving mode after the change and time information indicating the timing of the change.

[0027] The image acquisition means 102 acquires images from a camera 310 mounted on the bus 300. Therefore, the image acquisition means 102 functions as a second acquisition means 12 that acquires sensor information from a sensor for determining the state of an object to be inspected that is mounted on the mobile body.

[0028] The transmission means 103 transmits the mode change information and the camera image to the analysis server 400, which serves as a higher-level device. Note that the mode change information and the camera image do not necessarily need to be transmitted simultaneously. For example, the camera image may be transmitted periodically, while the mode change information may be transmitted only when necessary. Even in this transmission mode, the analysis server 400 can refer to the mode change information and analyze the image without any problems.

[0029] The analysis server 400 includes a receiving means (reference numeral 401 in FIG. 6) for receiving mode change information and sensor information, and a determining means (reference numeral 402 in FIG. 6) for determining the road condition using the mode change information and the sensor information.

[0030] The base station 500 is a base station for a fifth generation mobile communication system (5G) or LTE (Long Term Evolution).

[0031] Next, the operation of this embodiment will be described in detail with reference to the drawings. Fig. 8 is a flowchart showing the operation of the control center device 100 according to the first embodiment of the present invention. Referring to Fig. 8, first, the control center device 100 checks whether a change in the operation mode has occurred (step S001).

[0032] If the check reveals that no change in the driving mode has occurred, the control center device 100 transmits the image acquired by the camera 310 to the analysis server 400 (step S004).

[0033] On the other hand, if a change in the driving mode has occurred, the control center device 100 creates mode change information indicating the timing at which the driving mode was switched (step S002).

[0034] Next, the control center device 100 transmits the mode change information and the image acquired by the camera 310 to the analysis server 400 (step S003).

[0035] 9 is a diagram showing an example of an image (without mode change information) provided by the control center device 100 to the analysis server 400. The example in FIG. 9 shows an image of a road captured by the camera 310 of the bus 300. In the example in FIG. 9, a crack CR has occurred near the center line in the middle of the road. Furthermore, since such a crack CR may affect automated driving, the operator OP switches from automated driving mode to remote driving mode as soon as he senses an abnormality in the distance.

[0036] FIG. 10 is a diagram showing an example of an image provided to the analysis server 400 by the control center device 100 using mode change information. In the example of FIG. 10, a message M1 reading "Remote control operation in progress" is added to the upper right corner of the image. Note that while "Remote control operation in progress" is displayed in the example of FIG. 10, during automated driving, "Automated driving in progress" is displayed. The analysis server 400 and personnel viewing its display device can detect road abnormalities early based on this change in the driving mode display. Furthermore, by having the analysis server 400 change the image analysis mode in response to such a change in driving mode, detection of the starting point of a crack CR, etc., is also facilitated.

[0037] Furthermore, when the section in which a road abnormality is detected ends, the operator OP switches from remote driving mode to automatic driving mode. In this case, the control center device 100 acquires second mode change information indicating the timing of the switch from non-automated driving mode to automatic driving mode and transmits it to the analysis server 400. As a result, the display "Remote Control Driving" changes to "Automatic Driving." This change in the display of the driving mode allows personnel viewing the analysis server 400 or its display device to detect the end of the abnormal road section. Furthermore, by changing the image analysis mode in response to this change in driving mode, the analysis server 400 can easily detect the end point of the crack CR. Note that while the examples in FIGS. 9 and 10 illustrate the detection of cracks as a road abnormality, the abnormalities detectable by the analysis server 400 are not limited to cracks. By implementing inspection functions according to the type of inspection object in the analysis server 400, it becomes possible to detect other road abnormalities and inspect structures other than roads.

[0038] Second Embodiment In the above embodiment, the control center device 100 transmits images to the analysis server 400, but it is also possible to receive other sensor data from the bus 300 and send it to the analysis server 400. A second embodiment in which the control center device 100a transmits sensor information to the analysis server 400 will be described below.

[0039] Fig. 11 is a diagram showing the configuration of a second embodiment of the present invention. Referring to Fig. 11, the configuration includes a control center device 100a, a bus 300a equipped with a sensor 320, a base station 500 of a mobile communication network that realizes communication between the bus 300a and the control center device 100a, and an analysis server 400. This embodiment differs from the first embodiment in that the bus 300a is equipped with the sensor 320 and the control center device 100a has a transmission function for the sensor 320. Since the other configurations are the same as those of the first embodiment, the following description will focus on the differences.

[0040] 12 is a functional block diagram showing the configuration of a control center device 100a according to a second embodiment of the present invention, which uses the sensor data transmission method of the present invention. The difference from the control center device 100 according to the first embodiment shown in FIG. 7 is that the control center device 100a further includes a sensor data acquisition means 105, and the transmission means 103 is configured to transmit sensor data together with images to the analysis server 400.

[0041] The sensor data acquiring unit 105 acquires sensor data from a sensor 320 mounted on the bus 300a. The sensor 320 may be a speedometer, steering angle information, GPS (Global Positioning System) information, LiDAR (Light Detection and Ranging), an acceleration sensor, millimeter wave radar, or the like.

[0042] The transmission means 103 associates the driving mode information, the camera image, and the above-mentioned sensor data, and transmits them to the analysis server 400 .

[0043] Next, the operation of this embodiment will be described in detail with reference to the drawings. Fig. 13 is a flowchart showing the operation of the control center device 100a according to the second embodiment of the present invention. The difference from the operation of the control center device 100 according to the first embodiment shown in Fig. 8 is that in steps S103 and S104, the control center device 100a transmits sensor data in addition to images to the analysis server 400.

[0044] Fig. 14 shows an example of an image that the control center device 100a provides to the analysis server 400 using various information obtained from the bus 300a. In the example of Fig. 14, in addition to the same image as in the first embodiment, a display using sensor data is shown on the right side of Fig. 14. Symbol S1 represents the direction of the bus 300a (heading information) based on, for example, the steering angle of the steering wheel or GPS information. Symbol S2 represents the value of the acceleration sensor in the Z direction (vertical direction) of the bus 300a.

[0045] According to this embodiment, in addition to displaying the driving mode, the direction (Heading) of the bus 300a and changes in acceleration in the Z direction (vertical direction) can be confirmed as described above, making it easier to detect the starting point of a crack CR, etc.

[0046] Third Embodiment Next, a third embodiment will be described in which the control center device 100b transmits the results of object recognition in an image to the analysis server 400 in addition to the image and sensor data.

[0047] Fig. 15 is a diagram showing the configuration of a third embodiment of the present invention. Referring to Fig. 15, the configuration includes a control center device 100b, a bus 300a, a base station 500, and an analysis server 400. Fig. 16 is a functional block diagram showing the configuration of the control center device 100b of the third embodiment of the present invention. The control center device 100b differs from the control center device 100a of the second embodiment shown in Fig. 12 in that an object recognition unit 106 is added to the control center device 100b, and the transmission unit 103 is configured to transmit the object recognition results to the analysis server 400 together with images and sensor data.

[0048] The object recognition means 106 performs a recognition process for objects captured in an image captured by a camera 310 mounted on the bus 300. The object recognition means 106 may, as necessary, confirm the position by also using sensor data from a sensor 320, such as GPS information, and perform the object recognition process. Note that the object recognition process in the object recognition means 106 can be performed by, for example, inputting the image itself or data extracted from a portion of the image in which the object is captured, into a classifier created in advance using machine learning or the like.

[0049] The transmission means 103 associates the above-mentioned driving mode information, camera images, sensor data, and object recognition results, and transmits them to the analysis server 400.

[0050] Next, the operation of this embodiment will be described in detail with reference to the drawings. Fig. 17 is a flowchart showing the operation of the control center device 100b according to the third embodiment of the present invention. The first difference from the operation of the control center device 100a according to the second embodiment shown in Fig. 13 is that the control center device 100b performs object recognition processing in steps S202 and S204. The second difference from the operation of the control center device 100a according to the second embodiment is that the control center device 100b transmits object recognition results to the analysis server 400 in addition to images, sensor data, etc. in steps S203 and S205.

[0051] FIG. 18 is a diagram showing an example of an image provided by the control center device 100b to the analysis server 400. In the example of FIG. 18, in addition to the same image as in the second embodiment, object recognition results for structures and the like appearing in the image are added. Specifically, in the example of FIG. 18, object recognition results for white lines WB on both sides of the road, a center line CL, and the like are added. Also, in the example of FIG. 18, a "?" mark is added to cracks that could not be identified as a result of object recognition.

[0052] According to this embodiment, in addition to displaying the operation mode and the sensor values, the results of object recognition can be confirmed, which makes it easier to detect cracks, etc. Also, as shown in Figure 18, objects that could not be recognized can be marked with a question mark or highlighted, which can prevent abnormalities from being overlooked.

[0053] Fourth Embodiment Next, a fourth embodiment will be described in which a function for estimating the cause of switching of the driving mode is added to the control center device 100c.

[0054] Fig. 19 is a diagram showing the configuration of a fourth embodiment of the present invention. Referring to Fig. 19, the configuration includes a control center device 100c, a bus 300a, a base station 500, and an analysis server 400. Fig. 20 is a functional block diagram showing the configuration of the control center device 100c of the fourth embodiment of the present invention. The difference from the control center device 100a of the second embodiment shown in Fig. 12 is that an estimation means 107 is added to the control center device 100c.

[0055] When a change in driving mode is detected, the estimation unit 107 estimates the cause of the change in driving mode based on the image captured by the camera 310. For example, as shown in FIG. 21 , if an image captured after switching from the autonomous driving mode to the non-autonomous driving mode (remote driving mode) shows people, bicycles, etc. ahead of the bus 300a who may come into contact with the bus 300a, the estimation unit 107 determines that the change in driving mode was not due to a road abnormality, but rather that the bus was switched to the non-autonomous driving mode (remote driving mode) in order to safely pass through an area where many people are present. In this case, the control center device 100c refrains from sending mode change information to the analysis server 400.

[0056] Next, the operation of this embodiment will be described in detail with reference to the drawings. Fig. 22 is a flowchart showing the operation of the control center device 100c of the fourth embodiment of the present invention. The difference from the operation of the control center device 100 of the first embodiment shown in Fig. 8 is that after detecting a change in the driving mode (Yes in step S001), the control center device 100c adds a process of determining whether the change in driving mode is caused by a pedestrian (step S301).

[0057] If it is determined that the change in driving mode is due to a pedestrian (Yes in step S301), the control center device 100c suppresses transmission of mode change information to the analysis server 400 (proceed to step S004).

[0058] On the other hand, if it is determined that the change in driving mode was not caused by a pedestrian (No in step S301), the control center device 100c creates mode change information and transmits the mode change information together with the image to the analysis server 400, as in the first embodiment (step S003). Note that if the driving mode is changed back from the remote driving mode to the autonomous driving mode after determining that the change in driving mode was caused by a pedestrian, the creation and transmission of the mode change information may be omitted.

[0059] According to this embodiment, which operates as described above, it is possible to suppress notification of a driving mode change to the analysis server 400 in areas where the driving mode is frequently changed. Note that, in the above description, an example was given in which the presence of a pedestrian or a bicycle was detected as the cause of the driving mode change, but the cause of the driving mode change is not limited to this. For example, transmission of the mode change information may also be suppressed in a similar manner if worsening weather or a malfunction of the camera 310 is inferred from the image.

[0060] In the first to fourth embodiments described above, examples have been given in which the control center devices 100 to 100c function as information processing devices that transmit data to the analysis server 400, which is a higher-level device, but the information processing device may also be located on the vehicle (bus) side. A fifth embodiment will now be described in which the function of transmitting mode change information and images is located on the bus side.

[0061] Fig. 23 is a diagram showing the configuration of a fifth embodiment of the present invention. Referring to Fig. 23, the configuration includes a control center device 200, a bus 300a equipped with an on-board terminal 600, a base station 500, and an analysis server 400. Fig. 24 is a functional block diagram showing the configuration of the on-board terminal 600 of the fifth embodiment of the present invention. Referring to Fig. 24, the configuration of the on-board terminal 600 includes a driving mode information acquisition means 601, an image acquisition means 602, and a transmission means 603.

[0062] The driving mode information acquisition means 601 acquires driving mode information of the bus 300 from an ECU (Electronic Control Unit) of the bus 300a, etc. When the driving mode of the bus 300 is changed, the driving mode information acquisition means 601 creates mode change information indicating the timing at which the driving mode was switched, as in the first to fourth embodiments.

[0063] The image acquisition means 602 acquires images from the camera 310 mounted on the bus 300a.

[0064] The transmission means 603 associates the mode change information with the camera image and transmits them to the control center device 200 as a higher-level device. The control center device 200, which has received the mode change information and the camera image, transmits the mode change information and the camera image to the analysis server 400.

[0065] Next, the operation of this embodiment will be described in detail with reference to the drawings. Fig. 25 is a flowchart showing the operation of the in-vehicle terminal 600 according to the fifth embodiment of the present invention. Referring to Fig. 25, the in-vehicle terminal 600 first checks whether a change in the driving mode has occurred (step S401).

[0066] If the check reveals that the driving mode has not been changed, the in-vehicle terminal 600 transmits the image acquired by the camera 310 to the control center device 200 (step S404).

[0067] On the other hand, if a change in the driving mode has occurred, the in-vehicle terminal 600 creates mode change information indicating the timing at which the driving mode was switched (step S402).

[0068] Next, the in-vehicle terminal 600 transmits the mode change information and the image acquired by the camera 310 to the control center device 100 (step S403). The control center device 200 transfers the data received in step S403 or S404 to the analysis server 400.

[0069] As described above, according to this embodiment, it is possible to make the in-vehicle terminal 600 function as an information processing device. In the above description, the in-vehicle terminal 600 has been described as having functions equivalent to the control center device 100 of the first embodiment, but it is also possible to add functions equivalent to the control center device 100 of the second to fourth embodiments to the in-vehicle terminal 600.

[0070] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of the present invention. For example, the network configurations, element configurations, and data representation formats shown in the drawings are examples intended to aid in understanding the present invention, and the present invention is not limited to the configurations shown in these drawings.

[0071] For example, although the above-described embodiments have been described with reference to an example in which the moving body is a bus, the moving body is not limited to a bus. For example, the present invention can be similarly applied to cases in which images are obtained from other vehicles that can switch between an autonomous driving mode and a non-autonomous driving mode, an AVG (Automated Guided Vehicle), and the like.

[0072] (Hardware Configuration) In each embodiment of the present disclosure, each component of each device represents a functional unit block. Some or all of the components of each device are realized by an arbitrary combination of an information processing device 900 and a program, for example, as shown in FIG. 26 . FIG. 26 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration: - CPU (Central Processing Unit) 901 - ROM (Read Only Memory) 902 - RAM (Random Access Memory) 903 - Program 904 loaded into RAM 903 - Storage device 905 that stores the program 904 - Drive device 907 that reads and writes to a recording medium 906 - Communication interface 908 that connects to a communication network 909 - Input / output interface 910 that inputs and outputs data - Bus 911 that connects each component

[0073] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes the function. That is, the CPU 901 in FIG. 26 executes an operation mode detection program and a sensor information acquisition program, and performs an update process for each calculation parameter stored in the RAM 903, the storage device 905, etc. The program 904 that realizes the function of each component of each device is stored in the storage device 905 or the ROM 902 in advance, for example, and is read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via the communication network 909, or may be stored in advance on the recording medium 906, and the drive device 907 may read the program and supply it to the CPU 901.

[0074] Furthermore, this program 904 can display the processing results, including intermediate states, at each stage as necessary on a display device, or can communicate with the outside via a communication interface. Furthermore, this program 904 can be recorded on a computer-readable (non-transitive) program recording medium.

[0075] There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a separate information processing device 900 and a program for each component. Furthermore, multiple components included in each device may be realized by any combination of a single information processing device 900 and a program. That is, the communication terminals and network control devices shown in the first to third embodiments, and processors installed in these devices, may be realized by computer programs that cause the above-described processes to be executed using the hardware.

[0076] In addition, some or all of the components of each device may be realized by other general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus.

[0077] Some or all of the components of each device may be realized by a combination of the above-mentioned circuits and programs.

[0078] When some or all of the components of each device are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each device is connected via a communication network.

[0079] It should be noted that the above-described embodiments are preferred embodiments of the present disclosure, and the scope of the present disclosure is not limited to only the above-described embodiments. In other words, those skilled in the art can modify or substitute the above-described embodiments to construct various modified forms without departing from the gist of the present disclosure.

[0080] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0081] [Supplementary Note 1] An information processing device comprising: a first acquisition means for acquiring timing at which a mobile body capable of switching between at least two driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; a second acquisition means for acquiring sensor information from a sensor for determining the state of an inspection object mounted on the mobile body; and a transmission means for transmitting mode change information indicating the timing at which the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a predetermined host device. [Supplementary Note 2] The non-autonomous driving mode of the information processing device is preferably a remote driving mode in which the mobile body is remotely driven or a manual driving mode in which the mobile body is manually driven. [Supplementary Note 3] The first acquisition means of the information processing device may further acquire timing at which the mobile body switched from the non-autonomous driving mode to the autonomous driving mode, and the transmission means may be configured to transmit second mode change information indicating the timing at which the mobile body switched from the autonomous driving mode to the non-autonomous driving mode to the predetermined host device. [Supplementary Note 4] In the above-mentioned information processing device, the sensor information is an image captured by a camera mounted on a mobile body, and the device may further include object recognition means for performing object recognition processing to recognize objects captured in the image, and the transmission means may be configured to transmit the results of the object recognition processing to the predetermined host device. [Supplementary Note 5] In the above-mentioned information processing device, the device may further include estimation means for estimating the cause of the mobile body switching from the autonomous driving mode to the non-autonomous driving mode based on the sensor information, and the device may be configured to suppress transmission of the mode change information if the cause is not attributable to the state of the object being inspected. [Supplementary Note 6] In the above-mentioned information processing device, the information processing device may be mounted on the mobile body, and the predetermined host device may be a control center that remotely operates the mobile body in the non-autonomous driving mode.[Supplementary Note 7] In the above-mentioned information processing device, the information processing device may be arranged in a control center that remotely drives the mobile body in the non-autonomous driving mode, and the predetermined host device may be a server that analyzes images received from the control center. [Supplementary Note 8] A server comprising: a receiving means that receives the mode change information and the sensor information from the above-mentioned information processing device; and a determining means that determines a state of an object to be inspected using the mode change information and the sensor information. [Supplementary Note 9] An inspection system including: an information processing device including: a first acquisition means for acquiring timing at which a mobile body that moves by autonomous driving and is capable of switching between at least two or more driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; a second acquisition means for acquiring sensor information from a sensor for determining the state of an object to be inspected that is mounted on the mobile body; and a transmission means for transmitting mode change information that indicates the timing at which the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a predetermined higher-level device; and a server including: a reception means for receiving the mode change information and the sensor information from the information processing device; and a determination means for determining the state of the object to be inspected using the mode change information and the sensor information. [Supplementary Note 10] A sensor data transmission method comprising: acquiring timing when a mobile body capable of switching between at least two or more driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; acquiring sensor information from a sensor for determining the state of an object to be inspected, which is mounted on the mobile body; and transmitting mode change information indicating the timing when the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a predetermined host device.[Supplementary Note 11] A program recording medium having recorded thereon a program that causes a computer to execute the following processes: a process of acquiring timing at which a mobile body that moves by autonomous driving and is capable of switching between at least two or more driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; a process of acquiring sensor information from a sensor for determining the state of an inspection object mounted on the mobile body; and a process of transmitting mode change information indicating the timing at which the autonomous driving mode was switched to the non-autonomous driving mode and the sensor information to a predetermined host device. Note that the forms of Supplementary Notes 9 to 11 above can be expanded into the forms of Supplementary Notes 2 to 7, as with Supplementary Note 1.

[0082] The disclosures of the above-cited patent documents are incorporated herein by reference and may be used as the basis or part of the present invention, as necessary. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims), and further based on the basic technical concepts thereof. Furthermore, various combinations and selections (including partial deletions) of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concepts, including the claims. In particular, with regard to the numerical ranges described herein, any numerical value or subrange within that range should be construed as specifically described, even if not otherwise specified. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, in accordance with the spirit of the present invention, are also deemed to be included in the disclosures of this application.

[0083] V1 Mobile body 10 Information processing device 11 First acquisition means 12 Second acquisition means 13 Transmission means 20 Upper device PH Pothole 100, 100a, 100b, 100c, 200 Control center device 101, 601 Driving mode information acquisition means 102, 602 Image acquisition means 103, 603 Transmission means 104 Remote control means 105 Sensor data acquisition means 106 Object recognition means 107 Estimation means 300, 300a Bus 310 Camera 320 Sensor 400 Analysis server 401 Receiving means 402 Determination means 500 Base station 600 In-vehicle terminal 900 Information processing device 901 CPU (Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus CR Crack M1 Message OP Operator S1 Heading information S2 Acceleration sensor information V1 Moving body

Claims

1. A first acquisition means for acquiring a timing at which a moving body capable of switching between at least two or more driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; A second acquisition means for acquiring sensor information from a sensor for determining a state of an inspection object mounted on the moving body; A transmission means for transmitting mode change information indicating a timing of switching from the autonomous driving mode to the non-autonomous driving mode and the sensor information to a predetermined host device; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the non-automatic driving mode is a remote driving mode in which the moving object is remotely driven or a manual driving mode in which the moving object is manually driven.

3. The first acquisition means further acquires a timing when the moving body switches from the non-automatic driving mode to the automatic driving mode.

3. The information processing apparatus according to claim 1, wherein the transmission means transmits, to the predetermined higher-level device, second mode change information indicating a timing at which the automatic driving mode is switched to the non-automatic driving mode.

4. the sensor information is an image captured by a camera mounted on the moving object, The apparatus further includes an object recognition unit that performs an object recognition process to recognize an object shown in the image, 3. The information processing apparatus according to claim 1, wherein the transmitting means transmits a result of the object recognition process to the predetermined host device.

5. The present invention further includes an estimation means for estimating a cause of the moving body switching from the autonomous driving mode to the non-autonomous driving mode based on the sensor information, 3. The information processing apparatus according to claim 1, wherein transmission of the mode change information is suppressed if the cause is not due to a condition of the object to be inspected.

6. The information processing device is mounted on the moving body, 3. The information processing device according to claim 1, wherein the predetermined host device is a control center that remotely drives the mobile object in the non-automatic driving mode.

7. The information processing device is disposed in a control center that remotely operates the moving object in the non-automated driving mode, 3. The information processing apparatus according to claim 1, wherein the predetermined host device is a server that analyzes the image received from the control center.

8. a receiving means for receiving the mode change information and the sensor information from the information processing device according to any one of claims 1 to 7; a determination means for determining a state of an object to be inspected by using the mode change information and the sensor information; A server equipped with

9. Acquire a timing at which a moving body capable of switching between at least two or more driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; acquiring sensor information from a sensor for determining a state of an inspection object mounted on the moving body; Transmitting mode change information indicating the timing of switching from the autonomous driving mode to the non-autonomous driving mode and the sensor information to a predetermined host device; Sensor data transmission method.

10. A process of acquiring a timing at which a moving body capable of switching between at least two or more driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode; A process of acquiring sensor information from a sensor for determining a state of an inspection object mounted on the moving body; A process of transmitting mode change information indicating a timing of switching from the autonomous driving mode to the non-autonomous driving mode and the sensor information to a predetermined upper device; A program that causes a computer to execute the following.

Images