Information processing device, server, sensor data transmission method, and program

Abstract

[Problem] To improve performance of inspecting an object under inspection by using a mobile object. [Solution] This information processing device comprises: a first acquisition means that acquires a timing at which a mobile object, which is capable of switching between at least two driving modes including an autonomous driving mode for movement by autonomous driving and a non-autonomous driving mode, has switched from the autonomous driving mode to the non-autonomous driving mode; a second acquisition means that acquires sensor information from a sensor that determines the state of a road and is mounted on the mobile object; and a transmission means that transmits, to a predetermined host device, the sensor information and mode change information indicating the timing at which the mobile object has switched from the autonomous driving mode to the non-autonomous driving mode.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a server, an inspection system, a sensor data transmission method, and a program recording medium.

Background Art

[0002] Patent Document 1 discloses a deterioration diagnosis system that can diagnose the degree of deterioration of public facilities by using a photographing device attached to a moving body. According to this document, this deterioration diagnosis system includes a degree-of-deterioration analysis unit that analyzes the degree of deterioration of an inspection object reflected in a photographed image taken by the photographing device.Moreover, this deterioration diagnosis system includes a priority calculation unit that calculates the priority of the inspection object based on the degree of deterioration of the same inspection object reflected in a plurality of the photographed images and the driving status information of the moving body.For example, when a sudden steering wheel or sudden braking operation is performed due to the deterioration of the inspection object, this deterioration diagnosis system performs an operation of raising the priority of the inspection object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described deterioration diagnosis system, in addition to the image taken by the photographing device attached to the moving body, the degree of deterioration and the priority of the inspection object are calculated using the driving status information of the moving body.However, in this method, since it depends on the image and the driving status, there is a possibility that the grasp of slight deterioration of the inspection object may be delayed.

[0005] The present invention aims to provide an information processing device, a server, an inspection system, a sensor data transmission method, and a program recording medium that can contribute to improving the performance of inspection of objects using a moving device. [Means for solving the problem]

[0006] From a first perspective, an information processing device is provided, comprising: a first acquisition means for acquiring the timing when a mobile body capable of switching between at least two driving modes, including an automated driving mode and a non-autonomous driving mode, switches from the automated driving mode to the non-autonomous driving mode; a second acquisition means for acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected; and a transmission means for transmitting mode change information indicating the timing of the switch from the automated driving mode to the non-autonomous driving mode, and the sensor information, to a predetermined higher-level device.

[0007] From a second perspective, a server is provided that includes a receiving means for receiving the mode change information and the sensor information from the information processing device described above, and a determination means for determining the state of an object to be inspected using the mode change information and the sensor information.

[0008] From a third perspective, an inspection system is provided that includes the information processing device and the server described above.

[0009] From a fourth perspective, a sensor data transmission method is provided which includes a mobile body capable of switching between at least two driving modes, including an automated driving mode and a non-autonomous driving mode, acquires the timing when the mobile body switches from the automated driving mode to the non-autonomous driving mode, acquires sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected, and transmits mode change information indicating the timing of the switch from the automated driving mode to the non-autonomous driving mode, along with the sensor information, to a predetermined higher-level device.

[0010] From a fifth perspective, the process involves acquiring the timing when a mobile body capable of switching between at least two driving modes, including an automated driving mode and a non-autonomous driving mode, switches from the automated driving mode to the non-autonomous driving mode; acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected; transmitting mode change information indicating the timing of the switch from the automated driving mode to the non-autonomous driving mode, and the sensor information to a predetermined higher-level device. A program recording medium is provided that contains a program that causes a computer to execute it. [Effects of the Invention]

[0011] According to the present invention, an information processing device, a server, an inspection system, a sensor data transmission method, and a program recording medium are provided that can contribute to improving the performance of inspection of objects using a moving object. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of one embodiment of the present invention. [Figure 2] This is a diagram illustrating the operation of one embodiment of the present invention. [Figure 3] This is a diagram illustrating an image provided by an information processing device of one embodiment of the present invention. [Figure 4] This is a diagram illustrating an image provided by an information processing device shown as a comparative example. [Figure 5] Another diagram illustrating an image provided by an information processing device of one embodiment of the present invention. [Figure 6] This figure shows the configuration of the first embodiment of the present invention. [Figure 7] This is a functional block diagram showing the configuration of the control center device according to the first embodiment of the present invention. [Figure 8] This is a flowchart illustrating the operation of the control center device according to the first embodiment of the present invention. [Figure 9]It is a diagram showing an example of an image (without mode change information) provided by the control center device to the analysis server. [Figure 10] It is a diagram showing an example of an image provided by the control center device of the first embodiment of the present invention to the server. [Figure 11] It is a diagram showing the configuration of the second embodiment of the present invention. [Figure 12] It is a functional block diagram showing the configuration of the control center device of the second embodiment of the present invention. [Figure 13] It is a flowchart showing the operation of the control center device of the second embodiment of the present invention. [Figure 14] It is a diagram showing an example of data provided by the control center device of the second embodiment of the present invention to the server. [Figure 15] It is a diagram showing the configuration of the third embodiment of the present invention. [Figure 16] It is a functional block diagram showing the configuration of the control center device of the third embodiment of the present invention. [Figure 17] It is a flowchart showing the operation of the control center device of the third embodiment of the present invention. [Figure 18] It is a diagram showing an example of data provided by the control center device of the third embodiment of the present invention to the server. [Figure 19] It is a diagram showing the configuration of the fourth embodiment of the present invention. [Figure 20] It is a functional block diagram showing the configuration of the control center device of the fourth embodiment of the present invention. [Figure 21] It is a flowchart showing the operation of the control center device of the fourth embodiment of the present invention. [Figure 22] It is a diagram showing an example of data provided by the control center device of the fourth embodiment of the present invention to the server. [Figure 23] It is a diagram showing the configuration of the fifth embodiment of the present invention. [Figure 24] It is a functional block diagram showing the configuration of the in-vehicle terminal of the fifth embodiment of the present invention. [Figure 25] It is a flowchart showing the operation of the in-vehicle terminal of the fifth embodiment of the present invention. [Figure 26] This figure shows the configuration of a computer capable of functioning as an information processing device according to the present invention. [Modes for carrying out the invention]

[0013] First, an overview of one embodiment of the present invention will be described with reference to the drawings. The reference numerals in the drawings appended to this overview are provided for convenience as examples to aid understanding and are not intended to limit the present invention to the illustrated embodiments. Furthermore, the connecting lines between blocks in the drawings and other references in the following description include both bidirectional and unidirectional lines. Unidirectional arrows schematically represent the flow of the main signal (data) and do not exclude bidirectional communication. The program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. This computer device is also configured to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless. While there are ports or interfaces at the input / output connection points of each block in the figures, they are omitted from the illustration.

[0014] In one embodiment, the present invention can be realized with a configuration including, as shown in Figure 1, an information processing device 10 that can communicate with a mobile body V1 capable of switching between at least two driving modes, including an automated driving mode and a non-autonomous driving mode, and a higher-level device 20 that receives sensor information from the information processing device 10.

[0015] More specifically, the information processing device 10 comprises a first acquisition means 11, a second acquisition means 12, and a transmission means 13. The first acquisition means 11 acquires the timing when the mobile body V1 switches from automatic driving mode to non-automatic driving mode. The second acquisition means 12 acquires sensor information from a sensor mounted on the mobile body for determining the state of the object to be inspected. The transmission means 13 transmits mode change information indicating the timing of the switch from automatic driving mode to non-automatic driving mode, along with the sensor information, to a predetermined higher-level device.

[0016] For example, let's consider a scenario where a pothole PH is present in the road surface ahead of the moving vehicle V1, as shown in Figure 2, and we want to detect this as an anomaly. The dotted line extending from the sensor mounted on the vehicle in Figure 2 indicates the sensing range of the sensor. Point P1 indicates the point where the driver of the moving vehicle V1 or the person controlling it remotely (operator) visually confirmed the pothole PH and switched from automated driving mode to non-autonomous driving mode for reasons such as improving safety. Non-autonomous driving modes could include direct control mode (manual driving mode) where the driver operates the vehicle manually, or remote driving mode where the remote control operator operates the vehicle remotely.

[0017] As shown in Figure 3, when the information processing device 10 of this embodiment switches from automatic driving mode to non-automatic driving mode, it transmits mode change information and sensor information to the higher-level device 20. As a result, as shown in Figure 3, the higher-level device 20 can perform a detailed analysis of the sensor information from before the pothole PH. Consequently, it becomes possible to detect cracks in the road surface before the pothole PH at an early stage.

[0018] Figure 4 is an illustrative diagram of how sensor information is analyzed using the method described in the background technology. As shown in Figure 4, while the degree of road surface deterioration can be determined from the sensor information, information regarding the timing of the switch from autonomous driving mode to non-autonomous driving mode cannot be obtained. There is a possibility that cracks in the road surface located before potholes PH may be overlooked.

[0019] Furthermore, the information processing device 10 of this embodiment may also be configured to promptly transmit mode change information and sensor information to the higher-level device 20 when switching from non-autonomous driving mode to autonomous driving mode. For example, point P2 in Figure 5 indicates the point where the vehicle switches from non-autonomous driving mode to autonomous driving mode. In this case, the information processing device 10 will transmit the mode change information and sensor information to the higher-level device 20. Non Autonomous driving mode? FromAs soon as the system switches to manual operation mode, it promptly transmits mode change information indicating a return to automatic operation mode, along with sensor information, to the higher-level device 20. This allows the higher-level device 20 to complete detailed analysis of the sensor information. This reduces the transmission cost of sensor information and the computational cost of the higher-level device 20.

[0020] As explained above, this embodiment makes it possible to improve the performance of inspecting objects using the mobile unit V1. In the example described above, the driver of the mobile unit V1 or the remote control operator switched from automatic driving mode to non-automatic driving mode, but the person responsible for switching from automatic driving mode to non-automatic driving mode is not limited to these individuals. For example, the mobile unit V1 may switch to non-automatic driving mode if it determines that automatic driving is not possible based on the information it senses. In this case, the mobile unit V1 will notify the driver or remote control operator and then transmit mode change information and sensor information to the higher-level device 20. Cases in which the mobile unit V1 itself determines that automatic driving is not possible include when the mobile unit V1 detects an abnormality regarding the road or surrounding conditions, or when it detects a misperception regarding the road or surrounding conditions. Automatic driving may also include autonomous driving in which the mobile unit V1 autonomously determines whether or not it is possible to drive based on the information it senses.

[0021] [First Embodiment] Next, a first embodiment in which the functions of the present invention as an information processing device are located in a control center device 100 that controls a mobile body will be described in detail with reference to the drawings. Furthermore, in the following description, an example will be given in which the mobile body is a bus equipped with an autonomous driving function.

[0022] Figure 6 shows the configuration of a first embodiment of the present invention. Referring to Figure 6, the configuration of a road inspection system is shown, which includes a control center device 100, a bus 300, a base station 500 of a mobile communication network that enables communication between the bus 300 and the control center device 100, and an analysis server 400.

[0023] Bus 300 can switch between an automatic driving mode, in which it operates automatically based on information it senses itself, and a remote driving mode, in which it is operated by remote control from the control center device 100. Bus 300 is also equipped with a camera 310 capable of capturing images of the area in front of it. The switching between automatic and remote driving modes may be performed by the bus 300 itself or by the control center device 100. In the following description, it will be assumed that the operator OP at the control center device 100 switches between automatic and remote driving modes by referring to the image from the camera 310. In addition to the automatic and remote driving modes, bus 300 may also have a manual driving mode, for example. In this case, the switching of the driving mode may also be performed by the bus 300 itself or by the control center device 100.

[0024] Figure 7 is a functional block diagram showing the configuration of a control center device 100 according to the first embodiment of the present invention. Referring to Figure 7, the configuration of the control center device 100 is shown, which includes an operating mode information acquisition means 101, an image acquisition means 102, a transmission means 103, and a remote control means 104.

[0025] The remote control means 104 provides the operator OP with information from instruments mounted on the bus and images from the camera 310 using a display device (not shown). The operator OP operates the bus 300 based on the information provided by the remote control means 104. Specifically, the remote control means 104 receives commands from the information operator OP for operation of the bus 300 and commands to be given to the bus 300, and remotely controls the bus 300.

[0026] The driving mode information acquisition means 101 acquires driving mode information of the bus 300 from the remote control means 104. If there is a change in the driving mode of the bus 300, the driving mode information acquisition means 101 creates mode change information indicating the timing of the driving mode switch. Therefore, the driving mode information acquisition means 101 functions as a first acquisition means 11 that acquires the timing of the switch from automatic driving mode to non-automatic driving mode. This mode change information can be configured to include, for example, information on the changed driving mode and time information indicating the timing of the change.

[0027] The image acquisition means 102 acquires images from the camera 310 mounted on the bus 300. Therefore, the image acquisition means 102 functions as a second acquisition means 12 that acquires sensor information from a sensor for determining the state of an object to be inspected, which is mounted on the moving body.

[0028] The transmission means 103 transmits mode change information and camera images to the analysis server 400, which acts as a higher-level device. The transmission timing of the mode change information and camera images does not necessarily have to be simultaneous. For example, camera images may be transmitted periodically while mode change information is transmitted only when necessary. Even with such a transmission configuration, the analysis server 400 can analyze the images without problems by referring to the mode change information.

[0029] The analysis server 400 includes a receiving means (reference numeral 401 in Figure 6) for receiving mode change information and sensor information, and a determination means (reference numeral 402 in Figure 6) for determining the road condition using the mode change information and sensor information.

[0030] Base station 500 is a base station for the fifth-generation mobile communication system (5G) and LTE (Long Term Evolution).

[0031] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 8 is a flowchart showing the operation of the control center device 100 of the first embodiment of the present invention. Referring to Figure 8, first the control center device 100 checks whether or not a change in the operating mode has occurred (step S001).

[0032] If the confirmation confirms that no change in the operating mode has occurred, the control center device 100 transmits the image acquired from the camera 310 to the analysis server 400 (step S004).

[0033] On the other hand, if a change in the operating mode occurs, the control center device 100 creates mode change information indicating the timing of the switch in the operating mode (step S002).

[0034] Next, the control center device 100 transmits the mode change information and the image acquired from the camera 310 to the analysis server 400 (step S003).

[0035] Figure 9 shows an example of an image (without mode change information) that the control center device 100 provides to the analysis server 400. In the example in Figure 9, the image of the road is shown by the camera 310 on the bus 300. In the example in Figure 9, a crack (CR) has occurred near the center line in the middle of the road. Since such a crack (CR) may affect autonomous driving, the operator (OP) switches from autonomous driving mode to remote driving mode as soon as they detect an abnormality in the distance.

[0036] Figure 10 shows an example of an image provided by the control center device 100 to the analysis server 400 using mode change information. In the example in Figure 10, the message M1 "Remote control operation in progress" is added to the upper right of the image. Note that in the example in Figure 10, "Remote control operation in progress" is displayed, but during automatic operation, "Automatic operation in progress" will be displayed. Personnel viewing the analysis server 400 or its display device can detect road abnormalities early by such changes in the display of the driving mode. In addition, by changing the image analysis mode in response to such changes in the driving mode, the analysis server 400 can also easily detect the starting point of cracks CR, etc.

[0037] Furthermore, once the section of road abnormality has ended, the operator (OP) switches from remote driving mode to automatic driving mode. In this case, the control center device (100) acquires a second mode change information indicating the timing of the switch from non-automatic driving mode to automatic driving mode and transmits it to the analysis server (400). As a result, the display changes from "Remote Control Driving" to "Automatic Driving." Personnel viewing the analysis server (400) or its display device can detect the end of the abnormal section of the road by this change in the driving mode display. In addition, by changing the image analysis mode in response to such changes in driving mode, the analysis server (400) can also easily detect the end of cracks (CR), etc. Although the examples in Figures 9 and 10 illustrate the detection of cracks as a road abnormality, the analysis server (400) is not limited to detecting cracks. By implementing inspection functions according to the type of object being inspected in the analysis server (400), it becomes possible to detect other road abnormalities and inspect structures other than roads.

[0038] [Second Embodiment] In the above-described embodiment, the control center device 100 was described as transmitting images to the analysis server 400, but it can also receive other sensor data from the bus 300 and send it to the analysis server 400. Below, a second embodiment will be described in which the control center device 100a transmits sensor information to the analysis server 400.

[0039] Figure 11 shows the configuration of a second embodiment of the present invention. Referring to Figure 11, a configuration is shown that includes a control center device 100a, a bus 300a equipped with a sensor 320, a base station 500 of a mobile communication network that enables communication between the bus 300a and the control center device 100a, and an analysis server 400. This embodiment differs from the first embodiment in that the bus 300a is equipped with a sensor 320 and the control center device 100a has a transmission function for the sensor 320. The other configurations are the same as in the first embodiment, so the differences will be described below.

[0040] Figure 12 is a functional block diagram showing the configuration of a control center device 100a of a second embodiment using the sensor data transmission method of the present invention. The difference from the control center device 100 of the first embodiment shown in Figure 7 is that a sensor data acquisition means 105 is added to the control center device 100a, and the transmission means 103 is configured to transmit sensor data along with images to the analysis server 400.

[0041] The sensor data acquisition means 105 acquires sensor data from the sensor 320 mounted on the bus 300a. Possible sensors for the sensor 320 include a vehicle speedometer, steering angle information, GPS (Global Positioning System) information, LiDAR (Light Detection and Ranging), acceleration sensor, millimeter-wave radar, etc.

[0042] The transmission means 103 associates the driving mode information, the camera image, and the sensor data described above and transmits them to the analysis server 400.

[0043] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 13 is a flowchart showing the operation of the control center device 100a of the second embodiment of the present invention. The difference from the operation of the control center device 100 of the first embodiment shown in Figure 8 is that in steps S103 and S104, the control center device 100a transmits sensor data in addition to images to the analysis server 400.

[0044] Figure 14 shows an example of an image provided by the control center device 100a to the analysis server 400 using various information obtained from the bus 300a. In the example of Figure 14, in addition to the image similar to that of the first embodiment, the right side of Figure 14 displays sensor data. The code S1 represents, for example, the orientation of the bus 300a (heading information) from the steering angle of the steering wheel or GPS information. The code S2 indicates the value of the acceleration sensor in the Z direction (vertical direction) of the bus 300a.

[0045] According to this embodiment, in addition to displaying the operating mode, the orientation (heading) and acceleration changes in the Z direction (vertical direction) of the bus 300a can be confirmed as described above, making it easier to detect the starting point of the crack CR, etc.

[0046] [Third Embodiment] Next, a third embodiment will be described in which the control center device 100b transmits the object recognition results in the image to the analysis server 400, in addition to the image and sensor data.

[0047] Figure 15 is a diagram showing the configuration of a third embodiment of the present invention. Referring to Figure 15, a configuration including a control center device 100b, a bus 300a, a base station 500, and an analysis server 400 is shown. Figure 16 is a functional block diagram showing the configuration of the control center device 100b of the third embodiment of the present invention. The difference from the control center device 100a of the second embodiment shown in Figure 12 is that an object recognition means 106 is added to the control center device 100b, and the transmission means 103 is configured to transmit the object recognition results along with images and sensor data to the analysis server 400.

[0048] The object recognition means 106 performs object recognition processing on images captured by the camera 310 mounted on the bus 300. The object recognition means 106 may, if necessary, use sensor data from the sensor 320, such as GPS information, to confirm the location and perform object recognition processing. The object recognition processing in the object recognition means 106 can be performed, for example, by inputting the image itself or data extracted from the image containing the object into a classifier created in advance using machine learning or the like.

[0049] The transmission means 103 associates the above-mentioned driving mode information, camera image, sensor data, and object recognition results and transmits them to the analysis server 400.

[0050] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 17 is a flowchart showing the operation of the control center device 100b of the third embodiment of the present invention. The first difference from the operation of the control center device 100a of the second embodiment shown in Figure 13 is ,vinegar In steps S202 and S204, the control center device 100b performs object recognition processing. Another difference from the operation of the control center device 100a in the second embodiment is that in steps S203 and S205, the control center device 100b transmits the object recognition results to the analysis server 400 in addition to images, sensor data, etc.

[0051] Figure 18 shows an example of an image provided by the control center device 100b to the analysis server 400. In the example in Figure 18, in addition to the image similar to that of the second embodiment, object recognition results for structures and other objects in the image are added. Specifically, in the example in Figure 18, object recognition results for white lines (WB) on both sides of the road, center line (CL), etc. are added. Also, in the example in Figure 18, a "?" mark is added to cracks that could not be identified as a result of object recognition.

[0052] According to this embodiment, in addition to displaying the operating mode and sensor values, the results of object recognition can be confirmed, making it easier to detect cracks and other abnormalities. Furthermore, as shown in Figure 18, by marking objects that could not be recognized with a "?" or highlighting them, it is possible to prevent overlooking abnormalities.

[0053] [Fourth Embodiment] Next, a fourth embodiment will be described in which a function for estimating the cause of the operation mode switch is added to the control center device 100c.

[0054] Figure 19 is a diagram showing the configuration of the fourth embodiment of the present invention. Referring to Figure 19, a configuration including a control center device 100c, a bus 300a, a base station 500, and an analysis server 400 is shown. Figure 20 is a functional block diagram showing the configuration of the control center device 100c of the fourth embodiment of the present invention. The difference from the control center device 100a of the second embodiment shown in Figure 12 is that an estimation means 107 has been added to the control center device 100c.

[0055] When a change in driving mode is detected, the estimation means 107 estimates the cause of the change in driving mode based on the image captured by the camera 310. For example, as shown in Figure 21, if the image taken after the switch from automatic driving mode to non-automatic driving mode (remote driving mode) shows people, bicycles, etc., that could potentially come into contact with the bus 300a ahead of the bus 300a, the estimation means 107 determines that the change in driving mode was not due to a road abnormality, but rather to safely pass through an area with a large number of people in it. In this case, the control center device 100c suppresses the transmission of mode change information to the analysis server 400.

[0056] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 22 is a flowchart showing the operation of the control center device 100c of the fourth embodiment of the present invention. The difference from the operation of the control center device 100 of the first embodiment shown in Figure 8 is that, after detecting a change in the driving mode (Yes in step S001), the control center device 100c has an additional process to determine whether or not the change in the driving mode is caused by a pedestrian (step S301).

[0057] If the determination is made that the change in driving mode was caused by a pedestrian (Yes in step S301), the control center device 100c suppresses the transmission of mode change information to the analysis server 400 (proceed to step S004).

[0058] On the other hand, if it is determined that the change in driving mode is not caused by a pedestrian (No. in step S301), the control center device 100c creates mode change information and transmits the mode change information along with the image to the analysis server 400, similar to the first embodiment (step S003). However, if the driving mode returns from remote driving mode to automatic driving mode after it has been determined that the change in driving mode is caused by a pedestrian, the creation and transmission of mode change information may be omitted.

[0059] According to this embodiment, which operates as described above, it is possible to suppress notifications of driving mode changes to the analysis server 400 in areas where the driving mode is frequently switched. In the above description, examples were given in which the presence of pedestrians or bicycles was detected as the cause of the driving mode switch, but the cause of the driving mode switch is not limited to these. For example, if deterioration of weather or malfunction of the camera 310 is estimated from the image, the transmission of mode change information may also be suppressed in the same way.

[0060] [Fifth Embodiment] In the first to fourth embodiments described above, the control center devices 100 to 100c were described as functioning as information processing devices that transmit data to the higher-level analysis server 400. However, the information processing devices may also be located on the vehicle (bus) side. Below, a fifth embodiment in which the mode change information and image transmission functions are located on the bus side will be described.

[0061] Figure 23 is a diagram showing the configuration of a fifth embodiment of the present invention. Referring to Figure 23, a configuration including a control center device 200, a bus 300a equipped with an in-vehicle terminal 600, a base station 500, and an analysis server 400 is shown. Figure 24 is a functional block diagram showing the configuration of an in-vehicle terminal 600 of the fifth embodiment of the present invention. Referring to Figure 24, a configuration of an in-vehicle terminal 600 equipped with a driving mode information acquisition means 601, an image acquisition means 602, and a transmission means 603 is shown.

[0062] The driving mode information acquisition means 601 acquires driving mode information of the bus 300 from the ECU (Electronic Control Unit) of the bus 300a, etc. If there is a change in the driving mode of the bus 300, the driving mode information acquisition means 601 creates mode change information indicating the timing of the driving mode switch, similar to the first to fourth embodiments.

[0063] The image acquisition means 602 acquires images from the camera 310 mounted on the bus 300a.

[0064] The transmission means 603 associates the mode change information with the camera image and transmits it to the control center device 200, which acts as a higher-level device. The control center device 200, having received the mode change information and the camera image, transmits the mode change information and the camera image to the analysis server 400.

[0065] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 25 is a flowchart showing the operation of the in-vehicle terminal 600 of the fifth embodiment of the present invention. Referring to Figure 25, first the in-vehicle terminal 600 checks whether a change in the driving mode has occurred (step S401).

[0066] If the confirmation confirms that no change in the driving mode has occurred, the in-vehicle terminal 600 transmits the image acquired from the camera 310 to the control center device 200 (step S404).

[0067] On the other hand, if a change in the driving mode occurs, the in-vehicle terminal 600 creates mode change information indicating the timing of the change in driving mode (step S402).

[0068] Next, the in-vehicle terminal 600 transmits mode change information and images acquired from the camera 310 to the control center device 100 (step S403). The control center device 200 then forwards the data received in step S403 or S404 to the analysis server 400.

[0069] As described above, this embodiment makes it possible to make the in-vehicle terminal 600 function as an information processing device. Furthermore, although the above description assumed that the in-vehicle terminal 600 has functions equivalent to the control center device 100 of the first embodiment, it is also possible to add functions equivalent to the control center device 100 of the second to fourth embodiments to the in-vehicle terminal 600.

[0070] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and further modifications, substitutions, and adjustments can be made without departing from the basic technical idea of ​​the present invention. For example, the network configuration, the configuration of each element, and the data representation form shown in each drawing are examples to aid in understanding the present invention, and the present invention is not limited to the configurations shown in these drawings.

[0071] For example, in the embodiments described above, the example of a bus as the moving object was used, but the moving object is not limited to a bus. For example, the same method can be applied when obtaining images from other vehicles that can switch between autonomous driving mode and non-autonomous driving mode, or from an AVG (Automated Guided Vehicle).

[0072] (Regarding hardware configuration) In each embodiment of this disclosure, each component of each device represents a functional unit block. Some or all of each component of each device is realized by any combination of an information processing device 900 and a program, for example, as shown in Figure 26. Figure 26 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration. ·CPU(Central Processing Unit)901 • ROM (Read Only Memory) 902 ·RAM(Random Access Memory)903 • Program 904 loaded into RAM903 • Storage device 905 for storing program 904 • Drive device 907 for reading and writing recording medium 906 • Communication interface 908 connected to communication network 909 • Input / output interface 910 for data input and output. • Bus 911 connecting each component

[0073] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes these functions. That is, the CPU 901 in Figure 26 executes the operation mode detection program and the sensor information acquisition program, and performs the update process of each calculation parameter held in the RAM 903, storage device 905, etc. The program 904 that realizes the functions of each component of each device is, for example, stored in advance in the storage device 905 or ROM 902, and read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via the communication network 909, or it may be stored in advance in the recording medium 906, and the drive device 907 may read the program and supply it to the CPU 901.

[0074] Furthermore, this program 904 can display its processing results, including intermediate states, step by step via a display device, or communicate with the outside world via a communication interface, as needed. This program 904 can also be recorded on a computer-readable (non-transitive) program recording medium.

[0075] There are various variations in how each device is implemented. For example, each device may be implemented by any combination of a separate information processing device 900 and a program for each component. Alternatively, the multiple components of each device may be implemented by any combination of a single information processing device 900 and a program. In other words, the communication terminals and network control devices shown in the first to third embodiments described above, and the processors mounted on these devices, can be implemented by computer programs that use their hardware to execute the aforementioned processes.

[0076] Furthermore, some or all of the components of each device are realized by other general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be made up of a single chip or multiple chips connected via a bus.

[0077] Some or all of the components of each device may be realized by a combination of the circuits and programs described above.

[0078] When some or all of the components of each device are implemented by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-and-server system or a cloud computing system.

[0079] The embodiments described above are preferred embodiments of this disclosure and do not limit the scope of this disclosure to these embodiments alone. That is, a person skilled in the art can modify or substitute the embodiments described above to construct various modified forms without departing from the gist of this disclosure.

[0080] Some or all of the above embodiments may also be described as follows, but are not limited to these.

[0081] [Note 1] A first acquisition means for acquiring the timing when a mobile body capable of switching between at least two driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode, A second acquisition means for acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected, A transmission means for transmitting mode change information indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode, and the sensor information to a predetermined higher-level device. An information processing device equipped with the following features. [Note 2] The non-automatic operation mode of the information processing device described above is preferably a remote operation mode for remotely operating the mobile body or a manual operation mode for manually operating the mobile body. [Note 3] The first acquisition means of the information processing device described above further acquires the timing when the moving body switches from the non-autonomous driving mode to the autonomous driving mode. The transmitting means may be configured to transmit a second mode change information to a predetermined higher-level device, indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode. [Note 4] In the above-mentioned information processing device, The aforementioned sensor information is an image captured by a camera mounted on a mobile device. Furthermore, the system includes object recognition means for performing object recognition processing to recognize objects depicted in the image, The transmission means can be configured to transmit the results of the object recognition process to the predetermined higher-level device. [Note 5] In the above-mentioned information processing device, Furthermore, the system includes estimation means for estimating the cause of the mobile body switching from the automatic driving mode to the non-automatic driving mode based on the sensor information, If the aforementioned cause is not due to the state of the object being inspected, a configuration can be adopted to suppress the transmission of the mode change information. [Note 6] In the above-mentioned information processing device, The information processing device is mounted on the mobile body, The aforementioned predetermined higher-level device can be configured as a control center that remotely operates the mobile body during the non-automatic operation mode. [Note 7] In the above-mentioned information processing device, The information processing device is located in a control center that remotely operates the mobile body during the non-automatic driving mode. The aforementioned predetermined higher-level device can be configured as a server that analyzes images received from the control center. [Note 8] A receiving means for receiving the mode change information and the sensor information from the information processing device described above, A determination means that uses the mode change information and the sensor information to determine the state of the object to be inspected, A server equipped with [the following features]. [Note 9] An information processing device comprising: a first acquisition means for acquiring the timing when a mobile body capable of switching between at least two driving modes, including an automated driving mode and a non-autonomous driving mode, switches from the automated driving mode to the non-autonomous driving mode; a second acquisition means for acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected; and a transmission means for transmitting mode change information indicating the timing of the switch from the automated driving mode to the non-autonomous driving mode, and the sensor information, to a predetermined higher-level device; A server comprising: receiving means for receiving the mode change information and the sensor information from the information processing device; and determination means for determining the state of an object to be inspected using the mode change information and the sensor information; A testing system that includes this. [Note 10] A mobile vehicle capable of switching between at least two driving modes, including an autonomous driving mode and a non-autonomous driving mode, acquires the timing when it switches from the autonomous driving mode to the non-autonomous driving mode. Sensor information is acquired from a sensor mounted on the aforementioned mobile body for determining the state of the object to be inspected. Mode change information indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode, and the sensor information are transmitted to a predetermined higher-level device. Sensor data transmission method. [Note 11] A process for acquiring the timing when a mobile body capable of switching between at least two driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode, The process involves acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected, A process of transmitting mode change information indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode, and the sensor information to a predetermined higher-level device, A program recording medium that stores a program that causes a computer to execute. Furthermore, the forms described in appendices 9 to 11 above can be expanded into the forms described in appendices 2 to 7, similar to appendice 1.

[0082] Furthermore, each disclosure in the above-mentioned patent documents is incorporated into this document by reference and may be used as the basis or part of the present invention as necessary. Within the framework of the full disclosure of the present invention (including the claims), further modifications and adjustments to the embodiments or examples are possible based on the basic technical concept. Also, within the framework of the disclosure of the present invention, various combinations or selections (including partial deletions) of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. In particular, with respect to the numerical ranges described in this document, any numerical value or sub-range included within that range should be interpreted as being specifically described, even if not otherwise stated. Furthermore, each disclosure in the above-mentioned cited documents may, as necessary, be used in part or in whole as part of the disclosure of the present invention, in accordance with the spirit of the present invention, and this is also considered to be included in the disclosure of this application. [Explanation of symbols]

[0083] V1 Mobile Unit 10 Information Processing Devices 11. First means of acquisition 12. Second means of acquisition 13 Transmission method 20 Higher-level devices pH potholes 100, 100a, 100b, 100c, 200 Control Center Equipment 101, 601 Means for acquiring operating mode information 102, 602 Image acquisition means 103, 603 Transmission means 104 Remote control means 105 Sensor data acquisition means 106 Object recognition means 107 Estimation means 300, 300a bus 310 Camera 320 sensors 400 analysis servers 401 Receiving means 402 Judgment means 500 base stations 600 In-vehicle terminals 900 Information Processing Equipment 901 CPU(Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording media 907 Drive unit 908 Communication Interface 909 Communication Network 910 Input / Output Interface 911 Bus CR cracks M1 Message OP Operator S1 Heading Information S 2 Accelerometer sensor information Information

Claims

1. A first acquisition means for acquiring the timing when a mobile body capable of switching between at least two driving modes, including an automated driving mode and a non-autonomous driving mode, switches from the automated driving mode to the non-autonomous driving mode, A second acquisition means for acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected, A transmission means for transmitting mode change information indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode, and the sensor information to a predetermined higher-level device. Equipped with, Furthermore, the system includes estimation means for estimating the cause of the mobile body switching from the automatic driving mode to the non-automatic driving mode based on the sensor information, If the aforementioned cause is not due to the condition of the object being inspected, the transmission of the mode change information will be suppressed. Information processing device.

2. The information processing apparatus according to claim 1, wherein the non-automatic driving mode is a remote driving mode in which the mobile body is driven remotely or a manual driving mode in which the mobile body is driven manually.

3. The first acquisition means further acquires the timing when the moving body switches from the non-autonomous driving mode to the autonomous driving mode. The information processing apparatus according to claim 1 or 2, wherein the transmitting means transmits to the predetermined higher-level device a second mode change information indicating the timing of the switch from the non-automatic driving mode to the automatic driving mode.

4. The sensor information is an image captured by a camera mounted on the mobile body. Furthermore, the system includes object recognition means for performing object recognition processing to recognize objects depicted in the image, The information processing apparatus according to claim 1 or 2, wherein the transmission means transmits the result of the object recognition process to the predetermined higher-level device.

5. The information processing device is mounted on the mobile body, The information processing device according to claim 1 or 2, wherein the predetermined higher-level device is a control center that remotely operates the mobile body during the non-automatic operation mode.

6. The information processing device is located in a control center that remotely operates the mobile body during the non-automatic driving mode. The information processing device according to claim 1 or 2, wherein the predetermined higher-level device is a server that analyzes images received from the control center.

7. Receiving means for receiving the mode change information and the sensor information from the information processing apparatus of claim 1 or 2, A determination means that determines the state of the object to be inspected using the mode change information and the sensor information, A server equipped with [the following features].

8. A mobile vehicle capable of switching between at least two driving modes, including an autonomous driving mode and a non-autonomous driving mode, acquires the timing when it switches from the autonomous driving mode to the non-autonomous driving mode. Sensor information is acquired from a sensor mounted on the aforementioned mobile body for determining the state of the object to be inspected. Furthermore, based on the sensor information, the cause of the mobile body switching from the automatic driving mode to the non-automatic driving mode is estimated. If the cause is due to the condition of the object being inspected, mode change information indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode, and the sensor information are transmitted to a predetermined higher-level device. If the aforementioned cause is not due to the condition of the object being inspected, the transmission of the mode change information will be suppressed. Sensor data transmission method.

9. A process for acquiring the timing when a mobile body capable of switching between at least two driving modes, including an autonomous driving mode and a non-autonomous driving mode, switches from the autonomous driving mode to the non-autonomous driving mode, The process involves acquiring sensor information from a sensor mounted on the mobile body for determining the state of an object to be inspected, A process of transmitting mode change information indicating the timing of the switch from the automatic driving mode to the non-automatic driving mode, and the sensor information to a predetermined higher-level device, Have the computer run it, Furthermore, the process involves estimating the cause of the mobile body switching from the automatic driving mode to the non-automatic driving mode based on the sensor information, If the aforementioned cause is not due to the state of the object being inspected, the process of suppressing the transmission of the mode change information is performed. A program that is executed by a computer.

Images