Programs, server equipment, and in-vehicle equipment

Abstract

Sharing and utilizing information about in-vehicle incidents across multiple vehicles. [Solution] The program according to the embodiment causes a server device, which is provided to be able to communicate with a plurality of in-vehicle devices installed in a plurality of vehicles carrying passengers, to execute a receiving procedure for receiving incident information regarding incidents that occur inside the vehicle from the plurality of in-vehicle devices; an analysis procedure for analyzing combinations of elements of the incident information that are likely to cause the incident based on the received incident information; a generation procedure for generating shared information among the plurality of in-vehicle devices that indicates situations in which the incident is likely to occur based on the analyzed combinations of elements; and a transmission procedure for transmitting the shared information to the plurality of in-vehicle devices.

Description

Technical Field

[0007] ,

[0001] The disclosed embodiments relate to a program, a server device, and an in-vehicle device.

Background Art

[0002] Conventionally, vehicles that perform passenger transportation, typified by shared vehicles in the general shared passenger automobile transportation business under the Road Transportation Law, such as route buses, demand buses, and shared taxis, are known.

[0003] In such vehicles, incidents such as passengers losing their balance, being exposed to large shakes and feeling uncomfortable, etc. may occur inside the vehicle due to factors such as the way the driver drives at curves or corners, or the situation where passengers do not hold handrails or suspension straps. Eventually, it may even lead to accidents such as passengers falling down, rather than just incidents.

[0004] Regarding such situations, for example, in Patent Document 1, a technique has been proposed that detects the fall or stagger of passengers and estimates emotions such as the uneasiness of passengers from the images of cameras inside the vehicle, and evaluates the driver based on the results. Such a technique is considered applicable to the analysis and prediction of situations where incidents occur inside the vehicle.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the above-mentioned conventional technology has not been sufficiently considered in terms of sharing and utilizing information regarding incidents inside the vehicle among multiple vehicles.

[0007] For example, in public transport vehicles, the combination of driver and passengers is almost always different. Furthermore, there are many different combinations of various situations, such as the age range of passengers, the degree of crowding, and whether passengers are standing or sitting. In contrast, the conventional technology described above is based on the premise of evaluating the individual driver when an incident occurs in an individual vehicle, and is not capable of analyzing or predicting situations that take into account the various combinations of situations mentioned above.

[0008] Therefore, when using the conventional technology described above, it is not possible to share and utilize information about incidents occurring inside the vehicle among multiple types of passenger vehicles, such as route buses and on-demand buses.

[0009] One embodiment, made in view of the above, aims to provide a program, server device, and in-vehicle device that can share and utilize information regarding incidents inside a vehicle among multiple vehicles. [Means for solving the problem]

[0010] A program according to one embodiment causes a server device, which is provided to be able to communicate with a plurality of in-vehicle devices installed in a plurality of vehicles carrying passengers, to execute a receiving procedure for receiving incident information regarding incidents occurring inside the vehicle from the plurality of in-vehicle devices; an analysis procedure for analyzing combinations of elements of the incident information that are likely to cause the incident to occur based on the received incident information; a generation procedure for generating shared information among the plurality of in-vehicle devices that indicates situations in which the incident is likely to occur based on the analyzed combinations of elements; and a transmission procedure for transmitting the shared information to the plurality of in-vehicle devices. [Effects of the Invention]

[0011] According to one embodiment, incident information received from multiple in-vehicle devices is analyzed in an integrated manner, and information indicating situations prone to incidents is shared and provided based on combinations of elements of incident information that are likely to cause incidents. This allows information regarding in-vehicle incidents to be shared and utilized among multiple vehicles. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is an overview diagram of the information provision method according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information provision system according to an embodiment. [Figure 3] Figure 3 shows an example of vehicle data. [Figure 4] Figure 4 shows an example of ambient condition data. [Figure 5] Figure 5 shows an example of passenger data. [Figure 6] Figure 6 shows an example of the configuration of an in-vehicle device according to the embodiment. [Figure 7] Figure 7 shows an example of the configuration of a center device according to an embodiment. [Figure 8] Figure 8 shows the processing sequence executed by the information provision system according to the embodiment. [Figure 9] Figure 9 shows an example of output information based on incident sharing information for route buses. [Figure 10] Figure 10 shows an example of output information based on incident sharing information in the on-demand bus. [Figure 11] Figure 11 shows an example of output information based on incident sharing information on a user terminal. [Modes for carrying out the invention]

[0013] Hereinafter, referring to the accompanying drawings, embodiments of the program, server device, and in-vehicle device disclosed in the present application will be described in detail. Note that the present invention is not limited to the embodiments shown below.

[0014] Also, hereinafter, it is assumed that the server device according to the embodiment is the center device 100 (see FIG. 1) included in the information providing system 1 (see FIG. 1) according to the embodiment. The program according to the embodiment and the information providing method according to the embodiment are assumed to be executed by the controller 103 (see FIG. 7) of this center device 100.

[0015] Also, it is assumed that the in-vehicle device according to the embodiment is the in-vehicle device 10 (see FIG. 1) included in the information providing system 1. The information processing in the in-vehicle device 10 regarding the information providing system 1 is assumed to be executed by the controller 13 (see FIG. 6) of this in-vehicle device 10. Also, hereinafter, it is assumed that the vehicle is a vehicle for passenger transportation.

[0016] Also, hereinafter, when it is necessary to distinguish multiple identical elements, this element may be numbered in the form of "-k" (k is a natural number of 1 or more) after the reference numeral indicating this element. When there is no particular need to distinguish, this numbering is not performed.

[0017] Also, expressions such as "predetermined", "specific", and "constant" in the following description may be read as "pre-determined".

[0018] First, an overview of the information providing method according to the embodiment will be described using FIG. 1. FIG. 1 is a schematic explanatory diagram of the information providing method according to the embodiment. As shown in FIG. 1, the information providing system 1 includes in-vehicle devices 10-1, 10-2,... 10-m (m is a natural number of 3 or more), a center device 100, and user terminals 200-1, 200-2,... 200-n (n is a natural number of 3 or more).

[0019] The on-board device 10 is a computer installed in each vehicle. Figure 1 shows an example where on-board device 10-1 is installed in, for example, a route bus B1, on-board device 10-2 is installed in, for example, an on-demand bus D2, and on-board device 10-m is installed in, for example, a taxi Tm. Multiple on-board devices 10 may be installed in a single vehicle.

[0020] The in-vehicle device 10 according to this embodiment is connected to at least one camera 5a (see Figure 6). The camera 5a is configured to capture at least images of the inside of the vehicle. Multiple cameras 5a may be provided in a single vehicle. The camera 5a may also be configured to capture images of the outside of the vehicle.

[0021] The controller 13 of the in-vehicle device 10 records a certain period of operational record data, including video footage captured by the camera 5a, in a ring buffer memory, for example, so that it can be overwritten during vehicle operation. The operational record data includes not only video footage but also date and time information, location information (latitude and longitude), acceleration, etc.

[0022] Furthermore, the controller 13 of the in-vehicle device 10 is configured to perform image recognition processing using an AI model for image recognition on, for example, video footage captured by camera 5a, in parallel with recording operation record data. The AI ​​model is, for example, a Deep Neural Network (DNN) model trained using a machine learning algorithm. This AI model is pre-trained to be able to estimate, for example, the attributes, state, skeletal structure, facial expressions, etc., of any passenger appearing in the in-vehicle video footage.

[0023] Furthermore, the controller 13 of the in-vehicle device 10 is designed to detect the occurrence of an incident inside the vehicle based on the processing results of the image recognition process described above. An incident, as used here, primarily refers to an event that may result in an accident involving a passenger. However, in cases where an accident such as a fall occurs, the events immediately preceding the accident may also be included as an incident.

[0024] In detecting the occurrence of such incidents, the aforementioned camera 5a functions as a surveillance camera for estimating the skeletal structure and facial expressions of any passenger. The controller 13 of the in-vehicle device 10 detects the occurrence of an incident from changes in the skeletal structure and facial expressions of any passenger monitored by the camera 5a. For example, the controller 13 detects an incident when a passenger stumbles based on changes in the skeletal structure of the passenger monitored by the camera 5a. Alternatively, the controller 13 may detect an incident from a change in the facial expression of a passenger that the camera 5a is monitoring, indicating discomfort. Examples of discomfort include expressions of surprise when a passenger is subjected to sudden acceleration or deceleration, or large swaying on curves, as well as expressions of grimacing, anger, or distress. In this way, incidents such as a passenger stumbles or a passenger experiencing discomfort can be detected based on changes in the passenger's skeletal structure and facial expressions.

[0025] Here, as shown in Figure 1, the controller 13 of the in-vehicle device 10-1 detects that passenger U1 of route bus B1 has lost their balance and stumbled, which is considered an incident (step S1).

[0026] Then, the controller 13 of the in-vehicle device 10-1 transmits incident information, which is information about the detected incident, to the center device 100 (step S2). If an incident is detected inside the on-demand bus D2, the controller 13 of the in-vehicle device 10-2 transmits incident information to the center device 100. Also, if an incident is detected inside the taxi Tm, the controller 13 of the in-vehicle device 10-m transmits incident information to the center device 100.

[0027] As shown in Figure 1, incident information includes vehicle data, passenger data, and surrounding environment data. Vehicle data is, for example, operational record data for a certain period before and after the incident detection. Passenger data is data about the passengers in the vehicle. Passenger data includes attribute information about the passengers' attributes, status information about their condition, posture information about their posture, and facial expression information about their facial expressions, at least at the time the incident was detected. In addition to passengers involved in the incident, passenger data also includes data about passengers who were not involved in the incident.

[0028] Ambient conditions data includes data on the surrounding conditions, such as weather and temperature, at least at the time the incident was detected.

[0029] Furthermore, the controller 13 does not need to transmit the video captured by camera 5a to the center device 100, taking into consideration the communication load and other factors. Instead, the controller 13 estimates and quantifies each element included in the passenger data based on the processing results of the image recognition processing on the aforementioned video, and transmits it to the center device 100. Specific examples of each element of the passenger data will be explained later using Figure 8.

[0030] In this way, by performing video-based analysis and quantification using edge computing on the in-vehicle device 10, the amount of data transmitted can be reduced, lowering the communication load and reducing the processing load on the central device 100. Furthermore, by not transmitting video, privacy can be protected. However, the video captured by camera 5a is not necessarily excluded from transmission; it may be included in the vehicle data of the incident information and transmitted to the central device 100. For example, if the incident escalates to an accident such as a passenger falling, the controller 13 may transmit the video to the central device 100 for detailed situation analysis.

[0031] Then, the controller 103 of the central device 100 receives and stores incident information transmitted from each in-vehicle device 10 (step S3). The controller 103 of the central device 100 also analyzes combinations of incident information elements that are likely to cause incidents based on the stored incident information (step S4).

[0032] As a first method in step S4, the controller 103 of the center device 100 calculates, for example, locations where incidents occur frequently based on statistical data. The controller 103 also extracts combinations of incident information elements related to the incidents that occur frequently. Furthermore, the controller 103 calculates predicted locations where situations similar to the extracted combinations of incident information elements are likely to occur, by referring to, for example, map information. This makes it possible to calculate both frequent incident locations and predicted locations using frequent incident locations.

[0033] As a second method, the controller 103 extracts all possible combinations of each element of the incident information. The controller 103 also calculates the probability of an incident occurring for each of the extracted combinations, categorized by the type of incident (e.g., staggering, drunkenness). The controller 103 then extracts combinations of each element of the incident information for which the probability of occurrence is above a threshold, for each type of incident. The controller 103 also calculates predicted locations where situations similar to the extracted combinations of each element of the incident information are likely to occur, using, for example, map information. This makes it possible to predict situations where incidents are likely to occur for each type of incident, based on a kind of brute-force approach to all possible combinations of each element of the incident information.

[0034] The first and second methods described above are merely examples and do not limit the analysis method used in step S4. Furthermore, the controller 13 databases the analysis results from step S4.

[0035] Then, the controller 103 of the center device 100 generates incident sharing information (step S5) that indicates situations where incidents are likely to occur, based on the analysis results in step S4. Incident sharing information is information for sharing situations where incidents are likely to occur among multiple vehicles based on combinations of each element of incident information. Incident sharing information includes the aforementioned high-incidence locations and predicted locations. Output information output based on the incident sharing information is output as visual information, for example, in which the aforementioned high-incidence locations and predicted locations are mapped onto map information. Examples of such output will be described later using Figures 9 to 11.

[0036] Then, the controller 103 of the central device 100 transmits and provides the generated incident sharing information to each in-vehicle device 10 (step S6). The controller 13 of each in-vehicle device 10 automatically notifies the in-vehicle of the incident sharing information according to the type of vehicle, current location, set route in the car navigation function, etc. Alternatively, the controller 13 notifies the driver of the incident sharing information, and the driver notifies the in-vehicle through an in-vehicle announcement or the like.

[0037] In addition, the controller 103 can also transmit incident sharing information to the user terminal 200. The user terminal 200 is a terminal device used by operators of the center device 100, or by general users such as passengers. These operators and general users can view the incident sharing information via the user terminal 200 and use it as learning material.

[0038] As described above, in the information provision method according to this embodiment, the controller 103 of the central device 100 receives incident information regarding incidents that occur inside the vehicle from multiple in-vehicle devices 10. The controller 103 also analyzes combinations of incident information elements that are likely to cause incidents based on the received incident information. The controller 103 also generates shared information among the multiple in-vehicle devices 10 that indicates situations where incidents are likely to occur, based on the analyzed combinations of elements. The controller 103 also transmits the shared information to the multiple in-vehicle devices 10.

[0039] Therefore, according to the information provision method of this embodiment, incident information received from multiple in-vehicle devices 10 is analyzed integrally, and information indicating situations where incidents are likely to occur is shared and provided based on combinations of elements of incident information that are likely to occur. This makes it possible to share and utilize information regarding in-vehicle incidents among multiple vehicles.

[0040] Below, we will describe in more detail an example of the configuration of the information provision system 1 to which the information provision method according to the above embodiment is applied.

[0041] Figure 4 shows an example of the configuration of the information provision system 1 according to the embodiment. As shown in Figure 4, the information provision system 1 includes in-vehicle devices 10-1, 10-2, ..., 10-m, a center device 100, and user terminals 200-1, 200-2, ..., 200-n.

[0042] Each in-vehicle device 10, the central device 100, and each user terminal 200 are connected to each other via a network N1, such as the Internet, a mobile phone network, or a C-V2X (Cellular Vehicle to Everything) communication network.

[0043] As described above, the in-vehicle device 10 performs image recognition processing on the video captured by camera 5a in parallel with recording the operation record data, and detects the occurrence of an incident inside the vehicle based on the processing results.

[0044] Furthermore, when the in-vehicle device 10 detects the occurrence of an incident, it generates incident information including vehicle data, which is at least a portion of the operation record data for a certain period before and after the detection time, passenger data and surrounding condition data at the time the incident was detected. During this generation, as described above, the in-vehicle device 10 estimates and quantifies each element included in the passenger data based on the processing results of image recognition processing on the video captured by the camera 5a. The in-vehicle device 10 also transmits the generated incident information to the central device 100.

[0045] Here, we will explain specific examples of incident information transmitted to the center device 100. Figure 3 shows an example of vehicle data. Figure 4 shows an example of surrounding situation data. Figure 5 shows an example of passenger data.

[0046] As shown in Figure 3, the vehicle data includes, for example, date and time information, location information (latitude and longitude), vehicle speed, and acceleration. As described above, the vehicle data is at least a portion of the operational record data for a certain period before and after the time the incident was detected. As described above, the vehicle data may also include video footage captured by camera 5a. In addition, the vehicle data may include other elements besides those shown in Figure 3, such as driving information related to the accelerator, brakes, steering, etc. Furthermore, the vehicle data may include, for example, a driver identification number that indicates a difference in drivers.

[0047] Furthermore, as shown in Figure 4, the surrounding conditions data includes, for example, weather, temperature, and road conditions (curves, intersections, railway crossings, etc.). The surrounding conditions data may also include external video footage of the area around the vehicle captured by camera 5a. As mentioned above, if an incident escalates to an accident such as a passenger falling, the external video footage will be useful for detailed situation analysis at the center device 100. The vehicle data shown in Figure 3 and the surrounding conditions data shown in Figure 4 are examples of "driving information" related to the vehicle's movement.

[0048] Furthermore, as shown in Figure 5, passenger data includes, for example, attribute information, state information, posture information, and facial expression information. The controller 13 of the in-vehicle device 10 analyzes each element of this information and quantifies it. This quantification is performed, for example, by binarization to 0 or 1, score conversion to 0-100, weight conversion to 0-1, etc.

[0049] Attribute information refers to information about the passenger's attributes, including, for example, estimated age group, estimated gender, and body type. Each of these attribute information elements is analyzed and quantified based on image recognition processing using, for example, an AI model, the attribute / state estimation model 12b (see Figure 6).

[0050] Status information refers to information about the passenger's state, and each element includes, for example, standing / sitting, location within the vehicle, awake / sleeping, and whether or not the passenger is carrying belongings. Standing / sitting refers to whether the passenger is standing or sitting. Whether or not the passenger is carrying belongings refers to whether or not the passenger is carrying an object such as luggage, a cane, or a smartphone. Whether or not a passenger is carrying belongings may be considered as having belongings if they are holding a child.

[0051] Furthermore, the state information includes, for example, the orientation of the face and whether or not a support device is being used. The orientation of the face may be considered as the direction of gaze. The orientation of the face, along with the presence or absence of possessions, can be used to estimate, for example, whether or not a smartphone is being used. Support devices include handrails, straps, seat belts, etc. Using a support device refers to the case where the passenger is holding or wearing one of these support devices. Each of these elements of state information is analyzed and quantified based on image recognition processing using, for example, the attribute / state estimation model 12b described above. By including attribute information and state information as elements of incident information, the attributes and state of passengers at the time of an incident can be used as parameters for estimating situations in which an incident is likely to occur.

[0052] Posture information refers to information about the passenger's posture, and includes, for example, the degree of swaying of the upper body, the degree of swaying of the lower body, the degree of swaying of the whole body, and whether or not a fall has occurred. Each of these elements of posture information is analyzed and quantified based on image recognition processing using, for example, an AI model, the skeletal estimation model 12c (see Figure 6).

[0053] Furthermore, the posture information may be based on at least one of the passengers in the vehicle. Incident detection using skeletal estimation is considered easier to perform by targeting standing passengers rather than seated passengers, for example. Therefore, if there are both seated and standing passengers, posture information may be obtained by performing skeletal estimation on the standing passengers, who are monitored by camera 5a, and analyzing only the changes in the skeletons of those monitored passengers.

[0054] Furthermore, posture information may be obtained, for example, if the vehicle is crowded with standing passengers, by performing skeletal estimation on at least one of the standing passengers who is in a position that can be monitored by camera 5a, and obtaining the information from the changes in the skeleton of only that monitored passenger. This is because, for example, in a route bus B1, if it is crowded with standing passengers, the movement of the skeleton due to sudden acceleration or deceleration or curves can be assumed to be similar for all standing passengers.

[0055] Facial expression information refers to information about passengers' facial expressions, including, for example, their level of discomfort. Discomfort refers to the degree of surprise, anger, or pain. Each element of facial expression information is analyzed and quantified based on image recognition processing using, for example, the AI ​​model facial expression estimation model 12d (see Figure 6). By including posture information and facial expression information as elements of incident information, for example, the degree of a passenger's staggering or discomfort can be used as parameters to estimate situations in which incidents are likely to occur.

[0056] Furthermore, it is preferable to target passengers without companions, for example, with facial expression information. Incident detection through facial expression estimation is considered easier when targeting passengers without companions, rather than passengers with companions, whose emotions are more likely to be expressed through facial expressions due to conversations with companions, for example. Therefore, when there are both passengers with and without companions, posture information may be obtained by monitoring passengers without companions using camera 5a and performing facial expression estimation only on the changes in the facial expressions of those monitored. Please note that each element of the information shown in Figures 3 to 5 is merely an example.

[0057] Returning to the explanation of Figure 2, the in-vehicle device 10 receives incident sharing information generated in the center device 100 based on the incident information transmitted to the center device 100. The in-vehicle device 10 also converts the received incident sharing information into a format suitable for notification within the vehicle and notifies the vehicle accordingly.

[0058] Furthermore, the in-vehicle device 10 may implement real-time notifications, providing information about the incident in real time as soon as it detects the occurrence of the incident. The in-vehicle device 10 may also notify the driver to make an in-vehicle announcement, such as "Please hold on to the handrail or strap" or "Please fasten your seatbelt," if it detects an incident such as a passenger stumbling.

[0059] The central device 100 is implemented, for example, as a private cloud. The central device 100 is managed, for example, by a business operator that operates the data center of the information provision system 1. The central device 100 collects and stores incident information transmitted from each in-vehicle device 10.

[0060] Furthermore, the center device 100 analyzes combinations of incident information elements that are likely to cause incidents, based on the accumulated incident information. The center device 100 also generates incident sharing information that indicates situations where incidents are likely to occur, based on the analysis results. The center device 100 also transmits the generated incident sharing information to each in-vehicle device 10.

[0061] As described above, the user terminal 200 is a terminal device used by operators and general users, and can be implemented as a PC (Personal Computer), smartphone, etc. The user terminal 200 may also be implemented as a tablet device, wearable device, etc. If the user terminal 200 is a smartphone used by general users, it is desirable that only those who have been authenticated and registered through dedicated application software provided by, for example, the information provision system 1, bus companies, taxi companies, etc., can use it. The central device 100 may be implemented as a public cloud.

[0062] Next, an example of the configuration of the in-vehicle device 10 will be described. Figure 6 is a diagram showing an example of the configuration of the in-vehicle device 10 according to the embodiment. As shown in Figure 6, the in-vehicle device 10 has a communication unit 11, a storage unit 12, and a controller 13.

[0063] The communication unit 11 is implemented by a network adapter or the like. The communication unit 11 is wirelessly connected to the network N1 and transmits and receives information to and from the center device 100 via the network N1.

[0064] The memory unit 12 is implemented by a storage device such as ROM (Read Only Memory), RAM (Random Access Memory), flash memory, or HDD (Hard Disk Drive). In the example shown in Figure 6, the memory unit 12 stores operation record data 12a, attribute / state estimation model 12b, skeletal estimation model 12c, facial expression estimation model 12d, map information DB (Database) 12e, and incident sharing information DB 12f.

[0065] The operation record data 12a corresponds to the "operation record data" described above, so its explanation is omitted here. The attribute / state estimation model 12b is one of the AI ​​models described above. After the attribute / state estimation model 12b is loaded as an AI model into the controller 13, it functions as a passenger attribute / state estimation AI when each frame of the in-vehicle video is input to the controller 13.

[0066] The attribute / state estimation model 12b is designed to estimate each element of attribute information and state information in the passenger data described above, based on the in-vehicle video footage captured by camera 5a.

[0067] The skeletal estimation model 12c is also one of the AI ​​models mentioned above. After being loaded as an AI model into the controller 13, the skeletal estimation model 12c functions as a passenger skeletal estimation AI when each frame of the in-vehicle video is input to the controller 13.

[0068] The skeletal estimation model 12c is configured to detect the skeleton of the passenger being monitored based on the in-vehicle video footage captured by camera 5a. The skeletal estimation model 12c is pre-trained, for example, using a skeletal detection algorithm in deep learning. Furthermore, the skeletal estimation model 12c is configured to estimate each element of posture information in the passenger data described above.

[0069] The facial expression estimation model 12d is also one of the AI ​​models mentioned above. After being loaded as an AI model into the controller 13, the facial expression estimation model 12d functions as a passenger facial expression estimation AI when each frame of the in-vehicle video is input to the controller 13.

[0070] The facial expression estimation model 12d is designed to detect the facial expressions and emotions of passengers being monitored based on the in-vehicle video footage captured by camera 5a. The facial expression estimation model 12d is pre-trained, for example, using an emotion estimation algorithm in deep learning. Furthermore, the facial expression estimation model 12d is designed to estimate each element of facial information in the passenger data described above.

[0071] The map information DB12e is a database of map information, such as map information output by the controller 13 as visual information, and map information used by the controller 13 to estimate topographic information based on the current location. The incident sharing information DB12f is a database of incident sharing information received from the center device 100.

[0072] The controller 13 corresponds to a so-called processor. The controller 13 is implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), etc. The controller 13 executes a program according to an embodiment not shown, stored in the memory unit 12, using RAM as the working area. The controller 13 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0073] The controller 13 is connected to the HMI (Human Machine Interface) unit 3 and the in-vehicle sensor unit 5.

[0074] The HMI unit 3 is a component that provides interface components for input and output to the driver or other operator operating the in-vehicle device 10. The HMI unit 3 includes an input interface that receives input operations from the driver or other operator. The input interface is implemented, for example, by a touch panel. Alternatively, the input interface may be implemented by a microphone or the like. Furthermore, the input interface may be implemented by software components.

[0075] Furthermore, the HMI unit 3 includes an output interface for presenting visual and audio information to the driver, passengers, etc. The output interface is implemented by, for example, a display, speaker, earphones, etc. Multiple displays and speakers may be provided. The HMI unit 3 may also provide the input interface and output interface as an integrated unit, for example, by a touch panel display. In this embodiment, the HMI unit 3 may be implemented as a driving tablet used by the driver.

[0076] The on-board sensor unit 5 is a group of various sensors mounted on the vehicle. The on-board sensor unit 5 is connected to the on-board device 10 via an on-board network such as CAN (Controller Area Network).

[0077] The in-vehicle sensor unit 5 includes a camera 5a, a GPS (Global Positioning System) sensor 5b, a vehicle speed sensor 5c, and a G sensor 5d. The camera 5a has already been described, so its description is omitted here.

[0078] The GPS sensor 5b determines the GPS position (latitude and longitude) of the vehicle. The vehicle speed sensor 5c measures the vehicle's speed. The G sensor 5d measures the acceleration (G) applied to the in-vehicle device 10. The in-vehicle sensor unit 5 may also include various other sensors besides the camera 5a, GPS sensor 5b, vehicle speed sensor 5c, and G sensor 5d, such as an accelerator sensor and a brake sensor.

[0079] Based on the various sensor data from the on-board sensor unit 5 and the various information stored in the storage unit 12, the controller 13 executes information processing by the on-board device 10 in the processing sequence shown in Figure 8. An explanation using Figure 8 will be given later.

[0080] Next, an example of the configuration of the center device 100 will be described. Figure 7 is a diagram showing an example of the configuration of the center device 100 according to the embodiment. As shown in Figure 7, the center device 100 has a communication unit 101, a storage unit 102, and a controller 103. The center device 100 is also connected to an HMI unit 150.

[0081] The HMI unit 150 is a component that provides interface components for input and output to operators, etc., who operate the center device 100. The HMI unit 150 includes an input interface that receives input operations from operators, etc. The input interface is implemented, for example, by a touch panel. Alternatively, the input interface may be implemented by a keyboard, mouse, pen tablet, microphone, etc. Furthermore, the input interface may be implemented by software components.

[0082] Furthermore, the HMI unit 150 includes an output interface for presenting visual and audio information to the operator or other user. The output interface is implemented, for example, by a display or speaker. The HMI unit 150 may also provide the input interface and output interface as an integrated unit, for example, by a touch panel display.

[0083] The communication unit 101 is implemented by a network adapter or the like, similar to the communication unit 11 described above. The communication unit 101 is connected to the network N1 by wire or wireless connection and transmits and receives information between each in-vehicle device 10 and each user terminal 200 via the network N1.

[0084] The storage unit 102 is implemented by a storage device such as ROM, RAM, flash memory, or HDD. In the example shown in Figure 7, the storage unit 102 stores the collected data DB 102a, the analysis model 102b, the analysis results DB 102c, and the map information DB 102d.

[0085] The collected data DB102a is a database that stores incident information collected from each in-vehicle device 10. The analysis model 102b is a mathematical model that includes various calculation formulas used in step S4, etc., as described above. Alternatively, the analysis model 102b may be an AI model that has been trained to output predicted locations of incidents based on combinations of elements of incident information that are likely to occur, as analyzed in step S4.

[0086] The analysis results DB102c is a database in which the analysis results from step S4 described above are stored. The map information DB102d is the master information that forms the basis of the map information DB12e for each in-vehicle device 10. The map information DB102d is used by the controller 103, for example, when incident sharing information is generated in step S5 described above.

[0087] Although not shown in the diagram, the memory unit 102 may store AI models corresponding to each AI model for image recognition stored in the in-vehicle device 10 described above. When generating and updating incident sharing information, the controller 103 may include information based on the image recognition results recognized by the center device 100 in the incident sharing information by performing image recognition processing using such AI models.

[0088] Controller 103, like controller 13 described above, corresponds to a so-called processor. Controller 103 is implemented by a CPU, MPU, GPU, etc. Controller 103 executes a program according to an embodiment not shown, stored in the memory unit 102, using RAM as a working area. Controller 103 can also be implemented by an integrated circuit such as an ASIC or FPGA.

[0089] The controller 103 executes the information processing performed by the center device 100 in the processing sequence shown in Figure 8. Next, this processing sequence will be described. Figure 8 is a diagram showing the processing sequence executed by the information provision system 1 according to this embodiment.

[0090] As shown in Figure 8, the controller 13 of the in-vehicle device 10 acquires sensor data from each of the in-vehicle sensor units 5 while the vehicle is in operation (step S101).

[0091] Then, the controller 13 determines whether or not an incident has been detected inside the vehicle based on image recognition processing of the video captured by camera 5a included in the acquired sensor data (step S102). If no incident has been detected (step S102, No), the controller 13 repeats the process from step S101.

[0092] If an incident is detected (Step S102, Yes), the controller 13 analyzes and quantifies each element of the passenger data related to the passenger (Step S103). Then, the controller 13 transmits the incident information, including the quantified passenger data, to the central device 100 (Step S104).

[0093] Furthermore, the controller 13 provides the aforementioned real-time notification for the detected incident (step S105). Then, the controller 13 repeats the process from step S101.

[0094] Meanwhile, the controller 103 of the central device 100 receives and stores incident information transmitted from each in-vehicle device 10 (step S106).

[0095] Then, at any given time, the controller 103 analyzes the combinations of incident information elements that are likely to cause incidents, based on the accumulated incident information (step S107). The analysis results are stored in a database called analysis results DB102c.

[0096] Meanwhile, the controller 13 of the in-vehicle device 10 sends a request to the central device 100 to acquire incident sharing information at any time (step S108). The controller 103 of the central device 100 then generates incident sharing information based on the analysis results of step S107 in response to the acquisition request (step S109). The controller 103 then sends the generated incident sharing information to each in-vehicle device 10 (step S110). The controller 13 of the in-vehicle device 10 receives the incident sharing information sent from the central device 100 and stores it in the incident sharing information DB 12f (step S111).

[0097] The acquisition request in step S108 is transmitted to the center device 100, including the vehicle's current location, at any arbitrary timing, such as when the vehicle equipped with the in-vehicle device 10 is started up or at predetermined intervals while driving. In such cases, the controller 103 of the center device 100 generates incident sharing information corresponding to a predetermined range based on the current location included in the acquisition request, and transmits it to the in-vehicle device 10.

[0098] Furthermore, the acquisition request in step S108 is transmitted to the center device 100, including the set route, for example, when setting a route in the car navigation function of a vehicle equipped with the in-vehicle device 10. In such cases, the controller 103 of the center device 100 generates incident sharing information corresponding to a predetermined range along the route included in the acquisition request and transmits it to the in-vehicle device 10.

[0099] Then, the controller 13 determines, based on the sensor data from the in-vehicle sensor unit 5, whether or not a situation has arisen that is likely to cause an incident, as indicated by the incident sharing information (step S112).

[0100] If the relevant situation has not occurred (Step S112, No), the controller 13 repeats Step S112. If the relevant situation has occurred (Step S112, Yes), the controller 13 issues a notification in accordance with the incident sharing information (Step S113).

[0101] The occurrence of such a situation is, for example, approaching a location where incidents frequently occur or a predicted location, as included in the incident sharing information. When the controller 13 detects an approach to such a location based on the vehicle's current position, it notifies the driver of the approach via the operation tablet or the like included in the HMI unit 3.

[0102] The controller 13 also instructs the driver to make announcements such as, "Please hold on to the handrails or straps," or "Please fasten your seatbelts." The driver then makes the announcements in response to these instructions to alert the passengers.

[0103] Furthermore, the controller 13 may automatically alert passengers via the speaker included in the HMI unit 3 without giving instructions to the driver. Passengers who receive the alert can take precautions in advance, such as holding onto straps or handrails, to prepare for the impending lateral movement, thus reducing the chances of incidents or accidents occurring.

[0104] Here, Figure 9 shows an example of output information based on incident sharing information on route bus B1. Figure 10 shows an example of output information based on incident sharing information on on-demand bus D2. Figure 11 shows an example of output information based on incident sharing information on user terminal 200.

[0105] Figures 9 to 11 show examples of output where a heatmap indicating the degree of staggering, one of the incident types (passenger staggering), is superimposed on the map information. In Figures 9 to 11, the heatmap is partially superimposed on region R1 for the sake of readability, but the heatmap can be superimposed over the entire area of ​​the displayed map information.

[0106] As shown in Figure 9, in the case of route bus B1, this output information is output to the operation tablet, etc., shown as HMI unit 3. Since the current position P1 of route bus B1 is approaching a region R1 on the travel route RT1 where the degree of swaying is high, Figure 9 shows an example in which a guidance display G1 instructs an in-vehicle announcement. This allows the driver of route bus B1 to visually grasp immediately that it is approaching region R1 where passengers are likely to sway due to the curve, and to provide appropriate warnings through an in-vehicle announcement corresponding to the guidance display G1.

[0107] Furthermore, as shown in Figure 10, in the case of the on-demand bus D2, the output information is displayed on the car navigation screen, etc., as shown in HMI unit 3. Unlike the route bus B1, the on-demand bus D2 allows the driver to arbitrarily select their route. Therefore, Figure 10 shows an example in which the on-demand bus D2's route RT2, which is different from the route RT1 of the route bus B1, is automatically set while avoiding the high-staggering region R1. In this way, based on incident sharing information, it is possible to output information according to the vehicle type while coordinating with the vehicle's car navigation system.

[0108] Furthermore, as shown in Figure 11, the output information for operators and general users mentioned above is output to the user terminal 200. As shown in Figure 11, the user terminal 200 can output heatmaps according to the map type, time period, etc., that the operator or general user has specified as a display option. It is also possible to specify the map resolution to arbitrarily change the size of the cells that make up the heatmap. Although not shown in Figure 11, operators and general users can also arbitrarily specify, as a display option, attributes and statuses corresponding to each element of passenger data in incident information, in addition to map type and time period, and output them as heatmaps.

[0109] As described above, the center device 100 according to the embodiment is a server device provided to communicate with a plurality of in-vehicle devices 10 each installed in a plurality of vehicles carrying passengers, and includes a controller 103. The controller 103 receives incident information from the plurality of in-vehicle devices 10 regarding incidents that occur inside the vehicle, analyzes combinations of incident information elements that are likely to occur based on the received incident information, generates incident sharing information among the plurality of in-vehicle devices 10 that indicates situations where incidents are likely to occur (corresponding to an example of "shared information") based on the analyzed combinations of elements, and transmits the incident sharing information to the plurality of in-vehicle devices 10.

[0110] Therefore, according to the center device 100 of this embodiment, incident information received from multiple in-vehicle devices 10 is analyzed in an integrated manner, and information indicating situations where incidents are likely to occur is shared and provided based on combinations of elements of incident information that are likely to occur. This makes it possible to share and utilize information about in-vehicle incidents among multiple vehicles.

[0111] In this embodiment, the center device 100 may have the controller 103 determine that a situation similar to the combination of passenger discomfort level, passenger attribute information, and vehicle driving information estimated from camera images of multiple vehicles at the time of the incident, which is included in the received incident information, is a situation in which an incident is likely to occur. In this case, the center device 100 can predict situations in which an incident is likely to occur based on at least the combination of passenger discomfort level, passenger attribute information, and vehicle driving information, and the predicted situation can be shared and utilized among multiple vehicles.

[0112] Furthermore, in the above-described embodiment, incident sharing information is generated and utilized among multiple vehicles based on the analysis results of combinations of each element of the incident information. However, the analysis result DB102c and incident sharing information can also be utilized in other use cases.

[0113] For example, this data may be used as educational material for instructing, educating, and raising awareness among drivers about safe driving. In such cases, the output examples shown in Figures 9 to 11 can be used, for example, as e-learning materials.

[0114] This allows passenger transport operators to use the accumulated information to instruct, educate, and raise awareness among their employees, such as drivers, about safe driving practices.

[0115] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0116] 1. Information Provision System 5. Vehicle-mounted sensor unit 5a camera 5b GPS sensor 5c Vehicle speed sensor 5d G-sensor 10 Onboard equipment 11 Communications Department 12 Storage section 12a Operation record data 12b Attribute / State Estimation Model 12c Skeletal Estimation Model 12D facial expression estimation model 12e Map Information Database 12f Incident Sharing Information Database 13 Controllers 100 Center device 101 Communications Department 102 Storage section 102a Collected Data Database 102b Analysis Model 102c Analysis result DB 102d Map Information Database 103 Controller B1 Route Bus D2 On-Demand Bus TM Taxi

Claims

1. A server device is provided that can communicate with multiple on-board devices installed in multiple vehicles carrying passengers, A receiving procedure for receiving incident information regarding an incident that occurred inside a vehicle from the aforementioned multiple in-vehicle devices, An analysis procedure for analyzing combinations of elements of the incident information that are likely to cause the incident, based on the incident information received, A generation procedure for generating shared information among the multiple in-vehicle devices that indicates situations where the incident is likely to occur, based on the combination of each element analyzed, A transmission procedure for transmitting the shared information to the multiple in-vehicle devices, A program that executes the command.

2. The plurality of in-vehicle devices detect the occurrence of the incident based on the skeletal movements or facial movements of the passengers estimated by image analysis processing of camera images of the plurality of vehicles. The program according to claim 1.

3. The aforementioned incident information is, The system includes, as elements, driving information relating to the vehicle's movement, attribute information relating to the passenger's attributes, and state information relating to the passenger's condition. The program according to claim 2.

4. The aforementioned incident information further, The elements include the passenger's posture information estimated based on the skeletal structure, and the passenger's degree of discomfort estimated based on their facial expression. The program according to claim 3.

5. The aforementioned analysis procedure is: Based on the aforementioned incident information, statistically calculate the locations where the aforementioned incidents occur frequently. The combination of each element of the incident information corresponding to the frequently occurring incidents is extracted, The system calculates predicted locations where situations similar to the combination of elements of the extracted incident information are likely to occur. The program according to claim 1.

6. The aforementioned analysis procedure is: Extract all combinations of the aforementioned elements of the incident information, The probability of the aforementioned incident occurring for each of the extracted combinations is calculated for each type of incident. For each type of incident, extract the combinations of the elements of the incident information for which the probability of occurrence is above a threshold. The system calculates predicted locations where situations similar to the combination of elements of the extracted incident information are likely to occur. The program according to claim 5.

7. The aforementioned generation procedure is: The shared information is generated, which includes at least the locations with a high incidence rate and the predicted locations. The program according to claim 5 or 6.

8. A server device is provided that can communicate with multiple on-board devices installed in multiple vehicles carrying passengers, A receiving procedure for receiving incident information regarding an incident that occurred inside a vehicle from the aforementioned multiple in-vehicle devices, A determination procedure for determining that a situation similar to a combination of passenger discomfort, passenger attribute information, and vehicle driving information estimated from camera images of the multiple vehicles at the time the incident occurred, as included in the received incident information, is a situation in which the incident is likely to occur. A generation procedure for generating shared information among multiple in-vehicle devices that indicates the conditions under which the determined incident is likely to occur, A transmission procedure for transmitting the shared information to the multiple in-vehicle devices, A program that executes the command.

9. A server device that is capable of communicating with multiple on-board devices installed in multiple vehicles carrying passengers, and which includes a controller, The aforementioned controller, The multiple in-vehicle devices receive incident information regarding incidents that occurred inside the vehicle. Based on the incident information received, the combination of elements of the incident information that are likely to cause the incident is analyzed, Based on the analyzed combination of each of the above elements, shared information among the multiple in-vehicle devices indicating the conditions under which the incident is likely to occur is generated. The shared information is transmitted to the multiple in-vehicle devices. Server device.

10. An in-vehicle device installed in a vehicle carrying passengers, which is configured to communicate with a server device and includes a controller, The aforementioned controller, Based on camera images taken inside the vehicle, the system detects incidents that occur inside the vehicle. The incident information regarding the occurrence of the aforementioned incident is transmitted to the server device. The server device receives shared information among multiple in-vehicle devices, which is generated based on the incident information and indicates situations in which the incident is likely to occur. In-vehicle device.

Images