Vehicle device and information integration method

Abstract

To provide a vehicular device capable of integrating information, which relates to the same object, from different acquisition sources and an information integration method.SOLUTION: A vehicular device 1 comprises: an acquisition unit 10a for acquiring, as internal information, information detected by a vehicle-side sensor 5 or an on-vehicle sensor 9 provided on a self vehicle 2; a comparison unit 10b for comparing the internal information with external information received via a V2X communication unit 14 provided in the self vehicle 2, the external information being information acquired outside of the self vehicle 2; a determination unit 10c for determining whether or not information relating to the same object is included in both the internal information and the external information based on a result of the comparison by the comparison unit 10b; and an integration unit 10d for integrating information acquired from different acquisition sources and relating to the same object based on a result of the determination by the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a vehicle device and an information integration method for integrating information obtained from different sources. [Background technology]

[0002] Conventionally, an information processing system has been proposed that uses image sensors mounted on multiple vehicles to identify identical objects contained in images captured by the multiple vehicles, thereby identifying traffic events (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-125368 A Summary of the Invention [Problem to be solved by the invention]

[0004] Now, by receiving information obtained outside the vehicle, as in the above-mentioned information processing system, it is considered that each vehicle will be able to use information other than the information obtained by the vehicle itself for purposes such as hazard prediction.

[0005] However, in the past, information received from the outside was treated as different from information acquired by the vehicle itself, and therefore information relating to the same subject was treated separately.

[0006] The present disclosure has been made in consideration of the above-mentioned circumstances, and has an object to provide a vehicle device and an information integration method that are capable of integrating information about the same subject that is obtained from different sources. [Means for solving the problem]

[0007] In order to achieve the above object, a vehicle device according to one embodiment of the present disclosure includes an acquisition unit (10a) that acquires information detected by a sensor (5, 9) provided in the vehicle (2) as internal information, a comparison unit (10b) that compares the internal information with external information that is acquired outside the vehicle and is received via a communication unit (14) provided in the vehicle, a determination unit (10c) that determines whether or not information on the same object is included in both the internal information and the external information based on a comparison result by the comparison unit, and a determination unit (10d) that determines whether or not information on the same object is included in both the internal information and the external information based on a determination result by the determination unit. As integrated information An integration unit (10d) for integration; an adjustment unit (10e) that adjusts the amount of information of the integrated information according to a predetermined selection criterion; This allows for the identification of information on the same subject from different sources. Integrate and align It is possible. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a schematic diagram illustrating an example of an electrical configuration of a vehicle device according to a first embodiment; [Diagram 2] FIG. 2 is a diagram showing an example of a surrounding environment of a vehicle; [Diagram 3] FIG. 1 shows the flow of information integration processing executed by the vehicle. [Figure 4] A diagram showing an example of internal information, external information, and integrated information [Diagram 5] A diagram showing an example of setting types [Figure 6] FIG. 1 is a schematic diagram showing an example of a manner in which a danger is notified. [Figure 7] FIG. 2 is a schematic diagram showing an example of a manner in which a danger is notified. [Figure 8] FIG. 11 is a diagram showing a schematic diagram of an aspect of extracting feature points of an object according to a second embodiment; [Figure 9] FIG. 1 is a diagram showing an example of a mode in which a risk is predicted based on feature points; [Figure 10] FIG. 1 is a diagram showing an example of a method for determining identical objects based on feature points; [Figure 11] FIG. 13 is a schematic diagram illustrating an example of a surrounding environment of a vehicle according to a third embodiment; [Figure 12]FIG. 2 is a diagram showing a flow of information integration and adjustment processing executed by the host vehicle. [Figure 13] Diagram showing an example of color tone settings [Figure 14] A diagram showing an example of integrated information with the amount of information adjusted [Figure 15] Here is an example of lane settings: [Figure 16] FIG. 13 is a diagram showing an example of transmission information with an adjusted amount of information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, several embodiments will be described with reference to the drawings. In the first embodiment, the basic configuration and processing flow for integrating information will be described, and in the second and third embodiments, details of modified examples or extended examples will be described. In addition, parts that are substantially common to each embodiment will be described with the same reference numerals.

[0010] (First embodiment) In the first embodiment, a basic configuration and a process flow for integrating information will be described. As shown in Fig. 1, a vehicle device 1 of this embodiment is mounted on a vehicle 2 and is connected to a display device 3, an audio input / output device 4, a vehicle-side sensor 5, and a plurality of ECUs 6. Note that ECU is an abbreviation for Electronic Control Unit. Although the vehicle device 1 is also one of the electronic devices provided in the vehicle 2, just like the ECU 6, the two are distinguished here for the sake of explanation.

[0011] The display device 3 is, for example, a liquid crystal display, and receives image signals and video signals output from the vehicle device 1 to display images and videos. The display device 3 also functions as an input device for inputting various operations to the vehicle device 1 using a touch panel provided in correspondence with mechanical switches and a screen (not shown). In this embodiment, the display device 3 is assumed to be attached to the center console, but a so-called head-up display or a type that projects onto the windshield may also be adopted or combined with these.

[0012] The audio input / output device 4 is composed of a speaker and a microphone, and enables the audio signal output from the vehicular device 1 to be reproduced and audio to be output, and enables the user to input operations for the vehicular device 1 by voice. The audio input / output device 4 can be used in common with or in combination with a device for so-called hands-free calling.

[0013] The vehicle-side sensor 5 is composed of sensors such as a camera 5a, a LiDAR 5b, a millimeter-wave radar 5c, a gyro 5d, and a GNSS 5e. LiDAR is an abbreviation for Light Detection And Ranging, and GNSS is an abbreviation for Global Navigation Satellite System. However, the vehicle device 1 does not necessarily need to include all of these sensors, and it is sufficient if at least one of them is provided. Furthermore, the vehicle device 1 may include one of each sensor (camera 5a, LiDAR 5b, millimeter-wave radar 5c, and gyro 5d), or may include multiple sensors.

[0014] These vehicle-side sensors 5 are well known and therefore will not be described in detail. The camera 5a captures images or videos of the surroundings such as the front, rear, and sides of the vehicle 2, and in this embodiment, color imaging is possible. The camera 5a may be provided in a plurality of units in the vehicle device 1 (vehicle 2) depending on the purpose, such as for the front, rear, and sides. The LiDAR 5b measures scattered light when a laser is irradiated on an object, and detects the distance and direction to the object. The LiDAR 5b may be provided in a plurality of units in the vehicle device 1 (vehicle 2) depending on the purpose, such as for the front, rear, and sides, or may be a rotating type that can detect the entire periphery of the vehicle. The millimeter wave radar 5c irradiates an object with millimeter waves and detects the distance to the object, the positional relationship with the vehicle 2, the relative speed with the object, and the like with high accuracy. The millimeter wave radar 5c may be provided in a plurality of units in the vehicle device 1 (vehicle 2) depending on the purpose, such as for the front, rear, and sides. The millimeter wave radar 5c may be configured to have different frequencies for different installation positions or different detection targets. The GNSS 5e is capable of acquiring the position and current time of the vehicle 2 by receiving signals from artificial satellites, and is used for navigation control and route guidance of the vehicle 2, for example.

[0015] A plurality of ECUs 6 are provided in the vehicle 2 to control vehicle equipment 7 of environmental and control systems, such as an air conditioner, a drive unit, or brakes. The ECUs 6 are communicatively connected to each other and to the vehicle device 1 via an in-vehicle network 8. The ECUs 6 may also be connected to on-vehicle sensors 9 that detect, for example, vehicle speed, acceleration, turning state, accelerator opening, and brake pedal operation state. Hereinafter, the vehicle side sensors 5 and the on-vehicle sensors 9 are also simply referred to as sensors.

[0016] Information detected by the in-vehicle sensor 9 can be acquired by the vehicle device 1 via the in-vehicle network 8. Hereinafter, the information acquired by the ECU 6 side is also referred to as vehicle information for convenience. In addition, the vehicle information can be used by other ECUs 6 via the in-vehicle network 8.

[0017] The vehicular device 1 can acquire information from sensors. Hereinafter, information acquired by the vehicle 2 is referred to as internal information, and internal information here means information acquired directly by the vehicle 2. Therefore, even if the information is related to the outside of the vehicle 2, such as the road conditions ahead, it is internal information if it is detected by the vehicle-side sensor 5 or the in-vehicle sensor 9. Furthermore, information such as the speed and traveling direction of the vehicle 2 is of course included in the internal information.

[0018] The vehicular device 1 also includes a control unit 10, a storage unit 11, an input / output circuit 12, an in-vehicle communication unit 13, and a V2X communication unit 14. The control unit 10 is configured as a computer system including a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), an input / output interface, and the like (not shown), and controls the entire vehicular device 1 by executing a computer program stored in the storage unit 11.

[0019] In addition, the control unit 10 includes an acquisition unit 10a, a comparison unit 10b, a determination unit 10c, an integration unit 10d, an adjustment unit 10e, and a risk prediction unit 10f in relation to this embodiment. In this embodiment, each of these units is configured by software by executing a program. However, each unit may be configured by hardware, or may be configured by both software and hardware.

[0020] Although the details will be described later, the acquisition unit 10a executes a process of acquiring information detected by the vehicle side sensor 5 and the in-vehicle sensor 9 as internal information at a predetermined cycle. The acquired internal information and external information acquired and received outside the host vehicle 2 are temporarily stored in, for example, a RAM. The comparison unit 10b executes a process of comparing the internal information acquired by the host vehicle 2 with the external information acquired outside the host vehicle 2. The determination unit 10c executes a process of determining whether or not the internal information and the external information contain information on the same object. At this time, the determination unit 10c determines whether or not the internal information and the external information are the same object based on at least one or more pieces of information on the position, speed, direction, or shape of the object contained in the internal information and the external information.

[0021] The integration unit 10d executes a process of integrating information on the same object, which is included in the internal information and the external information and is acquired from different sources. At this time, the integration unit 10d integrates at least one of information on moving objects present around the vehicle 2 detected by the camera 5a, LiDAR 5b, or the like, or information on the vehicle 2 detected by the gyro 5d, GNSS 5e, or the in-vehicle sensor 9, as will be described later in detail. In addition, the integration unit 10d integrates the internal information and the external information in at least one of a state in which the information can be used to control the traveling of the vehicle 2 or a state in which the information can be used to notify the user, as will be described later in detail. In addition, when different information is acquired in the internal information and the external information on the same object, the integration unit 10d integrates the information based on the reliability of each piece of information, as will be described later in detail. However, each piece of information to be integrated is not limited to being integrated based only on the reliability, and can be integrated by other methods, such as calculating the average of a parameter such as speed.

[0022] The adjustment unit 10e, which will be described in detail later, selects the integrated information integrated by the integration unit 10d and adjusts the amount of information to be transmitted. The danger prediction unit 10f predicts danger based on at least one of the internal information, the external information, and the integrated information, and notifies the user of the predicted danger. Therefore, the danger prediction unit 10f also functions as a notification unit that notifies the user.

[0023] The storage unit 11 is configured with, for example, a semiconductor storage device such as an eMMC (embedded Multi Media Card) or a flash memory, or an HDD (Hard Disk Drive), and stores a computer program executed by the control unit 10, various data required for processing executed by the control unit 10, and the like. The storage unit 11 can also store information such as settings of the vehicle device 1. The input / output circuit 12 performs input / output of signals between the control unit 10 and peripheral devices such as the display device 3, the audio input / output device 4, or the vehicle-side sensor 5, and is configured with, for example, a signal conversion circuit that converts an electric signal output from the control unit 10 into an electric signal that can be input to the peripheral devices, and conversely, converts an electric signal output from the peripheral devices into an electric signal that can be input to the control unit 10. The in-vehicle communication unit 13 is realized as an interface for communicating with the ECU 6 via the in-vehicle network 8. In this embodiment, CAN communication is assumed as the in-vehicle communication unit 13, but other standards such as Ethernet (registered trademark) can also be adopted. CAN is an abbreviation for Controller Area Network.

[0024] The V2X communication unit 14 is realized as an interface for communicating with an external device present outside the vehicle 2, and in this embodiment, it is assumed that V2X communication is performed. This V2X communication is a generic term for Vehicle to Everything, that is, communication technology for connecting a vehicle and various devices and systems via wireless communication to enable mutual cooperation.

[0025] This V2X communication includes, for example, V2V (Vehicle to Vehicle), which enables communication between vehicles, V2I (Vehicle to Infrastructure), which enables communication between vehicles and infrastructure equipment such as roadside units, V2P (Vehicle to People), which enables communication between vehicles and terminals owned by people, and V2N (Vehicle to Network), which enables communication between vehicles and a network.

[0026] However, the V2X communication unit 14 does not necessarily need to support all of the above communication technologies. For example, in the present embodiment, the V2X communication unit 14 supports V2V and V2I. Of course, the V2X communication unit 14 may support all of the above communication technologies. Specifically, V2X communication is performed using DSRC (Dedicated Short-Range Communications), which is a dedicated narrow-area (short-distance) communication, or a cellular system in which an area is divided into specific zones, a base station is provided in each zone, and communication is performed between devices in the zone.

[0027] In the case of the vehicular device 1, communication can be performed between an external device capable of V2X communication and located in the vicinity of the vehicle 2. In this case, the external device can be, for example, a roadside device 20 installed in the vicinity of a road. In this case, the roadside device 20 includes a roadside sensor 21 that acquires information about the surrounding environment and a roadside communication unit 22 that can transmit the information detected by the roadside sensor 21 to the vehicle 2.

[0028] Another possible external device is another vehicle 23A that is present in the vicinity of the vehicle 2. The other vehicle 23A includes an other vehicle sensor 24 having a configuration similar to that of the vehicle sensor 5 and an other vehicle communication unit 25 that supports V2I and is capable of transmitting information detected by the other vehicle sensor 24 to the roadside unit 20, and can provide information to the vehicle 2 via the roadside unit 20. If the other vehicle communication unit 25 supports V2V, V2X communication can also be performed directly between the vehicle 2 and the other vehicle 23A.

[0029] Furthermore, in reality, it is assumed that there may be other vehicles 23B, pedestrians 26, bicycles 27, etc. that are not compatible with V2X communication around the vehicle 2. However, if the mobile communication terminal 26a carried by the pedestrian 26 or the image capture and recording device 27a attached to the bicycle 27 is capable of communicating with, for example, the V2P or the roadside device 20, then the mobile communication terminal 26a and the image capture and recording device 27a can also be regarded as external devices.

[0030] In this way, by connecting the vehicle 2 and external devices such as the roadside unit 20 and the other vehicle 23A so as to be able to communicate with each other, an information sharing system 30 is constructed that enables information acquired at a plurality of observation points, i.e., information acquired from different sources, to be shared and used by each other. At this time, since the information received by V2X communication is basically considered to be information in the vicinity of the vehicle 2, it is assumed that information on the same subject contained in the internal information acquired by the vehicle 2 is also contained in the external information acquired outside.

[0031] Next, the operation of the above-mentioned configuration will be described. For example, by receiving external information acquired by an external device such as the roadside device 20 or another vehicle 23A, the vehicle 2 can obtain more detailed information and more accurate information than the internal information, and can obtain new information that is not included in the internal information. By using the external information, it is considered possible to perform, for example, risk prediction with higher accuracy.

[0032] Incidentally, since the received external information is basically considered to be information about the vicinity of the vehicle 2, it is assumed that the external information includes information about the same object that has already been acquired as internal information. In that case, if the external information is treated separately as information different from the internal information, even information about the same object needs to be processed separately, which may increase the load of information processing or cause delays in processing.

[0033] Therefore, the vehicular device 1 integrates information about the same object contained in the internal information and the external information obtained from different sources. Hereinafter, the information integrated by the vehicular device 1 is referred to as integrated information, and integration here means aggregating information about the same object from the internal information and the external information so that it is not necessary to repeatedly execute the same processing on the same object. Below, a specific method by which the vehicular device 1 integrates information will be described, but to facilitate understanding, first, a hypothetical situation that may occur while the host vehicle 2 is traveling as shown in FIG. 2 will be assumed, and the flow of processing will be described according to that situation.

[0034] As shown in FIG. 2, when the north is 0°, the east is 90°, the south is 180°, and the west is 270°, the host vehicle 2 is traveling north on a two-lane road. The black arrows superimposed on each vehicle indicate the direction of travel of each vehicle. Also, an XY coordinate system is set with a predetermined reference position as the origin, and the right direction with respect to the traveling direction of the host vehicle 2, i.e., eastward, is the X direction, and the opposite direction to the traveling direction of the host vehicle 2, i.e., southward, is the Y direction. The XY coordinate system is set for convenience in order to make the explanation easier to understand, but the XY coordinate system can be set by converting the latitude and longitude acquired by GNSS5e into the distance from the reference position.

[0035] As the surrounding environment of the vehicle 2, an intersection exists in front of the vehicle 2, a roadside unit 20 is installed near the intersection, another vehicle 23A is traveling from the intersection to the right, i.e., toward the east, and another vehicle 23B is traveling from the left, i.e., toward the intersection, i.e., toward the east. In addition, a building 28 exists near the intersection on the left side of the vehicle 2, and the building 28 makes the range (H1) indicated by the dashed hatching an area that is impossible or difficult to detect by the vehicle-side sensor 5. In addition, a road sign exists near the roadside unit 20, and the road sign makes the range (H2) indicated by the dashed hatching difficult to detect by the roadside sensor 21, and a fallen object 29 exists in the range (H2).

[0036] That is, Fig. 2 shows a schematic situation in which there are other vehicle 23A, which is an object detectable by both the vehicle 2 and an external device such as the roadside unit 20, other vehicle 23B, which is an object that cannot be detected by the vehicle 2 but can be detected by the roadside unit 20, and a fallen object 29, which is an object that can be detected by the vehicle 2 but cannot be detected by the roadside unit 20. In other words, Fig. 2 shows a schematic situation in which information about the other vehicle 23A is included in both the internal information and the external information, information about the other vehicle 23B is included in the external information but not in the internal information, and information about the fallen object 29 is included in the internal information but not in the external information. However, the situation shown in Fig. 2 is just an example.

[0037] In order to integrate the internal information and the external information, the vehicular device 1 executes an information integration process shown in Fig. 3. Although this information integration process is executed by the above-mentioned comparison unit 10b, determination unit 10c, integration unit 10d, adjustment unit 10e, and the like, the following description will be focused on the vehicular device 1 for the sake of simplicity.

[0038] When the vehicular device 1 starts processing after power-on, it acquires internal information (S1). At this time, as shown in FIG. 4, the vehicular device 1 acquires information on the X coordinate, Y coordinate, speed, direction, type, and reliability of three objects N1, N2, and N3 as internal information. Note that a method for acquiring the internal information will be described in a second embodiment described later. The internal information is set mainly to reduce the load of V2X communication, since there is a risk of insufficient bandwidth for V2X communication or high load of processing the received information when transmitting and receiving image data acquired by the vehicle 2. Also, the types and numbers of information included in the internal information shown in FIG. 4 are merely examples.

[0039] The X coordinate and the Y coordinate indicate a position in the above-mentioned XY coordinate system, the speed indicates the moving speed of the object, and the direction indicates the moving direction of the object. The type indicates the type of the object, and is set as an identification number corresponding to the type of the object. In the case of this embodiment, the types are set as follows: 0 is a moving object, 1 is a motorcycle, 2 is a passenger car, 3 is a truck, 4 is a bus, 5 is a trailer, 6 is an emergency vehicle, 7 is a person, 8 is a bicycle 27, 100 is a stationary object, 101 is a fallen object 29 in the same lane as the host vehicle 2, and 102 is a fallen object 29 in another lane.

[0040] Therefore, if the type is known, it becomes possible to identify the shape of the object to some extent. In addition, by indicating the type of object by the type, it becomes possible to notify the outside of the information of the object and to obtain the information of the object from the outside without transmitting and receiving information with a relatively large capacity such as image data. It is assumed that the type is set in a common manner in the information sharing system 30. In addition, it is assumed that 0 is set as a moving object whose type cannot be identified, and 100 is set as a stationary object whose type cannot be identified. However, the types shown in FIG. 5 are only an example, and the types can be set based on laws and regulations such as the Road Traffic Act and the Vehicle Act, or common standards and specifications for using V2X communication.

[0041] The reliability indicates the accuracy of the information. As an example, it is considered that the speed of a moving object can be obtained by performing image processing on an image captured by the camera 5a. However, generally speaking, it is considered that the speed measured by, for example, the millimeter wave radar 5c is more accurate than the speed obtained by image processing.

[0042] Therefore, the reliability of the speed measured by the millimeter wave radar 5c is set relatively higher than that of the speed obtained by image processing. Also, even if it is the same millimeter wave radar 5c, it is considered that the accuracy is higher when the resolution is higher. Therefore, the reliability is set as a relative value taking into consideration the detection capabilities of the vehicle side sensor 5 and the road side sensor 21 and the measurement conditions such as the positions and orientations at which these sensors are installed.

[0043] 2, the vehicular device 1 acquires information on the other vehicle 23A and the fallen object 29 that are present within the detection range. As described above, the vehicular device 1 also acquires vehicle information on the host vehicle 2 as internal information, and therefore acquires information on three objects, N1 corresponding to the host vehicle 2, N2 corresponding to the other vehicle 23A, and N3 corresponding to the fallen object 29, as internal information.

[0044] Then, as shown in Fig. 3, the vehicular device 1 transmits the internal information to the outside as a message of V2X communication (S2) and receives the external information as a message of V2X communication (S3). At this time, the vehicular device 1 encodes the internal information, for example, in a Collective Perception Message (CPM). In this way, the vehicular device 1 transmits the internal information to an external device by V2X communication and acquires the external information from the external device by V2X communication, thereby enabling information sharing. However, the processing of steps S1 to S2 and the processing of step S3 can be executed in any order.

[0045] In the situation shown in Fig. 2, the vehicular device 1 transmits and receives messages to and from the roadside device 20, and the message received from the roadside device 20 includes information on three objects, E1 corresponding to the other vehicle 23B, E2 corresponding to the host vehicle 2, and E3 corresponding to the other vehicle 23A, as shown in Fig. 4. However, the types and number of pieces of information included in the external information shown in Fig. 4 are merely examples.

[0046] Next, as shown in Fig. 3, the vehicular device 1 compares the internal information with the external information (S4) and determines whether the internal information and the external information contain information on the same target (S5). This process is performed by the comparison unit 10b and the determination unit 10c. Specifically, the vehicular device 1 compares, for example, the information N1, N2, and N3 acquired as the internal information shown in Fig. 4 with the information E1 to E3 acquired as the external information.

[0047] Then, when the internal information and the external information have the same information such as X coordinate, Y coordinate, speed, direction, and type, for example, N1 and E2 or N2 and E3, or when the information is within a range where the two information are considered to be substantially the same, the vehicular device 1 determines that they are the same object. That is, the vehicular device 1 determines whether or not the objects are the same object based on the similarity of the vectors of the objects described in the second embodiment.

[0048] In other words, the vehicle device 1 judges whether the objects are the same object based on information about the object observed from different viewpoints, such as the object's position, speed, and orientation, contained in the internal information and the external information. However, it is not necessary to include all information in the comparison object, and the comparison object can be appropriately set or selected, for example, by comparing only the position, or comparing the position and the speed. Also, each piece of information can be weighted, and whether the objects are the same object can be judged based on the weighted result of the degree of agreement of each piece of information.

[0049] Then, when it is determined that the information of the same object is included (S5: YES), the vehicular device 1 integrates the internal information and the external information (S6). At this time, the vehicular device 1 integrates the internal information and the external information according to a predetermined condition, such as ranking the information or averaging the information. In other words, the vehicular device 1 integrates information about the moving object, including information about the host vehicle 2.

[0050] This makes it possible, for example, to grasp an object approaching the vehicle 2 and use the information to predict danger. Furthermore, if the information of the vehicle 2 is included in the external information, it can be confirmed that the vehicle 2 is at least recognized by the sender of the external information, and if the information of the vehicle 2 contained in the internal information and the external information match, it can also be confirmed that the information sharing system 30 is operating normally.

[0051] Specifically, when the vehicle device 1 acquires the internal information and the external information shown in Fig. 4, and determines that N1 and E2 are the same object and integrates the information, the reliability of N1 is higher than that of E2, so the internal information with higher reliability is prioritized over the external information. As a result, the same information as the internal information is stored for N1, as shown as integrated information.

[0052] In other words, when different pieces of information about the same object are acquired as internal information and external information, the vehicular device 1 integrates the information based on the reliability of each piece of information. In this case, by excluding the information of E2 included in the external information, the amount of information about the same object is reduced compared to the state in which it is distributed between the internal information and the external information.

[0053] Furthermore, when N2 and E3 have the same reliability but have a slight difference in speed, for example, the vehicular device 1 integrates them by averaging them. As a result, the average value of the internal information and the external information is stored for the speed of N2, as shown as integrated information.

[0054] In addition, the vehicular device 1 also stores, as integrated information, information that is included in the internal information but not included in the external information. For example, even if the information corresponding to N3 is not included in the external information, the vehicular device 1 stores the information as integrated information as it is correct information because the reliability of N3 is high. In other words, the vehicular device 1 also integrates information about stationary objects.

[0055] Now, when the vehicle device 1 integrates information related to the same object in step S6, or when it is determined in step S5 that information related to the same object is not included (S5: NO), it determines whether or not there is unacquired information in the external information (S7). Here, unacquired information means information that is not included in the internal information but is included in the external information. For example, in the case of the other vehicle 23B shown in FIG. 2, since it is located in a range (H1) that cannot be detected by the host vehicle 2, it is not included in the internal information as shown in FIG. 3, but is included in the external information as E1.

[0056] In this case, since there is no internal information related to the object corresponding to E1, the vehicular device 1 determines that there is unacquired information (S7: YES) and integrates the information related to E1 as unacquired information (S8). Specifically, the vehicular device 1 adds the information related to E1, which corresponds to the unacquired information, to the integrated information as N4 indicating a fourth object existing in the vicinity of the vehicle 2. In other words, when integrating the information, information that overlaps between the internal information and the external information is reduced, while information included in either one is stored as integrated information. This makes it possible to grasp objects that could not be detected by the vehicle 2.

[0057] Then, the vehicular device 1 passes the integrated information to, for example, the danger prediction unit 10f or the ECU 6 that controls the drive system or the brake system (S9). That is, the vehicular device 1 integrates the internal information and the external information into a state that can be used to notify the user or to control the traveling of the vehicle 2. This allows the vehicle 2 to use the internal information and the external information, which are separate pieces of information obtained from different sources, as a single piece of integrated information.

[0058] The integrated information can be used in a process related to risk prediction. For example, as shown in Fig. 2, when another vehicle 23B that is not detected by the vehicle 2 is approaching an intersection, the risk prediction unit 10f can grasp the relative position between the vehicle 2 and N4, and the type, direction, and speed of N4 by referring to the integrated information.

[0059] In other words, by integrating the external information and sharing the information detected by the sensor on the external device side, the risk prediction unit 10f is able to predict potential risks not detected by the vehicle 2. In addition, since the risk prediction unit 10f is able to predict the risk to the vehicle 2 by executing processing on the integrated information, there is no need to execute separate processing on the internal information and the external information, and overlapping or delaying of processing can be suppressed.

[0060] Then, for example, when the risk prediction unit 10f grasps that there is a possibility that a truck will enter the road on which the vehicle 2 is traveling, the risk prediction unit 10f can notify the user of the risk by displaying, for example, "A truck is coming from the left" on the display device 3 as shown in FIG. 6 since it predicts the possibility of a collision. Alternatively, the risk prediction unit 10f can notify the user of the risk by displaying, for example, "There is a falling object 100 m ahead" on the display device 3 as shown in FIG. 7 since it predicts the possibility of a falling object 29 not included in the external information coming into contact with the vehicle 2. However, the display method on the display device 3 is not limited to the method of displaying characters as shown in FIG. 6 or FIG. 7, and can be a configuration in which the risk is notified by a graphical display method such as displaying an illustration of a vehicle together with an illustration of, for example, a star or cloud that shows a schematic manner of a collision, or a plurality of notification modes combined with a voice notification.

[0061] In this way, by integrating internal information and external information, the vehicular device 1 can share information detected by a sensor or the like of an external device with the vehicle 2, and can improve the accuracy and reliability of risk prediction, such as by predicting risks not detected by the vehicle 2 as described above. In addition, the risk prediction unit 10f can also notify the risk by outputting a sound from the audio input / output device 4.

[0062] Moreover, by transferring the integrated information to the ECU 6, the vehicular device 1 can perform, for example, accelerator control and brake control based on the integrated information. In this case, the ECU 6 only needs to process the integrated information, and can quickly grasp and judge the situation compared to when processing the internal information and the external information separately. As a result, the vehicle 2 can be quickly controlled, and danger can be avoided, that is, safety can be improved.

[0063] According to the embodiment described above, the following effects can be obtained. The vehicle device 1 includes an acquisition unit 10a that acquires information detected by a sensor provided in the vehicle 2 as internal information, a comparison unit 10b that compares the internal information with external information that is acquired outside the vehicle 2 and is received via a communication unit provided in the vehicle 2, a determination unit 10c that determines whether or not both the internal information and the external information contain information relating to the same object based on the comparison result by the comparison unit 10b, and an integration unit 10d that integrates information relating to the same object acquired from different sources based on the determination result by the determination unit 10c.

[0064] This allows the vehicular device 1 to integrate information about the same object among objects contained in the internal information and external information obtained from different sources, and for example, to perform processing about the same object based on a single piece of information. In addition, by processing the internal information and the external information separately, overlapping or repeated processing of the same object is suppressed, thereby reducing the risk of an increase in processing load or a delay in processing. In other words, the vehicular device 1 can quickly process information about the same object.

[0065] Furthermore, if information regarding the same object can be processed quickly, it will be possible, for example, to quickly predict danger and quickly control the vehicle 2 to respond to the predicted danger, thereby improving safety. The integrating unit 10d of the vehicle device 1 integrates information related to moving objects. For example, it becomes possible to grasp an object approaching the vehicle 2 and use the information to predict danger.

[0066] Furthermore, the integration unit 10d of the vehicular device 1 integrates information related to the vehicle 2. As a result, if the information on the vehicle 2 is included in the external information, it is possible to confirm that the vehicle 2 is at least recognized as the source of the external information, and if the information on the vehicle 2 included in the internal information and the external information match, it is possible to confirm that the information sharing system 30 is operating normally.

[0067] Furthermore, the integration unit 10d of the vehicle device 1 integrates the information into a state that can be used to control the traveling of the vehicle 2. This makes it possible to deal with potential danger by performing accelerator control or brake control based on the integrated information, for example, when an object not detected by the vehicle 2 is recognized, thereby improving safety.

[0068] In addition, the integration unit 10d of the vehicle device 1 integrates the information into a state that can be used to notify the user. This makes it possible to notify the user of the danger of an object that is not detected by the vehicle 2, for example, thereby improving safety.

[0069] Furthermore, when different pieces of internal information and external information are acquired for the same object, the integration unit 10d of the vehicle device 1 integrates the information based on the reliability of each piece of information. This makes it possible to improve the reliability of the information and to use highly reliable information for, for example, risk prediction and control of the vehicle 2, thereby further improving safety.

[0070] In addition, the integration unit 10d of the vehicle device 1 integrates information that is not included in the internal information but is included in the external information as unobtained information. This allows the vehicle 2 to grasp objects that are not detected by the vehicle 2, improve the accuracy of risk prediction, and control the vehicle 2 to avoid risk, thereby further improving safety.

[0071] In addition, the integration unit 10d of the vehicle device 1 integrates information that is included in the internal information but not included in the external information as unacquired information. This makes it possible to prevent the information detected by the vehicle 2 from being discarded even if the external information does not contain information on the same object as the object detected by the vehicle 2, thereby improving the accuracy of risk prediction and controlling the vehicle 2 to avoid risk, thereby further improving safety.

[0072] In addition, the information integration method is for integrating information obtained from different sources in a vehicle, and includes the steps of acquiring information detected by a sensor installed in the vehicle 2 as internal information, comparing the internal information with external information obtained outside the vehicle 2 and received via a communication unit installed in the vehicle 2, determining whether or not both the internal information and the external information contain information relating to the same object, and integrating information obtained from different sources regarding the same object based on the determination result.

[0073] If such an information integration method can quickly process information about the same object, it will be possible to quickly predict danger and quickly control the vehicle 2 to respond to the predicted danger, thereby improving safety and achieving the same effects as those of the vehicle device 1. In addition, although the configuration in which the acquired internal information is transmitted has been exemplified, the configuration may be such that integrated information that has already been stored is updated with the acquired internal information and transmitted.

[0074] Second embodiment The second embodiment will be described below. In the second embodiment, a method for setting internal information and its utilization will be described. Although the external information is set by a similar method, the setting in the vehicle device 1 will be described as an example here for the sake of simplicity. In other words, the second embodiment can be combined with the first embodiment.

[0075] The vehicular device 1 extracts features from the acquired image and performs object detection, feature extraction, modality processing, compact representation generation, similarity scoring, identification, and association processing to set the classification. Note that these processes are performed by the control unit 10 using software, but the process may be performed by providing hardware such as an image processing IC.

[0076] The vehicular device 1 acquires images by capturing images of the surroundings using a camera 5a provided on the vehicle 2. At this time, the vehicular device 1 acquires images by capturing images using the camera 5a at preset capturing intervals, such as 5-second intervals, 10-second intervals, 30-second intervals, and the like. When the image is acquired, the vehicular device 1 identifies one or more objects contained in the captured image and extracts features of the identified objects. Note that the feature extraction method described below is a common method in the field of image processing, and therefore will be described here in a simplified manner to allow the flow of processing to be understood.

[0077] The vehicular device 1 extracts a set of multimodal features from the identified object. The multimodal features are also called modality features. Specifically, the vehicular device 1 normalizes the modality features to generate an initial feature vector (V). This initial feature vector includes, for example, the object's position and movement direction, texture features, color features, context features, viewpoint features, and the like.

[0078] The position of an object corresponds to the X and Y coordinates included in the internal information, and the direction of movement corresponds to the orientation included in the internal information. The texture features are described as textures that show various parts of the object, such as the hood, wheels, or bumper, and are, for example, geometric shapes, structures, or texture patterns. The color features are described as the color of the object.

[0079] The context feature is described as the background environment around the object, such as a situation where another vehicle 23 is traveling at an intersection. The viewpoint feature indicates the viewpoint from which an image including the object was captured, and includes the moving direction of the host vehicle 2, the mounting position of the camera 5a, and the orientation when the image was captured.

[0080] Next, the vehicle device 1 performs compact representation generation based on the extracted features. This compact representation generation is a process of generating a compact feature vector capable of expressing an object identified from an image in a compact state, i.e., in a state in which the amount of information is reduced. At this time, the vehicle device 1 generates a so-called learning model in order to efficiently execute the compact representation generation. This compact feature vector is used, for example, to identify the X coordinate, Y coordinate, or type contained in the internal information.

[0081] For example, as shown in the captured image in Fig. 8, an extracted object M1 corresponding to a standard automobile and an extracted object M2 corresponding to a truck are identified from an image captured by the camera 5a. In this case, the vehicle device 1 can acquire the above-mentioned texture features and color features for each object. As a result, the vehicle device 1 can determine the appearance and size of the object by identifying the position of the wheel from the texture features and identifying the relative position with respect to the vehicle 2 from the context features.

[0082] Then, by identifying the type of vehicle that corresponds to the size and appearance of the object, the type can be set. In other words, by using the type information, which has an overwhelmingly smaller amount of information than image data, it is possible to transmit and receive information that can identify the type of object while suppressing the communication load. In other words, by generating conversion information that converts the information acquired by the sensor into a state in which the characteristics of the object that can be identified by the information can be grasped and the amount of information is smaller than at the time of acquisition, and by performing communication using the conversion information, it is possible to share information in each device included in the information sharing system 30 while suppressing the communication load.

[0083] Furthermore, if the type can be identified, the center position of the object can be estimated. For example, when the distance is measured by the millimeter wave radar 5c, the distance between the vehicle 2 and the surface of the object is detected. However, when the distance is transmitted, there is a risk that the value contains an error for another vehicle 23 that is located at a different position from the vehicle 2. Therefore, the vehicle device 1 determines the center position of the object so that it can be used generally in the other vehicle 23.

[0084] Specifically, for example, in the case of the extracted object M1, since it is determined to be a passenger car, the distance to the detected object and half the width associated with the type are added to determine the center position (P0) of the extracted object M1 as shown as a compact in Figure 8. In addition, once the center position is determined, it becomes possible to identify the coordinates of each vertex (P1 to P8) of a rectangular parallelepiped that connects the outer edge of the object, that is, the three-dimensional shape of the object. The coordinates of this center position (P0) become the X and Y coordinates included in the internal information.

[0085] Similarly, the center position (P0) of the extracted body M2 is obtained. Once the center position is obtained, the coordinates of each vertex (P1 to P8) of the rectangular parallelepiped that connects the outer edge of the object can be specified. Then, the compact feature vector represented by the center position and each vertex makes it possible to provide information that can specify the three-dimensional shape of the object with less information volume than image data.

[0086] Then, by associating the compact feature vector with the initial feature vector indicated by the black arrow, it becomes possible to transmit the internal information summarizing the features of the object to an external device as a message, or to receive the internal information from an external device as a message. As a result, each vehicle can determine whether the objects are the same object by identifying the center position and type of the object. In other words, it is possible to share information capable of grasping the features of the object in a compact state, without transmitting and receiving image data between the vehicle 2 and the vehicle itself.

[0087] In addition, since the relative positional relationship and orientation with respect to the vehicle itself, as well as the three-dimensional shape of the object, can be obtained, each vehicle can further utilize the obtained information for vehicle control and risk prediction. For example, as shown in Fig. 9, assume a situation in which the vehicle 2 and the other vehicle 23H are traveling in opposite directions and the other vehicle 23I is crossing between them. In this case, when viewed from the center position of the object, the center position of the vehicle 2 indicated by the line segment (CL1) and the center position of the other vehicle 23H indicated by the line segment (CL2) are shifted from the center position in the front-rear direction of the other vehicle 23I indicated by the line segment (CL3).

[0088] In this case, the center position of the host vehicle 2 is shifted from the center position of the other vehicle 23I, and is also off the rear end of the other vehicle 23I in the three-dimensional shape, so it can be determined that there is little risk of contact, etc. On the other hand, although the center position of the other vehicle 23H is shifted from the center position of the other vehicle 23I, it can be seen that it is at the front end of the other vehicle 23 when viewed in the three-dimensional shape.

[0089] In other words, if the positional relationship between the other vehicles 23H and 23I remains the same, it is considered that there is a high possibility of collision, so the other vehicles 23H and 23I can reduce the speed of the vehicles or warn of a serious danger. shape It is believed that safety can be improved by taking this into account when controlling the vehicle and predicting danger.

[0090] Incidentally, the internal information and the external information are not limited to information about objects, but may include information about the surrounding environment. For example, as shown in Fig. 10, assume that another vehicle 23J is traveling ahead of the vehicle 2 on a two-lane road, and another vehicle 23K is traveling next to the other vehicle 23J. In this case, depending on the detection capability of the sensor, it may be detected that the other vehicle 23J and the other vehicle 23K are traveling in the same direction at approximately the same position.

[0091] In this case, if a boundary line (L1) indicating the boundary between the lanes is identified from the image captured by the camera 5a, and it can be determined that the other vehicle 23J is traveling in the lane to the left of the boundary line, and the other vehicle 23K is traveling in the lane to the right of the boundary line, the other vehicles 23J and 23K can be recognized individually. Then, by including the positional relationship between the object and the boundary line in the determination of whether they are the same object, it can be accurately determined whether they are the same object.

[0092] In this way, the vehicle device 1 judges whether or not the objects are the same based on at least one of the items of information among the object position, speed, direction, three-dimensional shape, and the conformity with the surrounding environment included in the internal information and the external information, or based on the conformity of a plurality of items of information, or based on the result of weighting each item of information, thereby making it possible to more accurately judge whether or not the objects are the same.

[0093] Also, the type of the internal information can be set by including the type of the vehicle itself in the information transmitted from each vehicle, instead of extracting the feature points as described above. Alternatively, the type of the internal information can be determined by comparing the appearance of an object identified in an image captured by the camera 5a with a database 11a that stores, for example, a plurality of vehicle images, and selecting an object with a similar appearance as the type.

[0094] Furthermore, with this configuration, it is possible to integrate information about the same object contained in internal information and external information obtained from different sources, and to quickly process information about the same object, thereby obtaining effects similar to those of the first embodiment.

[0095] Third embodiment A third embodiment will be described below. In the third embodiment, a method for adjusting the amount of information will be described. Although the external information can be adjusted by a similar method, the setting in the vehicular device 1 will be described here as an example for the sake of simplicity. Since the configuration of the vehicular device 1 is common to the other embodiments, the description will be given with reference to Figs. 1 to 10. A part of the configuration of the third embodiment is also related to the setting method of the internal information described in the second embodiment. In other words, the third embodiment can be combined with the first and second embodiments.

[0096] <Adjusting the amount of information to be integrated> First, a configuration for adjusting the amount of information when integrating information will be described. In the first embodiment, the individual pieces of information included in the acquired external information are selected or averaged, but the vehicular device 1 can select the pieces of information to be integrated using a specific criterion.

[0097] 11, a situation is assumed in which a plurality of roadside units 20A to 20D are installed and a plurality of other vehicles 23A to 23G that are compatible with V2V are present within a communication range (K1) of the V2X communication of the vehicle 2. Note that if a pedestrian 26 or a bicycle 27 has a device capable of communicating with a vehicle, the amount of acquired external information is assumed to further increase, but for the sake of simplicity of explanation, a situation is assumed in which external information can be received from the roadside units 20A to 20D and the other vehicles 23C to 23E.

[0098] In this case, the vehicle 2 can integrate all the external information acquired from multiple sources, but if the amount of acquired external information is large, the load of the integration process or the processing of the integrated information may increase or be delayed. Therefore, the vehicular device 1 can prioritize the received external information and select the external information to be integrated, or select the information included in the external information to be integrated.

[0099] The vehicular device 1 adjusts the amount of information by executing an information integration adjustment process shown in Fig. 12. In Fig. 12, steps that are substantially the same as those in the information integration process described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0100] The vehicle device 1 acquires internal information (S1), acquires external information (S3), compares the internal information with the external information (S4), and if there is information relating to the same object (S5: YES), integrates the information relating to the same object (S6). Also, if there is unacquired information (S7: YES), the vehicle device 1 integrates the unacquired information (S8).

[0101] At this time, the vehicular device 1 can obtain the distance between the acquisition source and the vehicle 2 based on information such as the X coordinate, the Y coordinate, and the direction included in the acquired multiple pieces of external information, and select a predetermined number of pieces of external information as integration targets in order from the external information acquired at locations close to the vehicle 2. In the situation shown in FIG. 11, it is assumed that the distances from the vehicle 2 to the vehicle 2 are closest in the order of the roadside unit 20A, the other vehicle 23E, the other vehicle 23C, the roadside unit 20D, the roadside unit 20B, the roadside unit 20C, and the other vehicle 23D. In this case, the vehicular device 1 can preferentially select up to a predetermined upper limit number of pieces of external information as integration targets in order from the closest distance. However, the upper limit number can be appropriately set according to the processing capacity of the vehicular device 1, etc.

[0102] Furthermore, the vehicular device 1 can preferentially select, as an integration target, from among the multiple pieces of external information, external information that includes information on an object approaching the host vehicle 2 or an object that is expected to approach the host vehicle 2. In the case of the situation shown in Fig. 11, the other vehicle 23C is traveling in the same direction as the host vehicle 2, and the other vehicle 23D is traveling toward the road on which the host vehicle 2 is traveling. Therefore, it is considered that the other vehicles 23C and 23D may approach the host vehicle 2.

[0103] In this case, the vehicular device 1 can preferentially select as integration targets the external information acquired from the other vehicles 23C and 23D that are expected to have a large impact on the control and risk prediction of the vehicle 2. If there is a margin of processing capacity, other external information can be acquired, for example, based on distance.

[0104] Furthermore, the vehicular device 1 can preferentially select the external information transmitted from the roadside device 20 as the integration target. Generally speaking, the roadside devices 20 are considered to have common communication standards and sensor performance, and are also subject to regular inspections. Therefore, the external information acquired by the roadside device 20 is considered to be highly reliable, so the vehicular device 1 can preferentially select the external information acquired from the roadside device 20 as the integration target. If there is a margin in the processing capacity, it is possible to acquire other external information based on, for example, distance or a positional relationship with the vehicle 2.

[0105] Furthermore, the vehicular device 1 can preferentially select, for example, external information acquired on the planned travel route (T1) set by the navigation function or in the vicinity of the planned travel route as the integration target. When the communication range is wider than the detection range of the vehicle-side sensor 5, information on objects not detected by the vehicle 2 can be acquired as external information. Therefore, by integrating information acquired from acquisition sources in the vicinity of the planned travel route, it becomes possible to predict dangers on the planned travel route in advance, thereby improving safety.

[0106] In addition, the vehicular device 1 can select individual pieces of information included in the external information, rather than selecting the external information itself. For example, assume that the roadside device 20C detects another vehicle 23G and a bicycle 27 near the planned travel route, and the external information includes, as individual pieces of information, that the other vehicle 23 is moving away from the vehicle 2 and that the bicycle 27 is moving toward the vehicle 2.

[0107] In this case, the vehicular device 1 can individually select, from the external information received from the roadside device 20C, information on an object moving in a direction approaching the vehicle 2 as an object to be integrated. This makes it possible to obtain, as integrated information, information that is expected to have a large effect on the control of the vehicle 2 and risk prediction, thereby improving safety.

[0108] In this way, the vehicle device 1 does not integrate all of the received external information, but instead selects and integrates the external information and the individual information contained in the external information based on predetermined criteria, thereby reducing the risk of the amount of integrated information increasing, resulting in an excessively large processing load or processing delays.

[0109] <Adjusting the amount of information in the integrated information> Next, a configuration for adjusting the amount of integrated information will be described. When the vehicular device 1 integrates information in the information integration adjustment process shown in Fig. 12, it adjusts the amount of integrated information (S20). At this time, the adjustment of the amount of information is performed by the adjustment unit 10e. However, the information adjustment referred to here includes a configuration for increasing the amount of information to improve the convenience and usefulness of information shared with an external device, and a configuration for reducing the amount of information when transmitting to an external device mainly to reduce the communication load in V2X communication.

[0110] First, a configuration example in which the amount of information is adjusted by adding information will be described. As described above, the vehicular device 1 can detect the color of an object using the camera 5a. It is considered that if the color of an object is notified, the user can easily visually grasp the object. For this reason, the vehicular device 1 provides an item for a color tone indicating the color of the object as shown in FIG. 13, and assigns an identification number corresponding to the color, thereby adding the color of the detected object as additional information to the integrated information as shown in FIG. 14.

[0111] This makes it possible to issue an alert such as "There is a red falling object" when predicting danger regarding falling object 29 N3, for example, and makes it easier for users to identify which object is falling object 29 when they approach it, which is believed to improve safety.

[0112] Furthermore, in the case where the boundary line or center line is grasped by the camera 5a as described above, an item for the lane in which the object is located is provided as shown in Fig. 15, and an identification number corresponding to the lane and an identification number indicating a relative positional relationship with the vehicle 2 are assigned, so that the lane in which the detected object is located is added as additional information to the integrated information as shown in Fig. 14. For example, possible lanes include the same lane as the vehicle 2: ahead, the same lane: behind, the lane on the left side of the traveling direction, the lane on the right side of the traveling direction, the oncoming lane, the intersecting lane: connecting from the left, the intersecting lane: leaving to the right, bicycle lane 27, etc. However, what is shown here is merely an example, and the classification and number of lanes are not limited to these.

[0113] This makes it possible to issue a warning such as "a red falling object is in your path" when predicting danger regarding falling object 29 N3, for example. This makes it easier for users to understand which object 29 is when they approach it, and furthermore whether or not there is a possibility of contact with that object 29, which is believed to improve safety.

[0114] In this way, the vehicular device 1 can increase the convenience and effectiveness of the integrated information by adding some information to the integrated information, in other words, by adjusting the integrated information to increase the amount of information.

[0115] Next, a configuration example in which the amount of information is adjusted by reducing the amount of information will be described. By transmitting the above-mentioned integrated information to an external device, it becomes possible to share information on the external device side. However, if the integrated information is transmitted as is, there is a risk that the communication load of V2X communication will increase or the band will be insufficient. In addition, although the above-mentioned additional information is effective for the host vehicle 2, it may not necessarily be required for the other vehicle 23 or the external device.

[0116] Therefore, when adjusting the amount of information in step S20 shown in Fig. 12, the vehicular device 1 generates transmission information to be transmitted to an external device by reducing the amount of information. At this time, the vehicular device 1 adjusts the amount of information by filtering the integrated information based on a predetermined selection criterion. The selection criterion may be, for example, as follows, but the selection criterion shown here is merely an example, and other selection criteria may also be set.

[0117] - Time to collision between vehicle 2 and an object (TTC: Time-To-Collision) Distance between vehicle 2 and the object - Reliability of information about the object Object type · Whether information about the object is included in the internal information ·Whether information about the object is included in the external information -Did the number of objects reach a certain limit?

[0118] Then, the vehicular device 1 filters the integrated information to which the additional information shown in Fig. 14 is added according to one of the selection criteria or according to multiple conditions. For example, the vehicular device 1 extracts, from the integrated information shown in Fig. 14, N1, which is the host vehicle 2, N3, which is not included in the external information, and N4, which is close to the host vehicle 2, as targets for transmission, as shown in Fig. 16 as an example of transmission information No. 1, while excluding N4, which is integrated from the external information, from the targets for transmission. Note that the close distance to the host vehicle 2 suggests that the TTC is short and the risk is high, although this depends on the moving direction between the host vehicle 2 and the object.

[0119] Furthermore, the vehicular device 1 extracts the X coordinate, Y coordinate, speed, type, direction, and reliability from the items of information about the extracted object, and basically reduces information such as lane, color tone, and relative speed used by the vehicle 2. This generates transmission information with a reduced amount of information compared to the integrated information shown in Fig. 14. Note that while the transmission information can be generated and stored as information separate from the integrated information, it can also be configured to dynamically generate and transmit the transmission information at the time of transmission by extracting information from the integrated information according to the above-mentioned selection criteria.

[0120] In addition, the vehicular device 1 can change or add items for each object, as shown in Example 2 of the transmission information in Fig. 16. This is because, when selection is made according to the selection criteria, it is assumed that the number of objects to be transmission targets does not reach a predetermined upper limit, and in that case, it is considered that there is a margin in the communication band.

[0121] Therefore, the vehicular device 1 can include other objects in the transmission target according to the priority set in the selection criteria, or increase the amount of information about the object to be transmitted. For example, the vehicular device 1 can include information that can make the object easier to visually grasp as described above by including a color tone for N3, which is the falling object 29, in the transmission information. Alternatively, the vehicular device 1 can include information on the degree of danger recognized by the vehicle 2 for N4, which is considered to be a relatively high degree of danger due to the possibility of contact.

[0122] After adjusting the amount of information, the vehicular device 1 transmits a message in which the transmission information is encoded in a predetermined format to the outside (S21), as shown in Fig. 12. This makes it possible for the information transmitted from the vehicle 2 to be received by an external device, and the information can be shared.

[0123] In this way, the vehicular device 1 adjusts the amount of information to be transmitted. As a result, the amount of information to be transmitted is reduced compared to the amount of all information held by the vehicle 2, so that the communication load in the V2X communication can be reduced. Also, when there is a surplus in the V2X communication band, more information can be shared.

[0124] In this case, the vehicle device 1 can generate the information to be transmitted from integrated information to which additional information has been added as described above, or can generate the information to be transmitted from the acquired internal information.

[0125] With this configuration, it is possible to integrate information about the same object contained in internal information and external information obtained from different sources, and to quickly process information about the same object, thereby obtaining effects similar to those of the first embodiment.

[0126] Furthermore, when adjusting the amount of information in step S20, the vehicular device 1 can be configured to adjust the amount of information of the internal information acquired in step S1. That is, the vehicular device 1 is not limited to a configuration in which the amount of information of the integrated information is adjusted as described above, but can be configured to reduce the amount of communication when the information is transmitted as a message by adjusting the amount of information before integration. Furthermore, the vehicular device 1 can be configured to reduce the amount of communication when transmitting external information once acquired to an external device again as a message by transmitting information with a reduced amount of information as a message.

[0127] Specifically, the vehicular device 1 can reduce the amount of information by selecting and discarding information included in the internal information acquired in step S1 based on, for example, the selection criteria described above. Then, in step S21, the vehicular device 1 can reduce the amount of communication in the V2X communication with the external device by transmitting the information from which the amount of information has been deleted as a message to the external device. In this way, by configuring the device to be able to reduce the amount of information transmitted as a message, it is possible to reduce the risk of a shortage of resources such as communication bands, and since it is considered that the communication time can be shortened if the amount of information is small, it is possible to realize rapid communication.

[0128] Furthermore, not only the vehicle 2 but also the roadside unit 20 and the other vehicle 23 can be configured to reduce the amount of information before transmitting the information acquired by them as a message to the outside. This makes it possible to reduce the risk of insufficient communication bandwidth and realize rapid communication not only between the vehicle 2 and an external device but also between devices capable of V2X communication, such as the roadside unit 20 and the other vehicle 23, between the other vehicles 23, and between the roadside units 20, in other words, in the information sharing system 30.

[0129] In addition, the configuration for reducing the amount of information transmitted as a message can be applied to the information integration configuration as described above. Specifically, for example, when transmitting integrated information to which additional information has been added as a message, the information can be reduced before being transmitted as the message. With such a configuration, the amount of communication can be reduced, the risk of a shortage of communication band can be reduced, and rapid communication can be achieved.

[0130] (Other embodiments) The vehicular device 1 can be configured as a dedicated device for integrating information, but can also be configured to share or share with, for example, a navigation device. In that case, the display device 3 can also be configured to share or share with the navigation device. Also, the camera 5a can be configured to share or share with a so-called drive recorder.

[0131] In the embodiment, an XY coordinate system with a predetermined reference point as the origin is set, but for example, an XY coordinate system with the position of the vehicle 2 as the origin can be set, or an XYZ coordinate system with a Z coordinate indicating the elevation difference can be set. In that case, if the current position of the vehicle 2 is transmitted at the same time, the position of the object can be grasped on the external device side. Also, it is possible to configure the processing to use the latitude and longitude acquired by GNSS5e as they are without converting to the coordinate system.

[0132] In the embodiment, an example in which the unacquired information is integrated as it is has been shown, but it is also possible to adopt a configuration in which it is determined whether or not to integrate the unacquired information based on the X coordinate and Y coordinate of the unacquired information. Also, it is also possible to adopt a configuration in which it is determined whether the relevant object is in a range in which it is impossible or difficult to detect on the vehicle 2 side, and if the object is detectable but has not been acquired, the external information is regarded as erroneous information and is reacquired, or a configuration in which the user is notified of the possibility of a malfunction occurring in the sensor on the vehicle 2 side.

[0133] Although the present disclosure has been described based on the embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, and other combinations and forms including only one element, more than one element, or less than one element, are also included in the scope and concept of the present disclosure.

[0134] The control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer. [Explanation of symbols]

[0135] In the drawings, 1 indicates a vehicle device, 2 indicates the subject vehicle, 5 indicates a vehicle-side sensor, 5a indicates a camera, 5b indicates a LiDAR, 5c indicates a millimeter wave radar, 5d indicates a gyro, 5e indicates a GNSS, 9 indicates an on-board sensor, 10a indicates an acquisition unit, 10b indicates a comparison unit, 10c indicates a judgment unit, 10d indicates an integration unit, 10e indicates an adjustment unit, 14 indicates a V2X communication unit, 23 indicates another vehicle (object, moving body), 24 indicates a building (object, stationary object), 26 indicates a pedestrian (object, moving body), 27 indicates a bicycle (object, moving body), and 28 indicates a fallen object (object, stationary object).

Claims

1. an acquisition unit (10a) that acquires information detected by a sensor (5, 9) provided in the vehicle (2) as internal information; a comparison unit (10b) that compares the internal information with external information that is acquired outside the vehicle and is received via a communication unit (14) provided in the vehicle; a determination unit (10c) that determines whether or not the internal information and the external information both contain information about the same subject based on a comparison result by the comparison unit; an integration unit (10d) that integrates information about the same subject acquired from different sources into integrated information based on a determination result by the determination unit; and an adjustment unit (10e) that adjusts an amount of the integrated information according to a predetermined selection criterion.

2. The vehicle device according to claim 1 , wherein the integration unit integrates information related to a moving object.

3. The vehicle device according to claim 1 , wherein the integration unit integrates information related to the vehicle.

4. The vehicle device according to claim 1 , wherein the integration unit integrates the information into a state that can be used to control traveling of the host vehicle.

5. The vehicle device according to claim 1 , wherein the integration unit integrates the information into a state that can be used for notifying a user.

6. The vehicle device according to claim 1 , wherein when different pieces of information are acquired as the internal information and the external information regarding the same object, the integration unit integrates the pieces of information based on a reliability of each piece of information.

7. The vehicle device according to any one of claims 1 to 6, wherein the integration unit adds information that is not included in the internal information but is included in the external information to the integration target as unacquired information that has not been acquired by the vehicle.

8. The vehicle device according to any one of claims 1 to 7, wherein the integration unit adds information that is included in the internal information but not included in the external information to the integration target as unobtained information that has not been obtained outside the vehicle.

9. The vehicle device according to any one of claims 1 to 8, wherein the determination unit determines whether the internal information and the external information are the same object based on at least one of information on the consistency between the object's position, speed, orientation, three-dimensional shape, or surrounding environment contained in the internal information and the external information.

10. A vehicle device as described in any one of claims 1 to 9, wherein the adjustment unit adjusts the amount of information according to a priority based on the magnitude of the impact on the control of the vehicle or hazard prediction set in the selection criteria.

11. An information integration method for integrating information obtained from different sources in a vehicle, comprising: acquiring information detected by a sensor (5, 9) provided in the vehicle (2) as internal information; A step of comparing the internal information with external information acquired outside the vehicle and received via a communication unit (14) provided in the vehicle; determining whether the internal information and the external information both contain information about the same subject; a step of integrating information relating to the same subject acquired from different sources into integrated information based on a result of the determination; and adjusting an amount of information in the integrated information according to a predetermined selection criterion.

Images