Information processing device, program, and information processing method

Abstract

A data analysis device (120) comprises: a vehicle data acquisition unit (121) which acquires vehicle data acquired from the inside of a target vehicle; a vehicle exterior image data acquisition unit (122) which acquires vehicle exterior image data that indicates an image including a plurality of vehicles; a first feature change detection unit (124) which generates, from the vehicle data, first feature change data that indicates a change in the feature of the target vehicle; a second feature change detection unit (125) which specifies, from the vehicle exterior image data, changes in the features of the plurality of respective vehicles and generates a plurality of pieces of second feature change data each indicating a change in the feature of one of the plurality of vehicles; and a similarity specification unit (128) which specifies a similarity that indicates the degrees of likeness between the plurality of respective pieces of second feature change data and the first feature change data, specifies one piece of second feature change data that has the highest similarity, and determines that one vehicle, which is a generation source of the one piece of specified second feature change data, matches the target vehicle.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a program, and an information processing method. [Background technology]

[0002] Due to various types of automation related to automobile operation, a large amount of vehicle data, such as sensor data and control data, is being generated inside the automobile. In addition, with the increasing social demand for elucidating the causes of accidents, the recording of video data by a drive recorder inside the vehicle is becoming common as vehicle data.

[0003] Furthermore, for the purpose of road management, external cameras installed on road facilities, such as roadside cameras or intersection cameras, that detect road surface conditions or vehicle movements are becoming more widespread. In addition to roadside cameras installed on the side of the road, external cameras such as surveillance cameras are often installed in parking lots and the like for crime prevention purposes.

[0004] It has been proposed to perform advanced data analysis or provide services by linking vehicle data and outside-vehicle video data, which is video data from an outside-vehicle camera. In this case, in order to link outside-vehicle video data that includes multiple automobiles, such as camera video that captures multiple automobiles, with vehicle data generated inside an individual automobile, it is necessary to associate the vehicle data with which of the multiple automobiles included in the outside-vehicle video data the vehicle data belongs.

[0005] For example, as a conventional technique, a traffic information system described in Patent Document 1 includes a radio wave source detection device that receives radio waves from ITS vehicle-mounted devices mounted on multiple vehicles near an intersection and detects radio wave sources, an imaging unit that is installed near the intersection and captures video including multiple vehicles entering the intersection, and a Dedicated Short Range Communications (DSRC) roadside wireless device that compares the video captured by the imaging unit, the positions of the multiple ITS (Intelligent Transport Systems) vehicle-mounted devices detected by the radio wave source detection device, and map data for the intersection to identify vehicles that have changed course within the intersection. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2009-110153 A Summary of the Invention [Problem to be solved by the invention]

[0007] In the case of conventional technology, a calibration work needs to be performed when installing a DSRC roadside wireless device in order to obtain a positional correspondence between the image, map data, and the detection result of the radio wave emission source detection device. For example, it is necessary to measure the coordinates of the GPS (Global Positioning System) using a dedicated device and create correspondence data indicating where in the image the coordinates correspond. The correspondence data created in this way becomes meaningless if the imaging conditions, such as the installation position, orientation, or angle of view of the imaging unit, change, so that if the imaging conditions change, a calibration work needs to be performed again. For this reason, it was difficult to use the conventional system in practical operation.

[0008] Therefore, one or more aspects of the present disclosure aim to easily associate a vehicle that outputs vehicle data with a vehicle included in the outside-vehicle video data. [Means for solving the problem]

[0009] An information processing device according to an aspect of the present disclosure includes: a vehicle data acquisition unit that can indicate features of a target vehicle that is a vehicle to be detected and that acquires vehicle data that is data acquired inside the target vehicle; an outside-vehicle video data acquisition unit that acquires outside-vehicle video data that is video data captured from the outside of a plurality of vehicles and that indicates a video including the plurality of vehicles; a first feature change detection unit that generates first feature change data that indicates a change in the features of the target vehicle from the vehicle data; a second feature change detection unit that identifies a change in the features of each of the plurality of vehicles from the outside-vehicle video data and generates a plurality of second feature change data each indicating the change in the feature of one of the plurality of vehicles; and a similarity identification unit that identifies a similarity indicating a degree of similarity between each of the plurality of second feature change data and the first feature change data, identifies one second feature change data with the highest similarity, and determines one vehicle from which the one second feature change data was generated among the plurality of vehicles to match the target vehicle. the vehicle data is data capable of identifying display content of an exterior display device which is a device provided in the target vehicle and which displays information to the outside of the vehicle, the first feature change detection unit generates the first feature change data indicating a time series of changes in the display content from the vehicle data, the second feature change detection unit generates the second feature change data indicating a time series of changes in the display content of the exterior display device included in the video by analyzing the video if any of the plurality of vehicles is provided with an exterior display device, and the similarity identification unit identifies the similarity based on a degree of correlation between the time series indicated by the first feature change data and the time series indicated by each of the plurality of second feature change data. It is characterized by:

[0010] A program according to one aspect of the present disclosure causes a computer to function as: a vehicle data acquisition unit that acquires vehicle data, which is data that indicates characteristics of a target vehicle that is a vehicle to be detected and that is acquired inside the target vehicle; an outside-vehicle image data acquisition unit that acquires outside-vehicle image data that is image data that is captured from the outside of a plurality of vehicles and indicates an image including the plurality of vehicles; a first feature change detection unit that generates first feature change data that indicates a change in the characteristics of the target vehicle from the vehicle data; a second feature change detection unit that identifies a change in the characteristics of each of the plurality of vehicles from the outside-vehicle image data and generates a plurality of second feature change data, each of which indicates the change in the characteristic of one of the plurality of vehicles; and a similarity identification unit that identifies a similarity indicating a degree of similarity between each of the plurality of second feature change data and the first feature change data, identifies one second feature change data with the highest similarity, and determines one vehicle, which is a source of generation of the one second feature change data, among the plurality of vehicles, to match the target vehicle. the vehicle data is data capable of identifying display content of an exterior display device which is a device provided in the target vehicle and which displays information to the outside of the vehicle, the first feature change detection unit generates the first feature change data indicating a time series of changes in the display content from the vehicle data, the second feature change detection unit generates the second feature change data indicating a time series of changes in the display content of the exterior display device included in the video by analyzing the video if any of the plurality of vehicles is provided with an exterior display device, and the similarity identification unit identifies the similarity based on a degree of correlation between the time series indicated by the first feature change data and the time series indicated by each of the plurality of second feature change data. It is characterized by:

[0011] An information processing method according to one aspect of the present disclosure includes acquiring vehicle data that indicates characteristics of a target vehicle that is a vehicle to be detected, acquiring outside vehicle image data that is image data captured from the outside of a plurality of vehicles and that indicates an image including the plurality of vehicles, generating first characteristic change data that indicates a change in the characteristics of the target vehicle from the vehicle data, identifying the change in the characteristics of each of the plurality of vehicles from the outside vehicle image data, generating multiple second characteristic change data each indicating the change in the characteristics of one of the plurality of vehicles, identifying a similarity that indicates a degree of similarity between each of the multiple second characteristic change data and the first characteristic change data, identifying one second characteristic change data with the highest similarity, and determining that one vehicle from the plurality of vehicles that is the source of the generation of the one second characteristic change data matches the target vehicle. An information processing method, wherein the vehicle data is data capable of identifying display content of an exterior display device that is equipped in the target vehicle and displays information toward the outside of the vehicle, the first feature change data indicating a time series of changes in the display content is generated from the vehicle data, and by analyzing the video, if any of the plurality of vehicles is equipped with an exterior display device, the second feature change data indicating a time series of changes in the display content of the exterior display device included in the video is generated, and the similarity is identified based on a degree of correlation between the time series indicated by the first feature change data and the time series indicated by each of the plurality of second feature change data. It is characterized by: Effect of the Invention

[0012] According to one or more aspects of the present disclosure, it is possible to easily associate a vehicle that outputs vehicle data with a vehicle included in the outside-vehicle video data. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating a schematic configuration of a data analysis system including a data analysis device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of an image captured by a roadside camera. [Diagram 3] 1 is a block diagram illustrating a schematic configuration of a data analysis device according to an embodiment. [Figure 4] 1(A) to 1(C) are schematic diagrams for explaining an outline of processing in a data analysis device. [Diagram 5] 13A and 13B are schematic diagrams for explaining the process in which a vehicle tracking unit tracks each vehicle in an image. [Figure 6] 1A to 1C are schematic diagrams illustrating detection of the blinking state of the brake lights of a tracked vehicle. [Figure 7] 13(A) to 13(C) are schematic diagrams for explaining a process of comparing the similarity of a selected signal while shifting the signal in the time direction. [Figure 8] 13 is a flowchart illustrating an example of a process performed by a similarity specifying unit. [Figure 9] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the data analysis device. [Figure 10] 13(A) to 13(C) are schematic diagrams illustrating a process for identifying a vehicle using the timing of changes in door opening and closing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Embodiment FIG. 1 is a block diagram showing a schematic configuration of a data analysis system 100 including a data analysis device 120 according to an embodiment. The data analysis system 100 includes roadside cameras 110A, 110B, and 110C as vehicle exterior cameras, and a data analysis device 120 as an information processing device.

[0015] The roadside cameras 110A, 110B, and 110C are cameras serving as imaging devices arranged at the side of the road 101. The roadside cameras 110A, 110B, and 110C transmit outside-vehicle image data, which is image data showing images of the vehicles 102A-102G traveling on the road 101, to the data analysis device 120 via wired or wireless communication. The outside-vehicle image data is captured from outside the multiple vehicles 102A-102G, and shows an image including any one of the multiple vehicles 102A-102G depending on the location and time.

[0016] The vehicles 102A to 102G wirelessly transmit various data acquired within the vehicles 102A to 102G to the data analysis device 120 as vehicle data.

[0017] When there is no need to distinguish between the roadside cameras 110A, 110B, and 110C, any one of the roadside cameras 110A, 110B, and 110C will be referred to as the roadside camera 110. Although three roadside cameras 110A, 110B, and 110C are shown in Fig. 1, there is no particular limit to the number of roadside cameras 110. Moreover, when there is no need to distinguish between the vehicles 102A to 102G, any one of the vehicles 102A to 102G will be referred to as the vehicle 102.

[0018] FIG. 2 is a schematic diagram showing an example of an image captured by roadside camera 110B. In this example, three vehicles 102A to 102C are captured in an image Im1, which is a frame captured at a certain moment. The roadside camera 110 captures images over a relatively wide field of view, and often multiple vehicles 102 are captured in the image.

[0019] From the image captured by the roadside camera 110, it is possible to obtain information that cannot be detected by a sensor or an imaging device mounted on the vehicle 102. For example, the image captured by the roadside camera 110 may include the state of the vehicle 102 several vehicles ahead, or the appearance of a child who may run out into the road.

[0020] By analyzing the outside vehicle video data and the vehicle data in conjunction with each other, the data analysis device 120 can perform more efficient acceleration / deceleration control, preparatory actions to prevent accidents, and cause estimation that takes into account the surrounding conditions when an accident occurs.

[0021] When analyzing vehicle data and outside-vehicle image data in association with each other, it is necessary to identify which vehicle 102 in the image shown by the outside-vehicle image data is the transmission source of the vehicle data. For example, if braking operations are performed successively by vehicles 102A and 102B in image Im1 shown in FIG. 2, unless the transmitted vehicle data can be matched with the vehicle 102 in image Im1, it is not possible to analyze the causal relationship, such as whether the brakes were applied by leading vehicle 102A and therefore also by following vehicle 102B, or whether the brakes were applied by following vehicle 102B alone.

[0022] In addition, when a vehicle 102 detects a sudden sudden stop from a blind spot that cannot be seen by the vehicle 102 from the exterior video data of the roadside camera 110, it becomes impossible to identify which vehicle 102 should be instructed to brake suddenly.

[0023] Furthermore, even if a vehicle 102 is captured in the image shown by the exterior video data, for example, if the vehicle 102 is captured in both the image from roadside camera 110A and the image from roadside camera 110B shown in Figure 1, it is necessary to identify which roadside camera 110's image is the source of the vehicle data.

[0024] A vehicle 102 in an image is generally identified by recognizing its license plate, but in order to confirm the license plate from the image, a camera needs to be installed above the road 101, and there are limitations on where it can be installed.

[0025] In this embodiment, an example will be described in which vehicle data and outside-vehicle video data are simply linked in the data analysis device 120.

[0026] FIG. 3 is a block diagram showing a schematic configuration of a data analysis device 120 according to the embodiment. The data analysis device 120 includes a vehicle data acquisition unit 121 , an outside-vehicle video data acquisition unit 122 , a data selection unit 123 , a first feature change detection unit 124 , a second feature change detection unit 125 , and a similarity identification unit 128 .

[0027] The vehicle data acquisition unit 121 acquires the vehicle data transmitted from the vehicle 102. In the data analysis device 120, the vehicle 102 that has transmitted the vehicle data is treated as a target to be detected from within the image of the outside-vehicle image data, and therefore the vehicle 102 that has transmitted the vehicle data is also referred to as a target vehicle. The vehicle outside image data acquisition unit 122 acquires vehicle outside image data from the roadside camera 110 .

[0028] The data selection unit 123 selects data to be compared from the vehicle data acquired by the vehicle data acquisition unit 121 and the vehicle exterior image data acquired by the vehicle exterior image data acquisition unit 122. Here, the selected data from the vehicle data is also referred to as selected vehicle data, and the selected vehicle exterior image data from the vehicle exterior image data is also referred to as selected vehicle exterior image data.

[0029] For example, the data selection unit 123 selects data that can be compared between the vehicle data acquired by the vehicle data acquisition unit 121 and the outside-vehicle image data acquired by the outside-vehicle image data acquisition unit 122. Then, the data selection unit 123 provides the selected vehicle data and selected outside-vehicle image data, which are the selected data, to the first feature change detection unit 124 and the second feature change detection unit 125. Furthermore, the data selection unit 123 instructs the first feature change detection unit 124 and the second feature change detection unit 125 on the method of comparing the selected vehicle data with the selected outside-vehicle image data.

[0030] The data selection unit 123 may select and compare data according to, for example, predetermined conditions. At this time, the data selection unit 123 may select a combination of data that can be compared between the vehicle data and the external video data at a predetermined timing. For example, the data selection unit 123 may determine whether or not the external video data acquired by the external video data acquisition unit 122 includes information on the same time period and the same area based on information such as the recording time of various information held by the vehicle data acquired by the vehicle data acquisition unit 121, the travel route during that time period, or the vehicle position, thereby determining whether or not the data can be compared. In addition, the data selection unit 123 may identify the data to be selected and the comparison method by accepting input from an operator, for example, via an input unit not shown, or via the vehicle data acquisition unit 121 or the outside vehicle video data acquisition unit 122.

[0031] The first characteristic change detection unit 124 uses the selected vehicle data from the data selection unit 123 to detect a change in the characteristics of the vehicle 102 that transmitted the selected vehicle data, and generates first characteristic change data that is characteristic change data indicating the change in the characteristics. The generated first characteristic change data is provided to the similarity identification unit 128.

[0032] The second feature change detection unit 125 uses the selected outside-vehicle video data from the data selection unit 123 to detect a change in features for each vehicle 102, and generates second feature change data that is feature change data indicative of the change in features. The second characteristic change detection unit 125 includes a vehicle-specific tracking unit 126 and a vehicle-specific characteristic change detection unit 127 .

[0033] The vehicle tracking unit 126 tracks each vehicle 102 in the image shown by the selected exterior image data. The vehicle-specific characteristic change detection unit 127 generates second characteristic change data indicating a change in the characteristics of the vehicle 102 for each vehicle 102 being tracked by the vehicle-specific tracking unit 126. The second characteristic change data for each vehicle 102 is provided to the similarity identification unit 128.

[0034] The similarity identification unit 128 compares the first characteristic change data with the second characteristic change data for each vehicle 102 in accordance with the comparison method from the first characteristic change detection unit 124. Then, the similarity identification unit 128 associates the vehicle 102 that generated the second characteristic change data that has the highest degree of similarity to the first characteristic change data with the vehicle 102 that generated the first characteristic change data.

[0035] For example, the similarity identifying unit 128 identifies a similarity indicating the degree of similarity between each of the plurality of second feature change data and the first feature change data. Then, the similarity identifying unit 128 identifies one second feature change data with the highest similarity, and determines that one vehicle 102 that is the source of the identified one second feature change data among the plurality of vehicles 102 matches the target vehicle. Specifically, the similarity identifying section 128 identifies the similarity based on the degree of correlation between the time series indicated by the first characteristic variation data and the time series indicated by each of the plurality of second characteristic variation data.

[0036] The above-mentioned processing in data analysis device 120 will be described in detail with reference to FIGS. Here, a case where the comparison is made based on the braking operation of the vehicle 102 will be described as an example. In such a case, the data selection unit 123 selects data related to the brake lamps from the vehicle data acquired by the vehicle data acquisition unit 121. Specifically, the data selection unit 123 selects, as the selected vehicle data, data indicating a brake pedal operation signal, a brake lamp control signal, a power supply voltage for the brake lamps, or a detection result of a brake lamp monitor sensor. From among these data, the data selection unit 123 selects one or more pieces of data that are unlikely to cause discrepancies with the actual lighting state of the brake lamps and are easy to compare.

[0037] Furthermore, the data selection unit 123 selects data capable of detecting the state of the brake lights of the vehicle 102 from the outside-vehicle video data acquired by the outside-vehicle video data acquisition unit 122. For example, the data selection unit 123 selects, as the selected outside-vehicle video data, the outside-vehicle video data including the video at the time when the vehicle data was acquired.

[0038] When data indicating a brake lamp control signal is selected as the selected vehicle data, the first feature change detection unit 124 can generate data indicating a feature change signal SS indicating changes in the brake lamps being on and off over time as the first feature change data, as shown in Fig. 4(A). The feature change signal SS changes when the brake lamps are on and off.

[0039] In such a case, the first feature change detection unit 124 can also generate data indicating a time and an event, such as "time t1: brake lights on" and "time t2: brake lights off", as the first feature change data.

[0040] 4(B) is a schematic diagram showing an image captured by the roadside camera 110B. The image here is captured as a moving image and can be handled as a time-series sequence image. A plurality of vehicles 102A, 102B, and 102C are captured in the image. Depending on the timing, some of the vehicles 102A and 102B have their brake lights on. The outside-vehicle image data acquisition unit 122 acquires image data showing such images as outside-vehicle image data. From only the image shown in FIG. 4(B), it is not possible to identify which of the vehicles 102A, 102B, and 102C has transmitted the data in FIG. 4(A).

[0041] Therefore, the vehicle tracking unit 126 of the second feature change detection unit 125 tracks each of the vehicles 102A to 102C by identifying image portions corresponding to each of the vehicles 102A to 102C in the image shown in the image data, which is the selected exterior vehicle image data.

[0042] 5A and 5B are schematic diagrams for explaining the process in which the vehicle tracking unit 126 tracks each vehicle 102 from within the video. FIG. 5(A) shows the vehicle tracking unit 126 tracking the vehicle 102A in the video. For example, when tracking a specific vehicle 102 in a video, the vehicle tracking unit 126 recognizes the vehicle 102 in one frame by image recognition, assigns an identifier to it, and tracks it by using continuous changes in at least one of the vehicle position, vehicle size, and vehicle shape and color in the image while comparing the identification results in the previous and next frames. Continuous changes indicate, for example, that there is no jump in the change or that the degree of change is the lowest.

[0043] 5B shows an example of tracking vehicle 102A using the continuity of vehicle positions in the video. Even if vehicle 102A, vehicle 102B, and vehicle 102C cannot be distinguished at the time of vehicle recognition, the vehicle 102 whose position changes most continuously can be tracked as vehicle 102A.

[0044] Returning to Fig. 3, the vehicle-specific characteristic change detection unit 127 generates second characteristic change data by identifying a change in characteristic, such as the turning on and off of the brake lights, for each time from the change in luminance of the image portion corresponding to the tracked vehicle 102. As a result, for example, as shown in Fig. 4(C), second characteristic change data indicating the characteristic change signal S1 of the vehicle 102A, second characteristic change data indicating the characteristic change signal S2 of the vehicle 102B, and second characteristic change data indicating the characteristic change signal S3 of the vehicle 102C are obtained.

[0045] 6(A) to 6(C) are schematic diagrams for explaining detection of the blinking state of the brake lights of a tracked vehicle 102A. 6(A), the vehicle-specific characteristic change detection unit 127 estimates the position of the brake lights from the shape of the vehicle 102 recognized by performing image recognition on the image portion of the vehicle 102. The lighting state of the brake lights can be estimated by detecting changes in brightness and color near the estimated positions of the brake lights.

[0046] 6B shows a luminance change of the red component of the image near the estimated position of the brake light. The luminance change detected from the image is affected by changes in the position and attitude of the vehicle 102, the influence of external light, the response speed of the image sensor, etc. For this reason, it is difficult to use the luminance change as it is as data indicating whether the brake light is on or off.

[0047] Therefore, the vehicle-specific feature change detection unit 127 estimates the blinking state of the brake lights using techniques such as filtering, edge detection, or pattern matching, and generates second feature change data indicating a feature change signal such as that shown in Figure 6(C), for example.

[0048] Note that the vehicle-specific characteristic change detection unit 127 can generate the second characteristic change data for lighting devices such as turn signals or headlights in addition to brake lights in a similar manner. Furthermore, when the brake lights cannot be confirmed from the video, the vehicle-specific characteristic change detection unit 127 can generate the second characteristic change data from a driving operation that does not directly cause a change in appearance, such as an accelerator operation. For example, the vehicle-specific characteristic change detection unit 127 can generate the second characteristic change data by detecting a change in the characteristic by estimating the speed or acceleration from a change in the vehicle position in the video. In this case, the first characteristic change data to be compared can be data from a vehicle speed sensor or data from an acceleration sensor.

[0049] 4(C), the brake lights of vehicle 102A are turned on in the first half of the time span. After that, the brake lights of vehicle 102B are turned on. Also, the brake lights of vehicle 102B are turned on for a longer period than the brake lights of vehicle 102A.

[0050] The similarity identification unit 128 then detects the signal that is most similar to the characteristic change signal SS shown in Fig. 4(A) from among the characteristic change signals S1 to S3 shown in Fig. 4(C), thereby making it possible to associate the vehicle 102 that has sent the control signal corresponding to the characteristic change signal SS with the vehicle 102 in the video. The details of this process will now be described.

[0051] As described above, the feature change signal SS shown in Fig. 4(A) is a brake lamp control signal acquired by a control system mounted on a certain vehicle 102. This is considered to be a change in the feature on the target vehicle side.

[0052] 4(C) are signals indicating the blinking state of the brake lights of each vehicle 102 estimated by analyzing the video captured by the roadside camera 110B. These are regarded as changes in the characteristics of each vehicle captured by the roadside camera 110B.

[0053] If it is the same vehicle 102, the timing at which the internal brake lamp control signal changes should match the timing at which the brake lamps are turned on and off as observed from the outside.

[0054] Therefore, the vehicle-specific characteristic change detection unit 127 compares each of the characteristic change signals S1 to S3 shown in Figure 4(C) with the characteristic change signal SS shown in Figure 4(A) and examines correlations, for example, between the time the light is on, the time the light is off, the duration of the lighted state, and the number of flashes.

[0055] Specifically, when the characteristic change signal S1 and the characteristic change signal SS are compared, the number of blinks matches, but the on time, off time, and duration of the on state do not match. When the characteristic change signal S2 is compared with the characteristic change signal SS, the number of times the signal is turned on, the time when the signal is turned on, the time for which the signal is turned off, and the duration of the time when the signal is turned on are the same. When comparing the characteristic change signal S3 with the characteristic change signal SS, no agreement is found in the changes in characteristics, including the number of times the signal is turned on. Therefore, the vehicle 102B that is the source of the characteristic change signal S2 that is most similar to the characteristic change signal SS can be identified as the transmission source of the data corresponding to FIG. 4(A).

[0056] By identifying the vehicle 102 that sent the data as one of the vehicles in the video captured by the roadside camera 110, it becomes possible to analyze data other than the brake lights used for identification as data from the identified vehicle 102 in conjunction with the video from the roadside camera 110. It also becomes possible to perform analysis related to multiple vehicles 102, including relationships with the movements of not only one vehicle 102 but also other vehicles 102.

[0057] For example, although not shown in the figure, if data indicating acceleration is sent from vehicle 102B, and it can be confirmed from the image from roadside camera 110 that vehicle 102A traveling in front of vehicle 102B changed lanes immediately before, the meaning and causal relationship can be analyzed, such as "vehicle 102B accelerated because the road ahead became clear due to the lane change of the preceding vehicle."

[0058] In the above explanation, the correlation of feature changes is detected between the feature change signal SS and each of the feature change signals S1 to S3. However, in reality, the changes do not necessarily occur at exactly the same timing due to differences in detection methods and delays due to transfer and data processing. Therefore, when making a comparison to detect a correlation, it is necessary to make the comparison while allowing a certain degree of deviation, taking into consideration the overall time lag, the time fluctuation of individual change points, noise, etc. Therefore, the vehicle-specific characteristic change detection unit 127 needs to increase the accuracy of detecting a correlation by comparing and checking the consistency not only at one change point, but at multiple change points.

[0059] Furthermore, if the degree of agreement at multiple change points improves by taking into account the overall time lag, this time lag represents the time lag between systems and between data, and by correcting this time lag, the data can be synchronized. For example, it is possible to correct the discrepancy between the time information of the vehicle-side system in FIG. 4(A) and the time stamp value of the roadside camera 110 in FIG. 4(B).

[0060] Next, the process in the similarity specifying unit 128 will be described. For example, when the first feature change data indicates the feature change signal SS shown in FIG. 4(A), the similarity identification unit 128 detects the second feature change data indicating the feature change signals S1 to S3 shown in FIG. 4(C) that has the closest feature change.

[0061] Specifically, the similarity identification unit 128 selects one signal from the multiple feature change signals S1 to S3 and compares the selected signal with the feature change signal indicated by the first feature change data. At this time, in order to take delays, fluctuations, etc. into account, the similarity identification unit 128 compares the similarities while shifting the selected signal in the time direction, finds the point in time when the similarity is highest, and uses the similarity at this time as the evaluation value.

[0062] 7(A) to 7(C) are schematic diagrams for explaining a process of comparing the similarity while shifting the selected signal in the time direction. Fig. 7(A) shows the feature change signal SS indicated by the first feature change data, similarly to Fig. 4(A). Here, the feature change signal is a signal indicating the blinking of the brake lights, indicated by a brake light control signal sent from a certain vehicle 102. Here, it is assumed that the comparison method instructed by the data selection unit 123 indicates that the similarity is evaluated using the timing at which the brake lights are turned on and the timing at which they are turned off.

[0063] FIG. 7B shows an example in which the characteristic change signal S1 of the vehicle 102A, which is indicated by the second characteristic change data in FIG. 4C, is shifted in the time direction. As shown in Fig. 7(B), the similarity identifying unit 128 calculates the similarity with the feature change signal SS while shifting the feature change signal S1 like feature change signals S1#1 to S1#3. Here, the similarity identifying unit 128 compares the timings of turning on the lights and the timings of turning off the lights, and determines the similarity by, for example, taking the mean square of the shifts. In this case, the smaller the mean square value, the stronger the correlation and the higher the similarity. For example, when the feature change signal S1#2 has the smallest value, the value at that time is detected as the similarity of the vehicle 102A.

[0064] FIG. 7C shows an example in which the characteristic change signal S2 of the vehicle 102B, which is indicated by the second characteristic change data in FIG. 4C, is shifted in the time direction. As shown in Fig. 7(C), the similarity identifying unit 128 calculates the similarity with the feature change signal SS while shifting the feature change signal S2 like feature change signals S2#1 to S2#2. Here too, the similarity identifying unit 128 compares the timings of turning on the lights and the timings of turning off the lights, and determines the similarity by, for example, taking the mean square of each shift. In this case, the smaller the mean square value, the higher the similarity. For example, when the feature change signal S2 has the smallest value, the value at that time is detected as the similarity of vehicle 102B.

[0065] As described above, the similarity identifying unit 128 compares and evaluates the changes in the characteristics of all the vehicles 102A to 102C included in the video shown in the video data, i.e., the blinking state of the brake lights, with the brake light control signal sent from any of the vehicles 102A to 102C. Then, the similarity identifying unit 128 can determine that the vehicle 102B, which has the highest similarity among the vehicles 102A to 102C, has sent the brake light control signal. FIG. 8 is a flowchart showing an example of the process performed by the similarity specifying section 128. In this manner, it is possible to identify the vehicle corresponding to the data transmission source from among multiple vehicles 102 in the video.

[0066] In the above example, the comparison method for calculating the similarity by comparing the timing of the change in the brake lights has been described, but the present embodiment is not limited to such a comparison method. For example, time series classification using machine learning, pattern similarity, or the like may be used. Also, although the above example uses only brake lights, it is also possible to use lighting devices such as turn signals or headlights, speed, acceleration, steering operation, or the like.

[0067] The data analysis apparatus 120 described above can be realized, for example, by a computer 10 having a first communication I / F 11, a second communication I / F 12, an auxiliary storage device 13, a main memory 14, and a processor 15, as shown in FIG. 9.

[0068] The first communication I / F 11 is an interface that performs communication wirelessly. The second communication I / F 12 is an interface that performs communication wirelessly or via a wire. The auxiliary storage device 13 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The main memory 14 is a volatile memory. The processor 15 is a control circuit such as a CPU (Central Processing Unit).

[0069] For example, the vehicle data acquisition unit 121 can be realized by the first communication I / F 11. The outside-vehicle image data acquisition unit 122 can be realized by the second communication I / F 12. The data selection unit 123, the first feature change detection unit 124, the second feature change detection unit 125 and the similarity identification unit 128 can be realized by reading a program stored in the auxiliary storage device 13 into the main memory 14 and having the processor 15 execute that processor. Such a program may be provided via a network, or may be provided in a form recorded on a recording medium, i.e., as a program product.

[0070] In the above embodiment, an example using brake lights has been described, but the present embodiment is not limited to such an example. For example, the vehicle 102 can be identified using other data such as other lighting devices, speed, acceleration, or sound. Below, combinations of data that can be used to identify the vehicle 102 are listed as modified examples. It is not necessary to use all of the data listed here at the same time. Only one of the multiple data may be used. Also, multiple data may be used in combination.

[0071] Variation 1. Using changes in the on and off of lighting devices such as brake lights, turn signals, width lights, or headlights, the vehicle 102 that sent the vehicle data can be identified from the multiple vehicles 102 contained in the exterior video data.

[0072] In this case, the vehicle data may be data that can specify whether the lighting device provided in the target vehicle is on or off. For example, the vehicle data may be data indicating a light control signal for controlling the blinking of the lighting device, or data indicating a light monitoring sensor signal for monitoring the blinking of the lighting device. The light control signal includes a signal indicating a voltage or current for controlling the blinking of the lighting device.

[0073] The first characteristic change detection unit 124 generates the first characteristic change data indicating a time series of turning on and off the lighting device from the vehicle data.

[0074] The second feature change detection unit 125 analyzes the video represented by the selected outside-vehicle video data to generate a plurality of second feature change data indicating a time series of turning on and off the lighting devices of each of the plurality of vehicles 102. Specifically, the second feature change detection unit 125 can estimate the installation positions of the lighting devices from the video represented by the outside-vehicle video data, and detect changes in color and brightness in the vicinity, thereby generating second feature change data indicating the feature change signal.

[0075] In this case, similarities can be identified by comparing the types of lighting devices, the left / right direction of turn signals, the timing of lighting, and the timing of lighting / extinguishing.

[0076] Variation 2. Changes in the movement or position of the vehicle 102, such as the speed, acceleration, angular acceleration, direction, lane changes, right or left turns, sudden braking or collisions, can be used to identify the vehicle 102 that sent the vehicle data from among multiple vehicles 102 contained in the exterior video data.

[0077] In this case, the vehicle data may be any data that can identify the position or movement of the target vehicle. For example, the vehicle data may be a speed sensor value, an acceleration sensor value, an angular acceleration sensor value, an azimuth sensor value, a longitude / latitude / altitude value, an accelerator opening, a brake pedal pressure value, or data indicating a steering operation amount.

[0078] The first feature change detection unit 124 generates the first feature change data indicating a time series of the position or movement of the target vehicle from the vehicle data. The second characteristic change detection unit 125 generates a plurality of second characteristic change data indicating a time series of the positions or movements of each of the plurality of vehicles by analyzing the video represented by the selected outside-vehicle video data.

[0079] When the vehicle data is speed sensor values ​​or acceleration sensor values, similarity can be identified by calculating the strength of correlation between the speed data in which the speed sensor values ​​are arranged in chronological order or the acceleration data in which the acceleration sensor values ​​are arranged in chronological order and the speed data or acceleration data identified from the outside vehicle video data.

[0080] In addition, since acceleration data is correlated with the vehicle's accelerator or brake operation, similarities can be identified by comparing the timing or duration of the accelerator or brake operation identified in the vehicle data with the timing or duration of the accelerator or brake operation identified from the outside-vehicle image data.

[0081] Additionally, similarities can be detected by comparing changes in the orientation or movement of the vehicle 102 in the video with changes in steering indicated in the vehicle data. In addition, similarities can be identified by comparing the change in the direction of the vehicle 102 in the video with the change in the direction of the vehicle 102 indicated by the angular acceleration sensor value or azimuth angle sensor value, which are vehicle data.

[0082] Variation 3. The vehicle 102 that transmitted the vehicle data can be identified from multiple vehicles 102 included in the exterior video data by using changes in the positional relationship or distance between the vehicle 102 in front of the vehicle 102, the vehicle 102 behind the vehicle 102, or the vehicle 102 to the side of the vehicle 102, changes in the type or color of the vehicle 102, the vehicle 102 behind the vehicle 102, or the vehicle 102 to the side of the vehicle 102, or changes in the positional relationship or distance between the vehicle 102 and landmarks on the road around the vehicle or landmarks near the road.

[0083] In this case, the vehicle data may be data that can specify the distance or positional relationship between the target vehicle and an object (e.g., a vehicle or a target object) around the target vehicle. For example, the vehicle data may be data indicating the detection result of a radar (LADAR) mounted on the vehicle 102, the detection result of a lidar (LiDAR) mounted on the vehicle 102, an image captured by a stereo camera mounted on the vehicle 102, an image captured by a camera mounted on the vehicle 102, a distance estimated from the detection result or the image, and an image recognition result recognized from the image.

[0084] The first feature change detection unit 124 generates, from the vehicle data, first feature change data indicating a time series of the distance or positional relationship between the target vehicle and an object. The second feature change detection unit 125 generates second feature change data indicating a time series of the distance or positional relationship between each of the multiple vehicles and an object by analyzing the image indicated by the selected outside-vehicle image data. For example, the second feature change detection unit 125 can analyze the image shown in the exterior video data to detect vehicles 102, detect the distance between vehicles 102, detect objects around the road, estimate the positions of objects around the road, estimate the distance between the objects around the road and the vehicle 102, or detect the type or color of the vehicle, thereby identifying a change in the positional relationship or distance to a vehicle in front, behind, or to a side vehicle, a change in the type or color of the vehicle in front, behind, or to a side vehicle, or a change in the positional relationship or distance to landmarks on the road around the vehicle 102 or landmarks near the road.

[0085] In this case, similarity can be identified by comparing the timing of the change identified from the vehicle data with the timing of the change identified from the outside-of-vehicle video data.

[0086] Variation 4. By comparing the timing of changes in the content of the exterior display device that displays messages, etc. from the vehicle 102 to the outside of the vehicle, it is possible to identify the vehicle 102 that has sent the vehicle data from among the multiple vehicles 102 included in the exterior video data.

[0087] In this case, the vehicle data may be data capable of identifying the display content of an exterior display device, which is a device provided in the target vehicle and displays information to the outside of the vehicle. For example, the vehicle data may be control data for controlling an exterior display device.

[0088] The first characteristic change detection unit 124 generates first characteristic change data indicating a time series of changes in the display content from the vehicle data. The second feature change detection unit 125 analyzes the image shown in the selected exterior vehicle image data, and when any of the multiple vehicles 102 is equipped with an exterior vehicle display device, generates second feature change data indicating a time series of changes in the display content of the exterior vehicle display device included in the image.

[0089] In such a case, similarities can be identified by comparing the timing of changes in the content displayed on the exterior display device.

[0090] Variation 5. By using the opening and closing of the doors of the vehicle 102, the vehicle 102 that has transmitted the vehicle data can be identified from among the multiple vehicles 102 included in the exterior image data.

[0091] In this case, the vehicle data may be data capable of identifying whether the doors of the target vehicle are open or closed. For example, the vehicle data may be data indicating the detection results of a door sensor, an acoustic sensor, or an acceleration sensor and an angular acceleration sensor, or data indicating an image captured by a camera mounted on the vehicle 102. Specifically, when data indicating the detection result of the door sensor is available, it is possible to identify whether the door is open or closed based on the change in the value of the door sensor. Even when data indicating the detection result of the door sensor is not available, it is possible to identify the door opening or closing status by analyzing the video captured by the cabin side camera of the vehicle 102. Furthermore, it is also possible to estimate the timing of the door opening and closing by analyzing the detection result of the acoustic sensor or angular velocity sensor.

[0092] The first characteristic change detection unit 124 generates first characteristic change data indicating a time series of opening and closing of the doors of the target vehicle from the vehicle data.

[0093] The second characteristic change detection unit 125 generates second characteristic change data indicating a time series of the opening and closing of the doors of each of the multiple vehicles 102 by analyzing the video represented by the selected outside-vehicle video data. For example, the second feature change detection unit 125 can determine whether the door is open or closed by analyzing exterior video data from a surveillance camera installed in the parking lot, or by analyzing acoustic sensor data from a microphone installed in the parking lot.

[0094] Here, a case where cooperation with a surveillance camera in a parking lot is performed will be described. 10A to 10C are schematic diagrams for explaining a process for identifying a vehicle 102 using the timing of a change in the opening and closing of a door.

[0095] FIG. 10A shows a characteristic change signal SS# indicating the result of detection of door opening / closing by a door sensor mounted on the vehicle 102. The opening and closing of a door can be directly detected by a door sensor, so when the detection result of the door sensor is available, the characteristic change signal SS# shown in Fig. 10(A) can be obtained from the detection result of the door sensor.

[0096] 10B shows an image captured by a surveillance camera in a parking lot. Image Im2 is a frame taken at time t1, image Im3 is a frame taken at time t2, and image Im4 is a frame taken at time t3. By analyzing the images Im1 to Im3 as described above, the second feature change detection unit 125 can estimate the timing of the door opening and closing. Here, the second feature change detection unit 125 may estimate the timing of the door opening and closing by machine learning using the changes in the image of the vehicle 102 and the movement of the person as input.

[0097] FIG. 10C shows a feature change signal S4 acquired from the opening and closing of the door estimated as described above.

[0098] In such a case, similarity can be detected by comparing the timing of door opening and closing.

[0099] There are cases where data from the door sensor of the vehicle 102 cannot be used directly. For example, this is the case when data is obtained from a drive recorder that is installed later. In such cases, it is also possible to detect that the door is closed from signals from an acoustic sensor, an acceleration sensor, an angular acceleration sensor, or the like. If the drive recorder has an in-vehicle camera, it is also possible to detect whether the door is open or closed from the in-vehicle camera image.

[0100] In FIG. 10(A), signal S5 indicates the value detected by the acoustic sensor when the door is closed, and signal S6 indicates the value detected by the acceleration sensor when the door is closed. As shown in Fig. 10(A), when an acoustic sensor or an acceleration sensor is used, the sensor response is processed to detect the response when the door is opened or closed, and the detection is delayed from the actual opening or closing of the door. However, by measuring the amount of deviation between the opening or closing of the door and the detection by the sensor in advance, the amount of deviation can be corrected.

[0101] As described above, according to the present embodiment, by using the situations and data that occur daily in the vehicle 102 during normal operation, it is possible to link data acquired outside the vehicle 102 with data acquired inside the vehicle 102. This makes it possible to identify the vehicle 102 and link the data without the need for a special system or the creation of calibration information in advance. [Explanation of symbols]

[0102] 100 data analysis system, 110 roadside camera, 120 data analysis device, 121 vehicle data acquisition unit, 122 outside vehicle image data acquisition unit, 123 data selection unit, 124 first feature change detection unit, 125 second feature change detection unit, 126 vehicle-specific tracking unit, 127 vehicle-specific feature change detection unit, 128 similarity identification unit.

Claims

1. A vehicle data acquisition unit that can indicate characteristics of a target vehicle that is a vehicle to be detected and acquires vehicle data that is data acquired inside the target vehicle; an outside-vehicle image data acquisition unit that acquires outside-vehicle image data that is image data showing an image including the plurality of vehicles and is captured from the outside of the plurality of vehicles; a first feature change detection unit that generates first feature change data indicating a change in the feature of the target vehicle from the vehicle data; a second feature change detection unit that identifies a change in the feature of each of the plurality of vehicles from the outside-vehicle image data and generates a plurality of second feature change data each indicating a change in the feature of one of the plurality of vehicles; a similarity determination unit that determines a similarity indicating a degree of similarity between each of the plurality of second feature change data and the first feature change data, identifies one second feature change data having the highest similarity, and determines one vehicle among the plurality of vehicles that is a source of generation of the one second feature change data to match the target vehicle; The vehicle data is data that can identify the display content of an exterior display device that is provided in the target vehicle and that displays information to the outside of the vehicle, the first feature change detection unit generates the first feature change data indicating a time series of changes in the display content from the vehicle data; the second feature change detection unit generates, by analyzing the video, the second feature change data indicating a time series of changes in the display content of the exterior display device included in the video, when any of the plurality of vehicles is equipped with an exterior display device; the similarity identifying unit identifies the similarity based on a degree of correlation between the time series indicated by the first feature variation data and the time series indicated by each of the plurality of second feature variation data. An information processing device comprising:

2. Computer, a vehicle data acquisition unit that acquires vehicle data, which is data acquired inside the target vehicle and indicates characteristics of the target vehicle that is a vehicle to be detected; an outside-vehicle image data acquisition unit that acquires outside-vehicle image data that is image data showing an image including the plurality of vehicles and is captured from the outside of the plurality of vehicles; a first feature change detection unit that generates first feature change data indicating a change in the feature of the target vehicle from the vehicle data; a second feature change detection unit that identifies a change in the feature of each of the plurality of vehicles from the outside-vehicle image data and generates a plurality of second feature change data each indicating a change in the feature of one of the plurality of vehicles; and a similarity determination unit that determines a degree of similarity between each of the plurality of second feature change data and the first feature change data, identifies one second feature change data with the highest similarity, and determines one vehicle among the plurality of vehicles that is a source of generation of the one second feature change data to match the target vehicle; The vehicle data is data that can identify the display content of an exterior display device that is provided in the target vehicle and that displays information to the outside of the vehicle, the first feature change detection unit generates the first feature change data indicating a time series of changes in the display content from the vehicle data; the second feature change detection unit generates, by analyzing the video, the second feature change data indicating a time series of changes in the display content of the exterior display device included in the video, when any of the plurality of vehicles is equipped with an exterior display device; the similarity identifying unit identifies the similarity based on a degree of correlation between the time series indicated by the first feature variation data and the time series indicated by each of the plurality of second feature variation data. A program characterized by.

3. acquiring vehicle data, the vehicle being a target vehicle and capable of indicating characteristics of the target vehicle, the vehicle being a target of detection, the vehicle data being data acquired inside the target vehicle; Acquire outside-vehicle image data, which is image data showing an image including the plurality of vehicles and is captured from outside the plurality of vehicles; generating first characteristic change data indicative of a change in the characteristic of the target vehicle from the vehicle data; Identifying a change in the characteristic of each of the plurality of vehicles from the outside image data, and generating a plurality of second characteristic change data, each of which indicates a change in the characteristic of one of the plurality of vehicles; identifying a similarity indicating a degree of similarity between each of the plurality of second feature variation data and the first feature variation data; Identifying one second feature change data having the highest similarity; An information processing method for determining that one vehicle, which is a source of generation of the one second feature change data, among the plurality of vehicles, matches the target vehicle, comprising: The vehicle data is data that can identify the display content of an exterior display device that is provided in the target vehicle and that displays information to the outside of the vehicle, the first feature change data indicating a time series of changes in the display content is generated from the vehicle data; By analyzing the video, when any of the plurality of vehicles is equipped with an exterior display device, the second feature change data indicating a time series of changes in the display content of the exterior display device included in the video is generated, The similarity is specified based on a degree of correlation between the time series indicated by the first feature change data and the time series indicated by each of the plurality of second feature change data. An information processing method comprising:

Images