Video data collection and analysis device

Abstract

An image data collection and analysis device (1) according to the present technology includes a sensor value acquisition unit (13) that acquires a sensor value of a vehicle from a sensor mounted on the vehicle, an image acquisition unit (12) that acquires an outside vehicle image that captures an outside of the vehicle, an outside vehicle communication unit (11) that communicates with an external server, an abnormality degree determination unit (15) that determines an abnormality degree of a road on which the vehicle travels, based on the sensor value acquired by the sensor value acquisition unit, an analysis priority determination unit (16) that determines a priority in analyzing the outside vehicle image, based on the determined abnormality degree, and an image analysis unit (14) that analyzes a road surface state of a road reflected in the outside vehicle image, based on the priority.

Description

TECHNICAL FIELD

[0001] The present technology relates to an image data collection and analysis device. BACKGROUND

[0002] A drive recorder of related art acquires a driving image from a camera and saves the acquired image to a memory such as an SD. Since the storage capacity of the memory has an upper limit, when recording is performed in a case where the storage capacity is exceeded, the drive recorder secures the storage capacity by deleting old information.

[0003] However, recently, there is a study to detect road surface deterioration and road abnormalities and the like from a driving image captured with a drive recorder, and to upload the detection information to an external server using a communication function. In this technical field, for example, in Patent Literature 1, a technology is disclosed in which image data is uploaded by centralized uploading of image data that contributes to updating of a database generated by a center server, thereby saving the communication amount and the frequency band used in the communication.

[0004] Related Art Documents

[0005] Patent Literature

[0006] Patent Literature 1

[0007] Japanese Patent Application Laid-Open No. 2018-198004 SUMMARY

[0008] Problem to be Solved by the Invention

[0009] In the related art exemplified in Patent Literature 1, if the processing capacity of the image analysis section is sufficient, it is possible to continuously analyze the driving image, but if the processing capacity of the image analysis section is not sufficient, it can be a situation that cannot be caught up with. If this state continues, the storage capacity of the memory becomes tight and old information is deleted. Among the old information that is deleted, there are sometimes important data.

[0010] In view of the above problems, an object of the present technology is to provide an image data collection and analysis device that does not delete important data even if information deletion for securing the storage capacity is to be performed.

[0011] Technical Means for Solving the Problem

[0012] The image data collection and analysis apparatus according to the present technology includes a sensor value acquisition unit that acquires a sensor value of a vehicle from a sensor mounted on the vehicle, an image acquisition unit that acquires an outside vehicle image that captures an outside of the vehicle, an outside vehicle communication unit that communicates with an external server, an abnormality degree determination unit that determines an abnormality degree of a road on which the vehicle travels, based on the sensor value acquired by the sensor value acquisition unit, an analysis priority determination unit that determines a priority in analyzing the outside vehicle image, based on the determined abnormality degree, and an image analysis unit that analyzes a road surface state of a road reflected in the outside vehicle image, based on the priority.

[0013] Effects of Invention

[0014] According to the image data collection and analysis apparatus according to the present technology, the priority order of the images to be analyzed is determined based on the features of the road abnormality determined from the sensor information acquired other than the camera image and the server information, and the analysis is performed in the priority order from high to low. As a result, the image data collection and analysis apparatus according to the present technology prevents important image data from being deleted even when the processing of the image analysis unit is not caught up. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a diagram that schematically shows a configuration example of an image data collection and analysis system including the image data collection and analysis apparatus according to the present technology.

[0016] Figure 2 is a block diagram that shows the functions of the image data collection and analysis apparatus according to Embodiment 1.

[0017] Figure 3 is a flowchart that shows the processing flow of the image data collection and analysis apparatus according to Embodiment 1.

[0018] Figure 4 is a partial flowchart that shows the processing inside the step (ST3) of determining the analysis priority in the flowchart of Figure 3

[0019] FIG. 5 is a diagram that schematically shows an example of a road abnormality assumed by the image data collection and analysis apparatus according to the present technology. Figure 5A is a diagram that schematically shows a time series signal of an acceleration sensor corresponding to an abnormality in a road. Figure 5B is an example of a table that associates the types of the abnormality in the road with the analysis results of various sensors.

[0020] Figure 6 is a diagram that schematically shows an example of a road abnormality assumed by the image data collection and analysis apparatus according to the present technology. Figure 3 ​a part of the flow of the internal processing of the step of analyzing the image data (ST4) in the flowchart. DETAILED DESCRIPTION

[0021] The image data collection and analysis apparatus 1 according to the disclosed technology will be made clear through the following description of the drawings.

[0022] Embodiment 1

[0023] Figure 1 is a diagram showing an outline of the structure of an image data collection and analysis system equipped with the image data collection and analysis apparatus 1 according to the disclosed technology. As shown in Figure 1 , the image data collection and analysis system is composed of the image data collection and analysis apparatus 1 mounted in a vehicle, the camera 2 mounted in the vehicle and capturing the outside of the vehicle, and the sensor 3 mounted in the vehicle and detecting various states.

[0024] Figure 1 The camera 2 in the image data collection and analysis system is mounted on the vehicle and captures the outside of the vehicle, such as the front, the rear, and the left and right. The image captured of the outside of the vehicle is hereinafter referred to as "outside image". Figure 1 The sensor 3 in the image data collection and analysis system is mounted on the vehicle and can be a sensor that detects the state of the vehicle, a sensor that detects the surroundings of the vehicle, and other sensors for obtaining the time, etc. As the sensor that detects the state of the vehicle, an accelerometer, a gyroscope, a microphone, a wheel speed meter, etc. can be given. As the sensor that detects the surroundings of the vehicle, an angle sensor, a LiDAR, a laser displacement meter, etc. can be given. As the other sensors, a GNSS, etc. can be given.

[0025] Figure 2 is a block diagram showing the functions of the image data collection and analysis apparatus 1 according to Embodiment 1. As shown in Figure 2 , the image data collection and analysis apparatus 1 includes a memory 10, an outside communication section 11, an image acquisition section 12, a sensor value acquisition section 13, an image analysis section 14, an abnormality degree determination section 15, an analysis priority decision section 16, and a server information acquisition section 17.

[0026] The memory 10 stores the data acquired from the camera 2 and the sensor 3. The outside communication section 11 communicates various information with an external server through a public line and wireless communication such as WiFi. More specifically, the outside communication section 11 transmits the analysis result sent from the image analysis section 14 to the external server. In addition, the outside communication section 11 transmits the server information sent from the external server to the server information acquisition section 17.

[0027] The image acquisition section 12 acquires the image data of the outside image sent from the camera 2 and sends it to the image analysis section 14. The sensor value acquisition section 13 acquires the sensor data output from the sensor 3 and outputs it to the abnormality degree determination section 15.

[0028] The image analysis section 14 analyzes the image data sent from the image acquisition section 12, and determines the degree of abnormality of the road. The image analysis section 14 processes in order of the analysis priority decided by the analysis priority decision section 16. The unit of the image data analyzed by the image analysis section 14 can be the unit of the image frame, the unit of time, or a structure that can be set in advance by the user.

[0029] The degree of abnormality decision section 15 performs an arithmetic process on the sensor data output from the sensor value acquisition section 13, and determines the degree of abnormality of the road corresponding to the sensor data. As the arithmetic process on the sensor data, there can be mentioned comparison with a threshold value, calculation of an average value, calculation of a peak value, calculation of a spectral frequency, and the like. The unit of the image data mentioned above can also be decided in accordance with the kind of the arithmetic process on the sensor data. The information of the degree of abnormality determined by the degree of abnormality decision section 15 is output to the analysis priority decision section 16.

[0030] The analysis priority decision section 16 decides the analysis priority of each image on the basis of the information of the degree of abnormality output from the degree of abnormality decision section 15 and the server information output from the server information acquisition section 17. The decided analysis priority of each image is output to the image analysis section 14. The decision of the analysis priority performed by the analysis priority decision section 16 can be performed in the same unit as the unit of the image data analyzed by the image analysis section 14, and can be the unit of the image frame or the unit of time.

[0031] The server information acquisition section 17 outputs the server information sent from the vehicle exterior communication section 11 to the analysis priority decision section 16. The server information here contains information on which road abnormality is to be analyzed preferentially.

[0032] Figure 3 is a flowchart showing the processing flow of the image data collection and analysis apparatus 1 according to Embodiment 1. As shown in Figure 3 , in the processing of the image data collection and analysis apparatus 1, there are a step of acquiring a sensor value (ST1), a step of analyzing the sensor value (ST2), a step of deciding an analysis priority (ST3), and a step of analyzing image data (ST4).

[0033] The step of acquiring a sensor value (ST1) is a processing step of acquiring the sensor data output from the sensor 3, i.e., sampling, performed by the sensor value acquisition section 13.

[0034] The step of analyzing the sensor value (ST2) is performed by the degree of abnormality decision section 15, and determines the degree of abnormality occurring on the road. Here, the abnormality occurring on the road can be assumed to be a pothole, a crack, a rut (wheel mark) depression, a fallen object, an obstacle, a person, an animal, or the like.

[0035] The step of determining the analysis priority (ST3) is implemented by the analysis priority determination section 16. The details of the step of determining the analysis priority (ST3) will be made clear through the following description.

[0036] The step of analyzing the image data (ST4) is implemented by the image analysis section 14. The details of the step of analyzing the image data (ST4) will also be made clear through the following description.

[0037] Figure 4 is a partial flowchart of the internal processing of the step of determining the analysis priority (ST3) in the flowchart of Figure 3 As shown in Figure 4 , the step of determining the analysis priority (ST3) has: a step of determining whether an abnormality is detected (ST301); a step of acquiring position information and a time of the detected abnormality (ST302); a step of acquiring a feature of the abnormality (ST303); a step of acquiring server information (ST304); a step of determining the analysis priority based on the server information and the feature of the abnormality (ST305); and a step of setting the analysis priority to the lowest (ST306).

[0038] The step of determining whether an abnormality is detected (ST301) is the first step in the step of determining the analysis priority (ST3) and is implemented by the analysis priority determination section 16. The processing here is based on the information of the abnormality degree output from the abnormality degree determination section 15. If the abnormality degree is lower than a predetermined value, the processing proceeds to the step of setting the analysis priority to the lowest (ST306). In other cases, the processing proceeds to the step of acquiring position information and a time of the detected abnormality (ST302).

[0039] The step of acquiring position information and a time of the detected abnormality (ST302) is an internal step of the step of determining the analysis priority (ST3) and is implemented by the analysis priority determination section 16. The processing here is processing of acquiring the time and the position information when the abnormality degree exceeds the predetermined value. The image data collection analysis device 1 according to the present technology can also consider that the subject of the image of the camera 2 is the image of the road surface in front of the vehicle on which the device is mounted. That is, the analysis priority determination section 16 can also consider a delay time between the time when the abnormality is detected and the time when the image is captured by the associated camera 2.

[0040] The step of acquiring the feature of the anomaly (ST303) is an internal step of the step of determining the analysis priority (ST3), and is implemented by the analysis priority determination unit 16. The processing here is processing of acquiring the feature of the anomaly when the degree of anomaly exceeds a predetermined value. FIG. 5 is a schematic view showing an example of a road anomaly assumed by the image data collection analysis apparatus 1 according to the present technology. The step of acquiring the feature of the anomaly (ST303) is clarified by the following description of FIG. 5.

[0041] As shown in FIG. 5, among the road anomalies assumed by the image data collection analysis apparatus 1 according to the present technology, a pothole, a large crack, a small crack, a large obstacle, a small obstacle, and the like can be cited. Figure 5A is a schematic view showing a time series signal of an acceleration sensor corresponding to an anomaly in a road.

[0042] Figure 5A It is shown that, for example, a pothole and a crack can be detected by analyzing a time series signal of an acceleration sensor. In addition, Figure 5B It is shown that, for example, a pothole and a crack can be detected by analyzing a time series signal of an acceleration sensor. In addition,

[0043] Suppose that the feature quantity obtained from the sensor data of the plurality of sensors 3 is N kinds (N is an integer of 1 or more). Then, the road anomaly can be classified by kind and plotted in an N-dimensional feature quantity space. Here, as reference data, it is also preferable to classify and plot in the N-dimensional feature quantity space in the case of a normal road. The plotted information in the N-dimensional feature quantity space obtained from the past stored data can be adapted as a feature quantity space map classified by kind of road anomaly. The feature quantity space map can derive the kind of road anomaly to which the plot belongs with a higher likelihood from the plot of the feature quantity. The image data collection analysis apparatus 1 according to the present technology can also have a feature quantity space map generated offline. In addition, the image data collection analysis apparatus 1 according to the present technology can have a learning function, and can have a structure that updates the feature quantity space map by learning. Furthermore, the method of deriving the kind of road anomaly with a higher likelihood from the plot of the feature quantity can also use a decision tree.

[0044] As shown in FIG. 5, among the road anomalies assumed by the image data collection analysis apparatus 1 according to the present technology, a pothole, a large crack, a small crack, a large obstacle, a small obstacle, and the like can be cited. Figure 5B As shown in FIG. 5, among the road anomalies assumed by the image data collection analysis apparatus 1 according to the present technology, a pothole, a large crack, a small crack, a large obstacle, a small obstacle, and the like can be cited.

[0045] The step of acquiring server information (ST304) is an internal step of the step of deciding the analysis priority (ST3), and is implemented by the analysis priority decision unit 16. The processing here is processing of acquiring the server information sent from the server information acquisition unit 17. An example of the server information is an indication of the degree to which the server side desires to acquire the image data for each type of road anomaly. For example, the degree to which the server side desires to acquire the image data (hereinafter referred to as "request degree") can be indicated by a value of 0 or more and 1 or less. In the example here, the higher the priority, the closer the value is to 1, and the lower the priority, the closer the value is to 0.

[0046] The step of deciding the analysis priority based on the server information and the characteristics of the anomaly (ST305) is an internal step of the step of deciding the analysis priority (ST3), and is implemented by the analysis priority decision unit 16. The processing here is processing of deciding the analysis priority of the final image data. The information used here is the request degree indicated by a value of 0 or more and 1 or less acquired in the step of acquiring server information (ST304). The analysis priority can be used directly as the request degree. Alternatively, the request degree and the severity and reliability shown in the flowchart of Fig. 6 can be used together to decide the analysis priority. The severity and reliability can also be indicated by values of 0 or more and 1 or less, and the analysis priority of the final image data can be calculated by weighting and multiplying the three values. The weights at this time can also be set as appropriate by the user. Figure 5B

[0047] Figure 6 is a partial flowchart of the internal processing of the step of analyzing the image data (ST4) in the flowchart of Figure 3 As shown in Fig. 6, the step of analyzing the video data (ST4) has a step of acquiring the camera images (ST401), a step of selecting the image with the highest analysis priority (ST402), a step of determining whether the analysis priority is above a threshold value (ST403), a step of analyzing the image (ST404), and a step of sending the analysis result (ST405). Figure 6 The step of acquiring the camera images (ST401) is an internal step of the step of analyzing the image data (ST4), and is implemented by the image analysis unit 14. The processing here is processing of acquiring the image data from the camera 2 mounted on the vehicle.

[0048] The step of selecting the image with the highest analysis priority (ST402) is an internal step of the step of analyzing the image data (ST4), and is implemented by the image analysis unit 14. The processing here is processing of selecting the image with the highest priority among the analysis priorities decided by the analysis priority decision unit 16 as the object of analysis.

[0049] ​​

[0050] The step of determining whether the analysis priority is above the threshold (ST403) is an internal step of the step of analyzing the image data (ST4), and is implemented by the image analysis section 14. The processing here is processing that judges according to a certain condition, and the flow diverges depending on the result of the judgment. The condition here is that the analysis priority of the image selected as the analysis target is above the threshold decided in advance. If the analysis priority is above the threshold, the processing proceeds to the step of analyzing the image (ST404). In the case where the condition is not satisfied, the step of analyzing the image data of the image (ST4) ends.

[0051] In the step of determining whether the analysis priority is above the threshold (ST403), the threshold for the condition is considered and set as follows. The threshold is set to 0, which is synonymous with not setting the threshold, and all images become the analysis target, and are analyzed in order of the analysis priority. When the threshold is close to 1, the number of images analyzed in the image analysis section 14 decreases, so the load of the analysis can be reduced.

[0052] The step of analyzing the image (ST404) is an internal step of the step of analyzing the image data (ST4) that becomes the core, and is implemented by the image analysis section 14. The processing here is processing that implements the analysis of the image. More specifically, in the step of analyzing the image data (ST4), the road surface state of the road imaged in the vehicle exterior image is analyzed by image analysis. The image analysis section 14 can also use machine learning in the image analysis.

[0053] The step of transmitting the analysis result (ST405) is an internal step of the step of analyzing the image data (ST4), and is implemented by the image analysis section 14. The processing here is processing that receives the analysis result in the step of analyzing the image (ST404), and outputs the analysis result and the image data to the vehicle exterior communication section 11 as needed. The analysis result and the image data output are uploaded to the external server.

[0054] As described above, since the image data collection and analysis apparatus 1 according to Embodiment 1 has the above-described structure, even in the case where the processing of the image analysis section 14 falls behind, it is possible to prevent important image data from being deleted.

[0055] Embodiment 2.

[0056] The image data collection and analysis apparatus 1 according to Embodiment 1 is a mode in which it is used in one vehicle, but is not limited thereto. The image data collection and analysis apparatus 1 according to Embodiment 2 can be used in a plurality of vehicles.

[0057] A system that achieves one purpose in a plurality of vehicles can be referred to as a connected car system. The image data collection and analysis apparatus 1 according to Embodiment 2 can be considered as one example of the connected car system.

[0058] In the networked automobile system, a plurality of vehicles each have the image data collection analysis device 1 according to Embodiment 2. The image data collection analysis device 1 according to Embodiment 2 has a structure that enables sharing of the results of the image analysis, the results of the abnormality degree determination, and the determination results of the analysis priority, which are calculated by each image data collection analysis device 1, via the vehicle-external communication unit 11.

[0059] With the above structure, the image data collection analysis device 1 according to Embodiment 2 can also be used in the networked automobile system and share information, and even in a case where the processing of the image analysis unit 14 is not caught up, important image data can be prevented from being deleted.

[0060] Explanation of Reference Numerals

[0061] 1 image data collection analysis device, 2 camera, 3 sensor, 10 storage, 11 vehicle-external communication unit, 12 image acquisition unit, 13 sensor value acquisition unit, 14 image analysis unit, 15 abnormality degree determination unit, 16 analysis priority determination unit, 17 server information acquisition unit.

Claims

1. An image data collection and analysis device, characterized in that, include: Sensor value acquisition unit acquires sensor values ​​about the vehicle from sensors mounted on the vehicle; An image acquisition unit that acquires images of the exterior of the vehicle; External communication unit for communicating with external servers; An anomaly determination unit determines the degree of anomaly of the road on which the vehicle is traveling based on the sensor values ​​acquired by the sensor value acquisition unit. An analysis priority determination unit determines the analysis priority for analyzing each acquired exterior image based on the determined degree of anomaly; and The image analysis unit analyzes the road surface condition reflected in the external images of the vehicle in descending order of analysis priority.

2. The image data collection and analysis device as described in claim 1, characterized in that, The image analysis unit analyzes the images acquired in the vehicle exterior images when the analysis priority exceeds a predetermined threshold, in descending order of analysis priority.

3. The image data collection and analysis device as described in claim 1, characterized in that, It also includes a server information acquisition unit that obtains server information from the external communication unit. The analysis priority determination unit determines the analysis priority based on the degree of anomaly and the server information.

4. The image data collection and analysis device as described in claim 1, characterized in that, The image analysis unit only analyzes images whose analysis priority exceeds a predetermined threshold.

Images