Vehicle monitoring device, vehicle monitoring method, and program

Abstract

To estimate the state of a moving body approaching a vehicle with good accuracy by using detection results from a sensor that detects a moving body using electromagnetic waves.SOLUTION: A vehicle monitoring device 10 comprises an acquisition unit 110, a preliminary processing unit 120, and a generation unit 130. The acquisition unit 110 acquires detection data which is generated by a moving body sensor for detecting a moving body by irradiating an electromagnetic wave and detecting the reflected wave of the electromagnetic wave, and which indicates at least the position of a moving body around a vehicle. The preliminary processing unit 120 generates input data by processing the detection data. The generation unit 130 generates status information indicating the status of the moving body by processing the input data. When the status information satisfies a criterion, the preliminary processing unit 120 adds processing treatment to the input data so as to ensure that the status information corresponding to the input data after processing treatment indicates what is being detected is a physical object other than the moving body.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a vehicle monitoring device, a vehicle monitoring method, and a program. [Background technology]

[0002] In recent years, the state of a moving object has been detected using an image. For example, Patent Document 1 describes a device that performs the following processing. When a moving object is detected in an area of ​​interest, which is one of the areas in an input image, for a first period of time, this device transitions the area state of the area of ​​interest from an absent state to an entering state. When a moving object is detected in the area of ​​interest for a second period of time in the entering state, this device transitions the area state of the area of ​​interest to a staying state. Furthermore, when no moving object is detected in the area of ​​interest for a third period of time in the staying state, this device transitions the area state of the area of ​​interest to a leaving state. When no moving object is detected in the area of ​​interest for a fourth period of time in the leaving state, this device transitions the area state of the area of ​​interest to an absent state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-184233 Summary of the Invention [Problem to be solved by the invention]

[0004] A vehicle may be equipped with a sensor that detects a moving object using electromagnetic waves. One object of the present invention is to accurately estimate the state of a moving object approaching the vehicle using the detection results of the sensor. [Means for solving the problem]

[0005] The invention described in claim 1 is an acquisition unit that acquires detection data that indicates at least the position of a moving object around a vehicle, the detection data being generated by a moving object sensor that detects a moving object by irradiating electromagnetic waves and detecting reflected waves of the electromagnetic waves; a pre-processing unit that processes the detection data to generate input data; a generation unit that processes the input data to generate status information indicating a status of the moving object; Equipped with The pre-processing unit is a vehicle monitoring device that, when the status information meets a criterion, performs processing on the input data so that the status information corresponding to the input data after processing indicates that an object other than a moving object has been detected.

[0006] The invention described in claim 6 is a computer comprising: acquiring detection data indicating at least the position of a moving object around the vehicle, the detection data being data generated by a moving object sensor that detects a moving object by irradiating electromagnetic waves and detecting reflected waves of the electromagnetic waves; processing the detected data to generate input data; generating status information indicating a status of the moving object by processing the input data; This is a vehicle monitoring method in which, when the status information satisfies a criterion, processing is performed on the input data so that the status information corresponding to the processed input data indicates that an object other than a moving object has been detected.

[0007] The invention described in claim 7 is a computer-implemented method for implementing the invention. an acquisition unit that acquires detection data that indicates at least the position of a moving object around the vehicle, the detection data being data generated by a moving object sensor that detects a moving object by irradiating electromagnetic waves and detecting reflected waves of the electromagnetic waves; a pre-processing unit that processes the detection data to generate input data; a generation unit that processes the input data to generate status information indicating a status of the moving object; Let them have The pre-processing unit is a program that, when the status information satisfies a criterion, performs processing on the input data so that the status information corresponding to the input data after processing indicates that an object other than a moving object has been detected. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a usage environment of a vehicle monitoring device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a vehicle monitoring device. [Figure 3] FIG. 10 is a diagram illustrating an example of processing performed by a pre-processing unit. [Figure 4] FIG. 10 is a diagram illustrating an example of processing performed by a first generation unit. [Figure 5] FIG. 10 is a diagram illustrating an example of processing performed by a second generation unit. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a vehicle monitoring device. [Figure 7] 4 is a flowchart illustrating an example of processing performed by the vehicle monitoring device. [Figure 8] 10A and 10B are diagrams illustrating an example of processing performed by a pre-processing unit. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0010] 1 is a diagram illustrating an environment in which a vehicle monitoring device 10 according to an embodiment is used. The vehicle monitoring device 10 is used together with a sensor device 20 and a communication device 30.

[0011] The sensor device 20 is mounted on the vehicle 50 and has at least one moving object sensor. This moving object sensor detects the position of a moving object by emitting electromagnetic waves and detecting the reflected waves of the electromagnetic waves. The frequency of the electromagnetic waves is, for example, 300 MHz to 300 GHz, and the wavelength of the electromagnetic waves is, for example, 1 mm to 1 m. In the example shown in the figure, the sensor device 20 is attached to the upper part of the inner surface of the windshield of the vehicle 50 and has moving object sensors 22, 24, and 26. The moving object sensor 22 has a detection range in front of the vehicle 50, the moving object sensor 24 has a detection range in the right and rear of the vehicle, and the moving object sensor 26 has a detection range in the left and rear of the vehicle 50. The detection ranges of the moving object sensors 24 and 26 overlap at the rear of the vehicle 50. When looking at the motion sensors 22, 24, and 26 as a whole, it is desirable for the detection range of these motion sensors to cover the entire circumference of the vehicle 50, but since intrusion into the vehicle often occurs from the side, it is sufficient to at least cover the range of the side of the vehicle, and detection can be performed using motion sensors 24 and 26 excluding motion sensor 22.

[0012] The data generated by the motion sensor of the sensor device 20, i.e., the detection data, indicates at least the position of a moving object when the moving object is located within the detection range of the motion sensor. In other words, when a moving object enters the detection range of the motion sensor, the detection data begins to indicate the position of the moving object. On the other hand, when the moving object leaves the detection range of the motion sensor, the detection data no longer indicates the position of the moving object.

[0013] The detection data is transmitted from the sensor device 20 to the vehicle monitoring device 10. At least a part of this transmission route is, for example, wireless communication. This wireless communication may be, for example, a public communication network, or short-range wireless communication such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

[0014] The vehicle monitoring device 10 may be mounted on the vehicle 50. In this case, communication between the vehicle monitoring device 10 and the sensor device 20 may be wireless or wired.

[0015] The vehicle monitoring device 10 processes the detection data to generate and output information indicating the status of a moving object present near the vehicle 50, for example, the status of a person. Hereinafter, this information will be referred to as status information. Examples of the status include, but are not limited to, that a moving object has entered the detection range of the moving object sensor, i.e., that the moving object has approached the vehicle 50, that the moving object continues to exist around the vehicle 50, and that the moving object has entered the vehicle 50.

[0016] The vehicle monitoring device 10 transmits status information to the communication device 30. The communication device 30 is, for example, a terminal operated by the owner or manager of the vehicle 50. The communication device 30 may be a portable communication device or a fixed communication device. The vehicle monitoring device 10 may transmit all status information to the communication device 30, or may transmit only specific status information to the communication device 30. As an example, the vehicle monitoring device 10 transmits status information to the communication device 30 both when a moving object enters the detection range of the motion sensor and when a moving object intrudes into the vehicle 50. As another example, the vehicle monitoring device 10 does not transmit status information to the communication device 30 when a moving object enters the detection range of the motion sensor, but transmits status information to the communication device 30 when a moving object intrudes into the vehicle 50.

[0017] 2 is a diagram showing an example of the functional configuration of the vehicle monitoring device 10. In the example shown in this figure, the vehicle monitoring device 10 has an acquisition unit 110, a pre-processing unit 120, a generation unit 130, and an output unit 160. In the example shown in this figure, the generation unit 130 has a first generation unit 140 and a second generation unit 150.

[0018] The acquisition unit 110 acquires detection data. The acquisition unit 110 may acquire the detection data from the sensor device 20 via a communication network, or may acquire the detection data from a storage device that stores the detection data. The detection data indicates at least the position of the moving object, but may include multiple types of data. Examples of data included in the detection data include coordinate data indicating the position of the moving object, a first index related to the position of the moving object relative to the vehicle, a second index related to the direction of movement of the moving object relative to the vehicle, and the reception strength of the reflected wave, as well as the amount of change over time in these indices and reception strength, i.e., time-series data.

[0019] For example, the first index has a smaller value as the distance between the vehicle and the moving object increases. For example, the first index decreases linearly as the distance between the vehicle, i.e., the moving object sensor, and the moving object increases. The value of the first index may change depending on which part of the vehicle the moving object is closest to.

[0020] For example, the second index is calculated as follows. First, the direction of travel of the moving object is identified from the coordinates of two or more consecutive points of the moving object, which are data included in the detection data. Then, if the moving object is traveling away, the second index will have the lowest value, for example, "1." If the moving object is traveling parallel to the vehicle, the second index will have the next lowest value, for example, "2." Furthermore, if the moving object is traveling toward the vehicle, the second index will have a slightly higher value, for example, "3." Furthermore, if the moving object is traveling toward the door of the vehicle, the second index will have a higher value, for example, "4."

[0021] The second index may also have a high value if, for example, the person is standing near the door of the vehicle, and then decrease in value if the person is standing near the front or rear of the vehicle, and then decrease in value if the person is standing away from the vehicle.

[0022] The acquisition unit 110 stores the acquired detection data in a storage unit. This storage unit may be a part of the vehicle monitoring device 10, or may be located outside the vehicle monitoring device 10. The vehicle monitoring device 10 reads out and uses the detection data stored in the storage unit as needed.

[0023] The pre-processing unit 120 processes the detection data detected at the target timing to generate input data corresponding to the target timing. In this case, the pre-processing unit 120 preferably processes detection data having a length of 1 second to 5 seconds, including the target timing. For example, the pre-processing unit 120 uses detection data from a first reference time before the target timing to a second reference time after the target timing. An example of the first reference time is 0.5 seconds to 3 seconds. An example of the second reference time is also 0.5 seconds to 3 seconds. The first reference time and the second reference time may be equal to each other.

[0024] An example of input data is a two-dimensional image, as shown in Fig. 3. This image is, for example, a chart showing the time transition of a parameter based on at least a part of the detection data. This parameter is calculated based on, for example, the first index, the second index, and the received intensity of the reflected wave. However, this parameter is not limited to this example.

[0025] The generator 130 at least processes the input data using a machine learning model or a rule base.

[0026] In the example shown in this figure, the generation unit 130 includes a first generation unit 140 and a second generation unit 150.

[0027] The first generation unit 140 processes input data using a machine learning model to generate first status information indicating the status of the moving object. An example of the machine learning used here is a neural network such as a convolutional neural network (CNN), but is not limited to this. When the input data is an image and the machine learning model is a model suitable for image processing, such as a neural network, the first generation unit 140 can efficiently process the input data. The first generation unit 140 stores the generated first status information in a storage unit. This allows the vehicle monitoring device 10 to use the history of the first status information.

[0028] 4, the first status information indicates the status of a person when it is assumed that the moving object is a person. The status indicated by the first status information is, for example, as follows:

[0029] "01": No person is present within the detection range of the motion sensor. "02": A person entered the detection range of the motion sensor. "03": A person approached vehicle 50. "04": A person is checking vehicle 50. "05": A person entered vehicle 50. However, the status indicated by the first status information is not limited to these.

[0030] Furthermore, by using the history of detection data, it may be possible to estimate that the moving object is not a person. In this case, the first status information indicates that the moving object is likely to be an object other than a person. An example of such an object is an object with a movable portion, such as a plant or a flag, which cannot move by itself but whose portion moves due to airflow.

[0031] The first generating unit 140 may generate the first status information on a rule basis. In this case, the first generating unit 140 can generate the first status information using the detection data, and therefore processing by the pre-processing unit 120 does not have to be performed. Therefore, when the first generating unit 140 generates the first status information on a rule basis, the vehicle monitoring device 10 does not have to include the pre-processing unit 120.

[0032] The pre-processing unit 120 can also be considered as part of the first generating unit 140.

[0033] The second generation unit 150 generates second status information indicating a status to be output by the vehicle monitoring device 10 using transitions in the first status information. The second generation unit 150 generates the second status information by processing the first status information according to a transition diagram equivalent to a state machine, for example. In other words, the second generation unit 150 generates second status information indicating a status to be output at a target timing using first status information generated at the target timing and first status information generated at least either before or after the target timing. Examples of statuses indicated by the second status information are similar to the examples of the first status information. The second generation unit 150 stores the generated second status information in a storage unit. This allows the vehicle monitoring device 10 to use the history of the second status information.

[0034] The second generation unit 150 generates the second status information by processing the first status information according to the transition diagram shown in Fig. 5. In this example, the relationship between the first status information and the second status information is as follows:

[0035] "01": If the secondary status information indicates that no one is present This is the case when the latest first status information is "01".

[0036] "02": The second status information indicates that a person is within the detection range of the motion sensor, but is not approaching the vehicle 50. This is the case where both the immediately previous first status information and the latest first status information are "02".

[0037] "03" When the second status information indicates that a moving object is approaching the vehicle 50 This is the case when the immediately previous first status information is "03" and the latest first status information is "03." Furthermore, if the immediately previous first status information is "01" or "02" and the latest first status information is "03," "04," or "05," the second status information will also be "03."

[0038] "04" If the second status information indicates that a person is checking the vehicle 50 This is the case when the immediately previous first status information is "03" and the latest first status information is "04." Furthermore, if the immediately previous first status information is "04" and the latest first status information is "03" or "04," the second status information will also be "04."

[0039] "05": When the second status information indicates that a person has entered the vehicle 50 This is the case when the previous first status information is "03" or "04" and the latest first status information is "05." Furthermore, if the previous first status information is "05" and the latest first status information is "03," "04," or "05," the second status information will also be "05."

[0040] The second generation unit 150 may use, for example, the detection data to identify a location on the vehicle 50 to which the moving object is approaching, and add information indicating this location to the second status information. This information may be, for example, at least one of the left front door, right front door, left rear door, right rear door, front, rear, left front tire, right front tire, left rear tire, and right rear tire, but is not limited to these. The second generation unit 150 generates this information using, for example, the position and movement direction of the moving object included in the detection data.

[0041] The pre-processing unit 120 and the generation unit 130 repeatedly perform the above-described processing. That is, the pre-processing unit 120 repeatedly generates input data while shifting the target timing by a predetermined time. The generation unit 130 performs the above-described processing each time input data is generated, and generates status information. The cycle of this repetition, i.e., the amount of shift in the target timing, is, for example, between 100 milliseconds and 2 seconds.

[0042] The output unit 160 outputs the status information generated by the generation unit 130, for example, the second status information, to the communication device 30. As described with reference to FIG. 1 , the output unit 160 may transmit all of the status information to the communication device 30, or may transmit only specific status information to the communication device 30.

[0043] 6 is a diagram showing an example of the hardware configuration of the vehicle monitoring device 10. The vehicle monitoring device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0044] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0045] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0046] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0047] The storage device 1040 is an auxiliary storage device realized by removable media such as a hard disk drive (HDD), a solid state drive (SSD), or a memory card, or a read-only memory (ROM). The storage device 1040 stores program modules that realize each function of the vehicle monitoring device 10 (e.g., the acquisition unit 110, the pre-processing unit 120, the generation unit 130, the first generation unit 140, the second generation unit 150, and the output unit 160). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0048] The input / output interface 1050 is an interface for connecting the vehicle monitoring device 10 to various input / output devices.

[0049] The network interface 1060 is an interface for connecting the vehicle monitoring device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may be connected to the network wirelessly or by wire. The vehicle monitoring device 10 may communicate with the sensor device 20 and the communication device 30 via the network interface 1060.

[0050] FIG. 7 is a flowchart showing an example of processing performed by the vehicle monitoring device 10. The vehicle monitoring device 10 repeats the processing shown in this figure at intervals of, for example, 0.2 seconds or more and 2 seconds or less. First, the acquisition unit 110 acquires detection data (step S10). For example, when the vehicle monitoring device 10 acquires the latest detection data, it reads out from the storage unit detection data from this detection data up to a predetermined time before. Here, the predetermined time is, for example, 1 second or more and 5 seconds or less.

[0051] Next, the pre-processing unit 120 processes the detection data acquired and read by the acquiring unit 110 to generate input data (step S20).

[0052] Next, first generation unit 140 generates first status information (step S30), and further, second generation unit 150 generates second status information (step S40). After that, if the second status information satisfies a criterion (step S50), output unit 160 transmits the second status information to communication device 30 (step S60). Communication device 30 executes processing according to the second status information, for example, warning processing.

[0053] Note that the second status information described above is not necessarily correct. For example, even if the object is not a person, the second generation unit 150 may output any of the above-mentioned "02" to "04" as the second status information. An example of such a case is when the moving object is an object with a movable part, such as a plant or a flag, which cannot move by itself but whose part moves due to the flow of air.

[0054] In response to this, when the status information satisfies the criteria, the pre-processing unit 120 processes the input data so that the status information corresponding to the processed input data indicates that an object other than a moving object has been detected. The status information used here is, for example, the second status information. However, this status information may also be the first status information.

[0055] This criterion relates to, for example, the first index (i.e., the index relating to the distance between the moving object and the vehicle) and a value based on the reception strength of the reflected wave. For example, this criterion is that the product of the first index and the reception strength is equal to or less than a reference value.

[0056] An example of the processing is to invert and emphasize the waveform in a chart showing the time transition of a parameter based on at least a part of the input data, as shown in Fig. 8. As a result, the processed input data is significantly different from the input data when the first status information is "02" to "04" as described above, and as a result, the first status information generated by the first generation unit 140 indicates that the moving object is an object other than a person.

[0057] The machine learning model used by the first generation unit 140 also learns data that links the processed input data with status information indicating that the input data is a non-human object. Here, when the pre-processing unit 120 performs processing, the device that generates the machine learning model may re-train the machine learning model using the processed input data.

[0058] As described above, according to this embodiment, the state of a moving object approaching a vehicle can be accurately estimated using the detection results of a sensor that detects a moving object using electromagnetic waves. Furthermore, since the data used for one processing is, for example, from 1 second to 5 seconds, an increase in the amount of calculation can be suppressed, and the real-time nature of the processing can be improved.

[0059] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0060] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-mentioned embodiments can be combined to the extent that the content is not contradictory. [Explanation of symbols]

[0061] 10 Vehicle monitoring device 20 Sensor device 22 Motion Sensor 24 Motion Sensor 26 Motion Sensor 30 Communication equipment 50 vehicles 110 Acquisition Department 120 Pre-processing section 130 Generation part 140 1st generation part 150 Second generation part 160 Output section

Claims

1. an acquisition unit that acquires detection data that indicates at least the position of a moving object around the vehicle, the detection data being data generated by a moving object sensor that detects a moving object by irradiating electromagnetic waves and detecting reflected waves of the electromagnetic waves; a pre-processing unit that processes the detection data to generate input data; a generation unit that processes the input data to generate status information indicating a status of the moving object; Equipped with The pre-processing unit processes the input data when the status information satisfies a criterion, so that the status information corresponding to the processed input data indicates that an object other than a moving object has been detected.

2. 2. The vehicle monitoring device according to claim 1, the detection data includes an index relating to a distance between the moving object and the vehicle and a reception intensity of the reflected wave; The vehicle monitoring device, wherein the criteria relate to a value based on the indicator and the reception strength.

3. 3. The vehicle monitoring device according to claim 1, The generation unit a first generation unit that processes the input data using a machine learning model to generate first status information indicating a status of the moving object; a second generator that generates second status information indicating a status at the target timing to be output, using the first status information generated at the target timing and the first status information generated at least one of before and after the target timing; Equipped with The vehicle monitoring device, wherein the status information used by the pre-processing unit is second status information.

4. 4. The vehicle monitoring device according to claim 3, the pre-processing unit processes the detection data for 1 second to 5 seconds including a target timing, thereby repeatedly generating the first status information at the target timing; A vehicle monitoring device, wherein the target timing interval is between 100 milliseconds and 2 seconds.

5. 3. The vehicle monitoring device according to claim 1, The vehicle monitoring device, wherein the input data is an image.

6. The computer acquiring detection data indicating at least the position of a moving object around the vehicle, the detection data being data generated by a moving object sensor that detects a moving object by irradiating electromagnetic waves and detecting reflected waves of the electromagnetic waves; processing the detected data to generate input data; generating status information indicating a status of the moving object by processing the input data; A vehicle monitoring method in which, when the status information meets a criterion, processing is performed on the input data so that the status information corresponding to the processed input data indicates that an object other than a moving object has been detected.

7. On the computer, an acquisition unit that acquires detection data that indicates at least the position of a moving object around the vehicle, the detection data being data generated by a moving object sensor that detects a moving object by irradiating electromagnetic waves and detecting reflected waves of the electromagnetic waves; a pre-processing unit that processes the detection data to generate input data; a generation unit that processes the input data to generate status information indicating a status of the moving object; Let them have The pre-processing unit is a program that, when the status information meets a criterion, performs processing on the input data so that the status information corresponding to the input data after processing indicates that an object other than a moving object has been detected.

Images