Object detection device and object detection method

Abstract

An object detection device according to an embodiment of the present invention generates an object detection model by, in a training phase, training said model on the relationship between information indicating whether an object is an obstacle and a feature vector indicating the distribution shape of a detection point cloud based on reflected waves resulting from probe waves that were transmitted from a sensor being reflected by the object. Furthermore, in an estimation phase, the object detection device: calculates, on the basis of a plurality of acquired reception results, a detection point cloud as the position of the object; calculates, on the basis of the calculated detection point cloud, the feature vector indicting the distribution shape of the detection point cloud; determines, on the basis of the calculated feature vector and the object detection model, whether the object is an obstacle; and outputs the determination result.

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an object detection device and an object detection method. [Background technology]

[0002] Research and development has been ongoing into automatic door systems that automatically open and close vehicle (automobile) doors operated by passengers or those about to enter. An essential requirement of an automatic door system is that the door does not collide with obstacles (people, other vehicles, etc.) when opening. In the following, doors that open and close automatically are also referred to as "automatic doors." In addition, a swing-type door is assumed below.

[0003] To achieve the above-mentioned performance, for example, an object detection sensor (such as millimeter-wave radar) is installed inside the automatic door, and the position of the obstacle is estimated based on the detection point cloud obtained by sensing, and the door is controlled so as not to collide with the obstacle.

[0004] This method makes it possible to estimate (calculate) the position of an obstacle from the geometric information of the detection point cloud captured by the object detection sensor. This makes it possible to calculate the opening angle (opening angle) of the automatic door and to open it without hitting the obstacle. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-134090 [Patent Document 2] Japanese Patent Application Laid-Open No. 2003-271975 [Patent Document 3] Patent No. 6261489 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned conventional technology may have problems depending on the surrounding environment, such as the road surface. For example, on some types of road surfaces, such as asphalt roads, many detection points appear even when there are no obstacles. This can result in a large error in the detected position when an obstacle is present, or can lead to a false determination that an obstacle is present when there is no obstacle.

[0007] In other words, if a detection point cloud containing a certain amount of noise is used, problems may occur such as the door stopping its opening operation when the distance to the obstacle is still large, or the door not opening at all even when it is possible to open it.

[0008] Therefore, one object of the present invention is to provide an object detection device and an object detection method that can detect obstacles around a vehicle with high accuracy regardless of the surrounding environment such as the road surface. [Means for solving the problem]

[0009] An object detection device according to an embodiment of the present invention includes, for example, an acquisition unit that acquires multiple reception results of reflected waves generated when a probe wave transmitted from a sensor installed on a vehicle door is reflected by objects around the vehicle; a model generation unit that, in a learning phase, calculates a detection point cloud as the position of the object based on the multiple reception results acquired by the acquisition unit and generates an object detection model by machine learning the relationship between a feature vector indicating the distribution shape of the detection point cloud and information indicating whether the object is an obstacle; and a first calculation unit that, in an estimation phase, calculates the detection point cloud as the position of the object based on the multiple reception results acquired by the acquisition unit, a second calculation unit that calculates a feature vector indicating the distribution shape of the detection point cloud based on the detection point cloud calculated by the first calculation unit, and an estimation unit that determines whether the object is an obstacle based on the feature vector calculated by the second calculation unit and the object detection model and outputs a determination result. With this configuration, by using an object detection model generated in advance through machine learning, obstacles around the vehicle can be detected with high accuracy regardless of the surrounding environment, such as the road surface.

[0010] Furthermore, in the object detection device, for example, the model generation unit transforms the detection point cloud into three-dimensional coordinates based on the door on which the sensor is installed, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest, and uses the calculated feature vector as input data, and the second calculation unit transforms the detection point cloud calculated by the first calculation unit into the three-dimensional coordinates, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, and calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest. This configuration makes it possible to implement more specific processing, such as setting a region of interest using three-dimensional coordinates based on the door and calculating a feature vector that indicates the distribution shape of the detection point cloud.

[0011] Furthermore, in the object detection device, for example, the door is a swing door, and the object detection device further includes a control unit that controls a drive unit that opens and closes the door, and when the estimation unit outputs information indicating that the object is an obstacle, the control unit sets an opening angle of the door based on position information of the obstacle, and controls the drive unit so that the door opens to the set opening angle. With this configuration, by opening the door up to the set opening angle, it is possible to avoid collision between the door and an obstacle, and also to avoid the door unnecessarily stopping its opening operation too early.

[0012] In addition, in the object detection device, for example, the control unit controls the drive unit so that the door opens to the set opening angle based on a request for automatic opening of the door from the user of the vehicle. With this configuration, the vehicle user can open the door by performing a predetermined operation corresponding to the request for automatic door opening.

[0013] Furthermore, the object detection method of this embodiment includes, for example, an acquisition step of acquiring multiple reception results of reflected waves generated when a probe wave transmitted from a sensor installed on a vehicle door is reflected by objects around the vehicle; a model generation step of calculating a detection point cloud as the position of the object based on the multiple reception results acquired by the acquisition step and generating an object detection model by machine learning the relationship between a feature vector indicating the distribution shape of the detection point cloud and information indicating whether the object is an obstacle; and an estimation step of calculating a detection point cloud as the position of the object based on the multiple reception results acquired by the acquisition step, a second calculation step of calculating a feature vector indicating the distribution shape of the detection point cloud based on the detection point cloud calculated by the first calculation step, and an estimation step of determining whether the object is an obstacle based on the feature vector calculated by the second calculation step and the object detection model and outputting the determination result. With this configuration, by using an object detection model generated in advance through machine learning, obstacles around the vehicle can be detected with high accuracy regardless of the surrounding environment, such as the road surface.

[0014] An object detection method according to an embodiment of the present invention includes, for example, an acquisition step of acquiring a plurality of reception results of reflected waves generated when a probe wave transmitted from a sensor installed in a swing door of a vehicle is reflected by objects around the vehicle; a learning phase of calculating a detection point cloud as the position of the object based on the plurality of reception results acquired in the acquisition step, and generating an object detection model by machine learning the relationship between a feature vector indicating the distribution shape of the detection point cloud and information indicating whether the object is an obstacle; and an estimation phase of generating an object detection model based on the plurality of reception results acquired in the acquisition step. a first calculation step of calculating a detection point cloud; a second calculation step of calculating a feature vector indicating a distribution shape of the detection point cloud based on the detection point cloud calculated by the first calculation step; an estimation step of determining whether the object is an obstacle based on the feature vector calculated by the second calculation step and the object detection model and outputting a determination result; and a control step of, when information indicating that the object is an obstacle is output by the estimation step, setting an opening angle of the door based on position information of the obstacle and controlling a drive unit that opens and closes the door so that the door opens to the set opening angle. With this configuration, by using an object detection model generated in advance through machine learning, it is possible to detect obstacles around the vehicle with high accuracy, regardless of the surrounding environment such as the road surface, etc. Furthermore, by opening the door up to the set opening angle, it is possible to avoid collisions between the door and obstacles and also to prevent the door from stopping unnecessarily short of the target. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a side view of a vehicle on which a sensor unit according to an embodiment is mounted. [Figure 2] FIG. 2 is a functional configuration diagram of the automatic door system according to the embodiment. [Figure 3] FIG. 3 is a functional configuration diagram of the DSP according to the embodiment. [Figure 4]FIG. 4 is a flowchart showing the overall processing by the automatic door system of the embodiment. [Figure 5] FIG. 5 is a flowchart showing the details of the process in step S13 of FIG. [Figure 6] FIG. 6 is a flowchart showing the details of a modified example of the process of step S13 in FIG. [Figure 7] FIG. 7 is a flowchart showing the details of the process in step S202 of FIG. [Figure 8] FIG. 8 is a diagram illustrating an example of a detection point cloud when an obstacle is present in the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a detection point cloud when there is no obstacle in the embodiment. [Figure 10] FIG. 10 is a diagram showing an example of setting a region of interest when an obstacle is present in the embodiment. [Figure 11] FIG. 11 is a diagram showing an example of setting a region of interest when there is no obstacle in the embodiment. [Figure 12] FIG. 12 is a diagram showing an example of feature amounts extracted when an obstacle is present in the embodiment. [Figure 13] FIG. 13 is a diagram showing an example of feature amounts extracted when there is no obstacle in the embodiment. [Figure 14] FIG. 14 is a diagram showing an example of a feature vector created in the embodiment. [Figure 15] FIG. 15 is a graph for comparing two feature vectors in the embodiment. [Figure 16] FIG. 16 is a comparison table showing the performance of a plurality of machine learning machines in the embodiment in determining whether an object is present or absent. [Figure 17] FIG. 17 is a flowchart showing the details of the process in step S204 of FIG. [Figure 18] FIG. 18 is an explanatory diagram of the calculation of the door movable angle in the embodiment. [Figure 19] FIG. 19 is an explanatory diagram of object type discrimination based on statistical quantities of distribution of detection points in the embodiment. [Figure 20] FIG. 20 is an explanatory diagram of object type discrimination based on geometric features of the distribution of a group of detected points in this embodiment. [Figure 21] FIG. 21 is an explanatory diagram of object type discrimination based on the reflected energy value of the distribution of the detection point cloud in the embodiment. [Figure 22] FIG. 22 is a comparison table showing the object type discrimination performance of multiple machine learning machines according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] The following describes embodiments of an object detection device and an object detection method of the present invention. The configurations of the following embodiments, as well as the actions, results, and advantages brought about by the configurations, are examples. The present invention can be realized with configurations other than those disclosed in the following embodiments, and it is possible to obtain at least one of the various advantages based on the basic configurations and derivative advantages.

[0017] In the following embodiments, a case where "supervised learning (learning using teacher data)" is adopted as an example of machine learning will be described. In addition, with regard to machine learning, the learning situation will be referred to as a "learning phase," and the estimation situation will be referred to as an "estimation phase."

[0018] Fig. 1 is an external view of a vehicle 1 equipped with a sensor unit 3 according to an embodiment, as seen from the side. Fig. 2 is a functional configuration diagram of an automatic door system S according to an embodiment.

[0019] As shown in Fig. 1, a sensor unit 3 is installed at a predetermined position on a swing-type door 21 of a vehicle 1. Millimeter waves used by the sensor unit 3 are reflected and absorbed by metal but easily pass through plastic resin, so the sensor unit 3 is preferably mounted on the resin part of the panel of the door 21. Note that for the sake of simplicity of illustration and explanation, the sensor unit 3 is installed on only one door 21 in Fig. 1, but this is not limiting, and sensor units may be installed on two or more doors.

[0020] 2, the automatic door system S includes a sensor unit 3 and an automatic door unit 2. One automatic door system S is provided for each door 21.

[0021] The sensor unit 3 is a means for detecting an obstacle that hinders the automatic opening operation of the door 21. The sensor unit 3 includes a DSP 31 (Digital Signal Processor) and a millimeter wave radar 32 (sensor).

[0022] The millimeter-wave radar 32 is a sensor component that transmits millimeter waves (radio waves in the frequency band of 30 to 300 GHz) into the surrounding area, receives reflected millimeter waves, and generates and outputs an intermediate frequency (IF) signal by mixing the two waves. The output information from the millimeter-wave radar 32 is converted into a digital signal by an analog-to-digital conversion circuit. In recent years, millimeter-wave radar 32 has become smaller and thinner, making it easier to embed it inside the door 21 of the vehicle 1.

[0023] The DSP 31 calculates the position, speed, etc. of an obstacle based on the IF signal output from the millimeter wave radar 32. The DSP 31 is a device that performs specialized signal processing. Because the DSP 31 is a type of computer, it is possible to add and execute a program that adds further special signal processing based on the calculated information.

[0024] 3 is a functional block diagram of the DSP 31 according to the embodiment. As shown in FIG. 3, the DSP 31 includes a processing unit 5 and a storage unit 6.

[0025] The storage unit 6 stores programs executed by the processing unit 5 and data required for executing the programs. For example, the storage unit 6 stores an object detection program executed by the processing unit 5, as well as numerical data and door trajectory data required for executing the object detection program. The storage unit 6 is realized by, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The ROM stores each program, parameters, etc. The RAM temporarily stores various data used in calculations by the CPU (Central Processing Unit).

[0026] The processing unit 5 calculates the position of an object based on the information output from the millimeter-wave radar 32. The processing unit 5 is realized, for example, as a function of a CPU. The processing unit 5 includes, as functional components, an acquisition unit 51, a model generation unit 52, a first calculation unit 53, a second calculation unit 54, an estimation unit 55, and a control unit 56. The processing unit 5 operates as each functional component by, for example, reading an object detection program stored in the storage unit 6. Furthermore, some or all of each functional component may be realized by hardware such as a circuit including an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0027] The acquisition unit 51 acquires various pieces of information from various components. For example, the acquisition unit 51 acquires, from the millimeter-wave radar 32, a plurality of reception results of reflected waves generated when millimeter waves (prospecting waves) transmitted from the millimeter-wave radar 32 are reflected by objects around the vehicle 1.

[0028] In the learning phase, the model generation unit 52 calculates a detection point cloud as the position of an object based on the multiple reception results acquired by the acquisition unit 51, and generates an object detection model by machine learning the relationship between a feature vector indicating the distribution shape of the detection point cloud and information indicating whether the object is an obstacle. In this case, the model generation unit 52 converts the detection point cloud into three-dimensional coordinates based on the door 21 on which the sensor unit 3 is installed, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, and calculates a feature vector indicating the distribution shape of the detection point cloud in the set region of interest as input data (details will be described later).

[0029] In the estimation phase, the first calculation unit 53, the second calculation unit 54, and the estimation unit 55 perform the following processes.

[0030] The first calculation unit 53 calculates a detection point group as the position of the object based on the plurality of reception results newly acquired by the acquisition unit 51.

[0031] The second calculation unit 54 calculates a feature vector indicating the distribution shape of the detection point cloud, based on the detection point cloud calculated by the first calculation unit 53. In this case, the second calculation unit converts the detection point cloud calculated by the first calculation unit 53 into three-dimensional coordinates, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, and calculates a feature vector indicating the distribution shape of the detection point cloud in the set region of interest (details will be described later).

[0032] The estimation unit 55 determines whether the object is an obstacle or not based on the feature vector calculated by the second calculation unit 54 and the object detection model, and outputs the determination result.

[0033] The control unit 56 executes various controls. For example, when the estimation unit 55 outputs information indicating that the object is an obstacle as a discrimination result, the control unit 56 sets an opening movable angle of the door 21 (hereinafter also referred to as "door movable angle") based on the position information of the obstacle (details will be described later).

[0034] The DSP 31 outputs the processed information to the automatic door unit 2 via the in-vehicle network 4. The in-vehicle network 4 is, for example, a Controller Area Network (CAN) or a Flexible Data Rate (CAN-FD).

[0035] Returning to Figure 2, the automatic door unit 2 is a means for controlling the opening and closing operation of the door 21 based on obstacle detection information from the sensor unit 3. The automatic door unit 2 includes the door 21, a door drive unit 22 (a drive unit that opens and closes the door), and an ECU 23 (Electronic Control Unit).

[0036] The door drive unit 22 is an electric component that opens and closes the door 21 .

[0037] The ECU 23 is a device that executes specialized signal processing to determine how to control the door 21 based on information received from the DSP 31. Because the ECU 23 is a type of computer, it is possible to add and execute a program that adds special signal processing to the ECU 23.

[0038] The ECU 23 is a control unit that executes various controls. The ECU 23 controls the door drive unit 22 that is installed at the hinge portion of the door 21. For example, the ECU 23 controls the door drive unit 22 so that the door 21 opens to the door movable angle set by the DSP 31.

[0039] Furthermore, the ECU 23 controls the door drive unit 22 so that the door 21 opens up to a set opening angle, based on a request for automatic opening of the door 21 from the user of the vehicle 1, for example.

[0040] Next, Figure 4 is a flowchart showing the overall processing by the automatic door system S of this embodiment. First, an overview will be given. This overall processing starts as soon as the user of the vehicle 1 puts the vehicle 1 into an active state, and continues at all times while the vehicle 1 is in an active state. Furthermore, this overall processing is also stopped when the user of the vehicle 1 puts the vehicle 1 into an inactive state. However, this overall processing may be temporarily suspended when the vehicle 1 is in motion or when the vehicle 1 is in a resting state to reduce electrical energy consumption.

[0041] First, in step S11, the automatic door system S determines whether the vehicle 1 is stationary and the door 21 to be opened is fully closed. If the answer is Yes, the system proceeds to step S12; if the answer is No, the system ends the process. By making the automatic door opening operation conditional in this way, safety can be ensured.

[0042] In step S12, the automatic door system S determines whether or not a command has been input by the user to execute the automatic door opening operation. If the answer is Yes, the process proceeds to step S13; if the answer is No, the process ends. Here, the user refers to a person who is inside or outside the vehicle 1 (hereinafter also referred to as "inside or outside the vehicle") and who is capable of operating the vehicle. For example, the user may be a person inside or outside the vehicle who is in a position to assist others in getting in or out of the vehicle, or the user may be the person who actually gets in or out of the vehicle 1. Furthermore, when the vehicle is an autonomous vehicle, the artificial intelligence responsible for controlling the vehicle may also be the user.

[0043] In addition, the command input method for the automatic door opening operation can be selected from pressing a button on the key fob, the dashboard inside the car, a dedicated application on a smartphone, etc., or by executing a predetermined voice command or gesture.

[0044] In step S13, the automatic door system S executes the automatic door opening operation (details will be described later with reference to FIG. 5).

[0045] Next, in step S14, the automatic door system S executes the automatic door closing operation. The door closing operation may alternatively be performed manually by the user or another person. Furthermore, if the vehicle is an autonomous vehicle, artificial intelligence may close the door after recognizing that a person has boarded or disembarked. After the door 21 is fully closed, the operation flow loops to prepare for the next automatic door opening operation, etc.

[0046] Next, the details of the process of step S13 in Fig. 4 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 shows an example of a continuous operation type, and Fig. 6 shows an example of an event-driven type (details will be described later).

[0047] Fig. 5 is a flowchart showing the details of the process of step S13 in Fig. 4. This process flow in Fig. 5 continues periodically while the vehicle 1 is in an active state. What is important about this process flow is that obstacle detection by the millimeter wave radar 32 (steps S201 to S205 in Fig. 5) is always performed while the overall process in Fig. 4 is being executed, regardless of whether or not the user has input a command for the automatic door opening operation (Yes / No in step S12 in Fig. 4). This will be explained in detail below.

[0048] In step S201, sensing is performed by the millimeter-wave radar 32. That is, the millimeter-wave radar 32 detects an obstacle that is in the vicinity of the open trajectory of the door 21 and that may collide with the door 21. Examples of obstacles include people, vehicles, curbs, and building walls.

[0049] Next, in step S202, the estimation unit 55 of the DSP 31 determines whether or not an obstacle exists based on the sensing data from the millimeter-wave radar 32. Here, the distribution pattern of the detection point cloud captured by the millimeter-wave radar 32 is used as material for determining whether or not an obstacle exists (details will be described later). Note that, although not described here, the first calculation unit 53 and the second calculation unit 54 also perform processing as appropriate.

[0050] Next, in step S203, the control unit 56 of the DSP 31 determines whether or not there is an obstacle that prevents the automatic opening operation, and if Yes, the process proceeds to step S204, and if No, the process proceeds to step S205.

[0051] In step S204, the control unit 56 of the DSP 31 sets the door movable angle based on the position information of the obstacle. That is, the control unit 56 sets the door movable angle for avoiding collision of the door 21 with an obstacle that is in the vicinity of the open trajectory of the door 21 (details will be described later).

[0052] In step S205, the control unit 56 of the DSP 31 sets the door movable angle to the fully open position. For example, the control unit 56 simply determines the door movable angle to be equal to the fully open value of the door hinge.

[0053] Next, in step S206, the ECU 23 determines whether or not the user has input a command for an automatic door opening operation. If Yes, the ECU 23 proceeds to step S207, and if No, the ECU 23 returns to step S201.

[0054] In step S207, the ECU 23 starts the automatic opening operation of the door 21 by controlling the door drive unit 22. Specifically, the ECU 23 determines the speed and acceleration at which the door 21 should be opened, depending on, for example, the presence or absence of an obstacle near the opening trajectory of the door 21 and the current door opening degree, and controls the door drive unit 22 to open the door 21.

[0055] Next, in step S208, the ECU 23 determines whether the opening angle of the door 21 has reached the door movable angle, and if Yes, the ECU 23 proceeds to step S209, and if No, the ECU 23 proceeds to step S210.

[0056] In step S209, the door driving unit 22 performs the automatic opening operation of the door 21.

[0057] In step S210, the door drive unit 22 ends the automatic opening operation of the door 21. That is, the series of automatic door opening operations is ended.

[0058] Figure 6 is a flowchart showing the details of the processing of step S13a, which is a variation of step S13 in Figure 4. As mentioned above, Figure 6 is an example of an event-driven type. Steps S201 to S205 and S207 to S210 are the same as those in Figure 5. The differences from Figure 5 are that the processing flow starts after the user inputs a command for the automatic door opening operation in step S31, and that step S206 is not included.

[0059] The advantage of the event-driven system is that it does not require constant signal processing, thereby reducing the electrical energy consumption of the vehicle 1. However, this may reduce the responsiveness of the opening operation to a user's request for an automatic door opening operation.

[0060] Next, Fig. 7 is a flowchart showing details of the process of step S202 in Fig. 5. In step S41, as preprocessing, the first calculation unit 53 of the DSP 31 calculates a detection point group as the position of an object based on multiple reception results by the millimeter-wave radar 32. For example, the first calculation unit 53 calculates the distance, speed, and angle of the detection point group from the IF signal captured by the millimeter-wave radar 32.

[0061] Here, a detection point refers to a point in three-dimensional space that sends back a reflected wave to the millimeter-wave radar 32. Normally, multiple detection points appear for one detection by the millimeter-wave radar 32 (described later in FIG. 8). These are called a detection point cloud.

[0062] The detection point cloud may include not only points that reflect actual obstacles such as people and vehicles, but also noise detection points known as false or virtual images. Noise detection points are often the result of millimeter waves emitted by the millimeter-wave radar 32 being reflected multiple times by structures such as road surfaces and building walls. Therefore, in most cases, there is nothing present at the location where a noise detection point appears. In conventional technology, it is not easy to distinguish between detection points that reflect actual obstacles and noise detection points, resulting in low accuracy in determining the presence or absence of an obstacle. Therefore, in this embodiment, the accuracy of determining the presence or absence of an obstacle is improved by performing the following processing from step S42 onwards. Details of the processing of each step will be described in Figure 8 and subsequent figures.

[0063] In step S42, the second calculation unit 54 converts the detection point group calculated in step S41 into three-dimensional coordinates, and sets at least one region of interest in the three-dimensional coordinates based on the detection point group.

[0064] Next, in step S43, the second calculation unit 54 calculates the feature amount of the detection point group in the set region of interest.

[0065] Next, in step S44, the second calculation unit 54 calculates a feature vector based on the feature amount.

[0066] Next, in step S45, the estimation unit 55 determines whether or not an obstacle is present based on the feature vector.

[0067] Next, an example of the detection point cloud will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram showing an example of the detection point cloud when an obstacle is present in the embodiment. Fig. 8 shows a detection example under the condition that a triangular cone (gray area) is placed as an obstacle in front of the millimeter-wave radar 32 and 60 cm away from the door 21.

[0068] 9 is a diagram showing an example of a detection point cloud when there is no obstacle in the embodiment, which shows a detection example when there is nothing around the door 21.

[0069] In Figures 8 and 9, the X axis is the coordinate axis along the longitudinal direction of the vehicle, the Y axis is the coordinate axis along the lateral direction of the vehicle, and the Z axis is the coordinate axis along the vertical direction. In addition, the black dashed line in each figure indicates the open trajectory of the automatic door (door 21). Each circle represents the position of one detection point. The size and color of the circle indicate the magnitude of the reflected energy value at that detection point. The data in Figures 8 and 9 was obtained on an asphalt road surface. As can be seen from Figures 8 and 9, a detection point cloud appears whether or not an obstacle is present.

[0070] Note that the processing step of calculating the distance, velocity, and angle of the detection point group from the IF signal is omitted here because it is a fundamental part of the millimeter-wave radar 32 and not a technical feature of this embodiment. As a result of this processing step, for each detection point, polar coordinate values ​​of the distance, velocity, and angle in a three-dimensional coordinate system (hereinafter referred to as the radar coordinate system) with the center of the millimeter-wave radar 32 as the origin, and reflected energy values ​​are obtained.

[0071] In the conversion process for use in determining the presence or absence of an obstacle, first, for each detection point, coordinate conversion is performed from the radar coordinate system to a three-dimensional coordinate system centered on the door (hereinafter referred to as the door coordinate system). The origin of the door coordinate system is located on the surface of the door 21 of the vehicle 1, and if a point that is likely to collide with an obstacle is selected, it is easier to calculate the door movement angle. In this case, a calculation is required to eliminate the offset from the center of the millimeter-wave radar 32 embedded inside the door 21. Furthermore, if the millimeter-wave radar 32 is set at an angle inside the door 21, a coordinate rotation process is performed to eliminate the angle. After the coordinate conversion, conversion to a Cartesian coordinate system and noise reduction process are performed as necessary. The noise reduction process may involve, for example, temporal averaging or spatial averaging to reduce the number of noise detection points.

[0072] Next, setting of a region of interest will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is a diagram showing an example of setting a region of interest when an obstacle is present in the embodiment. Fig. 11 is a diagram showing an example of setting a region of interest when no obstacle is present in the embodiment. In step S42 of Fig. 7, a process is performed to extract a necessary space to determine whether an obstacle is present. This space is called a region of interest.

[0073] Figure 10 shows an example of setting a region of interest (ROI) based on the detection point cloud data shown in Figure 8. Figure 11 shows an example of setting a region of interest (ROI) based on the detection point cloud data shown in Figure 9. In these examples, the detection point with the highest reflected energy value among the detection point clouds was set as the center (marked with an x) of the region of interest (ROI). The size of the region of interest (ROI) was set to a cube with each side measuring 0.9 m.

[0074] The method for determining the center of the region of interest is not limited to this, and other methods may be used, such as using the center of gravity, average value, or median value of the three-dimensional coordinates of the detection point cloud. The method for determining the center and size of the region of interest may be determined arbitrarily, for example, based on the discrimination accuracy described below. Furthermore, in order to determine the presence or absence of multiple obstacles around door 21, two or more regions of interest may be set, and processing may be performed for each region of interest.

[0075] Next, extraction of feature amounts will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a diagram showing an example of feature amounts extracted when an obstacle is present in the embodiment. Fig. 13 is a diagram showing an example of feature amounts extracted when no obstacle is present in the embodiment.

[0076] In step S43 of Fig. 7, processing is performed to extract the characteristics of the distribution pattern of the detection point cloud within the region of interest as a preliminary process for determining the presence or absence of an obstacle. Fig. 12 shows an example in which the characteristics of the distribution pattern of the detection point cloud are obtained based on the X / Y / Z coordinate values ​​of each detection point within the region of interest (ROI) shown in Fig. 10. In this example, sections are set every 0.1 m, with the center of the region of interest (ROI) as the base, and the maximum reflected energy value of the detection point cloud within each section is obtained and plotted as a histogram. Fig. 13 shows the results of similar processing performed on each detection point within the region of interest (ROI) shown in Fig. 11.

[0077] As can be seen by comparing Figures 12 and 13, in Figure 12, when an obstacle is present, at least (a) has a clear peak in the histogram, and (b) and (c) tend to have high bin heights overall. On the other hand, in Figure 13, when no obstacle is present, there is no (or only a small) tendency for such a clear peak in the histogram or high bin heights overall. Therefore, it can be seen that the presence or absence of an obstacle can be determined with high accuracy based on such feature amounts.

[0078] Next, an example of a feature vector will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of a feature vector created in the embodiment. In step S44 of Fig. 7, a process of creating a feature vector from feature amounts extracted from the distribution pattern of the detection point cloud is performed as a pre-stage process for determining the presence or absence of an obstacle.

[0079] Figure 14(a) shows the feature vector corresponding to Figure 12. Figure 14(b) shows the feature vector corresponding to Figure 13. Here, we show an example of a feature vector created by simply connecting the extracted features. Specifically, each of the three histograms consisted of nine intervals, so the created feature vector was 27 dimensions in length.

[0080] Next, FIG. 15 is a graph for comparing two feature vectors in this embodiment. That is, FIG. 15 is a graph in which feature vectors based on detection point clouds obtained from previously accumulated tests under conditions with and without obstacles are superimposed for comparison. Graph G1 is the average of the feature vectors obtained under the condition with an obstacle. Graph G2 is the average of the feature vectors obtained under the condition without an obstacle. The error bars corresponding to each graph indicate the standard deviation. Each feature vector is normalized based on the maximum amplitude of all feature vectors.

[0081] Here, the obstacle-present condition was detection data when obstacles such as people, vehicles, stairs, metal poles, traffic cones, and curbs were placed 0.2 m to 1.4 m away from the door. Clear differences can be found in the size and shape of the feature vectors between the two conditions.

[0082] In order to determine whether an obstacle is present, it is important to create feature vectors in such a way that the differences between them become clear. If necessary, instead of simply connecting feature vectors as in the above example, a processing step called feature engineering may be added to create new feature vectors from these feature vectors. One example of feature engineering is edge enhancement processing. By emphasizing edges, this may further improve the accuracy of determining whether an obstacle is present.

[0083] Next, the performance of a plurality of machine learning machines in determining the presence or absence of an object will be described with reference to Fig. 16. Fig. 16 is a comparison table showing the performance of a plurality of machine learning machines in determining the presence or absence of an object in an embodiment.

[0084] In step S45 of Fig. 7, a process of determining the presence or absence of an obstacle is performed using a machine learning machine. Fig. 16 shows the results of testing the determination of the presence or absence of an obstacle based on the group of feature vectors exemplified in Fig. 15, using representative machine learning machines such as LightGBM (Gradient Boosting Machine), k-nearest neighbor algorithm, random forest algorithm, decision tree algorithm, and support vector machine.

[0085] The numbers in the table show the average value of each index after 10-fold cross-validation. The accuracy rate of classification differed depending on the machine learning machine used, with the lowest being 91.9% when using a support vector machine. The highest was 97.4% when using LightGBM.

[0086] There was also variation in accuracy in metrics such as recall, precision, and F1, likely reflecting the characteristics of each machine learning device. The learning time also varied widely, ranging from 0.048 seconds to 0.710 seconds per run. A standard Windows® computer was used for this calculation.

[0087] These results suggest that by using a machine learning machine, it is possible to determine with extremely high accuracy whether or not an obstacle that may hinder the opening of an automatic door actually exists, based on the distribution pattern of the detection point cloud data captured by the millimeter-wave radar 32. When selecting a machine learning machine, it is considered best to choose one that can achieve higher discrimination accuracy with shorter learning and estimation times, taking into consideration the performance of the DSP 31 that can be installed in the vehicle 1.

[0088] Furthermore, no matter how good the feature extraction and feature engineering, it may be difficult to create a machine learning machine that consistently achieves a 100% accuracy rate in a single classification. In such cases, it is possible to combine the classification results obtained over multiple past cycles and then perform the final classification. For example, a machine learning machine with a single classification accuracy rate of 97% has a 3% chance of giving a false positive or false negative answer, but the probability of making an incorrect classification two times in a row is 0.09%. The probability of making an incorrect classification three times in a row is only 0.0027%. By implementing this technique, it is expected that the accuracy rate for determining the presence or absence of obstacles will be nearly 100%.

[0089] Furthermore, while the above describes the use of a single machine learning machine to determine whether an obstacle is present or not, it is also possible to use an ensemble learning method in which multiple machine learning machines are used in parallel to perform the determination and reach a consensus. In this case, the bias and variance of the determination are minimized, which may further improve the determination accuracy.

[0090] Next, the details of the process of step S204 in Fig. 5 will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the details of the process of step S204 in Fig. 5.

[0091] In step S204, a process is performed to calculate the door movement angle in order to estimate whether there is still room for the door 21 to continue opening before it hits an obstacle. Here, if the door movement angle = 0, it means that the door 21 will hit an obstacle.

[0092] In step S51, the control unit 56 of the DSP 31 calculates the door movement angle based on the position information of the obstacle.

[0093] In step S52, the control unit 56 determines the type of obstacle.

[0094] 18 is an explanatory diagram of the calculation of the door movement angle in this embodiment. First, the process begins by determining the key detection point. It is preferable to select the detection point closest to the door 21 as the key detection point. However, there is a possibility that a noise detection point that appears closer to the actual obstacle position may be selected. This possibility can be reduced, for example, by selecting the detection point closest to the door 21 from among those that have a reflected energy value above a certain threshold.

[0095] Next, a virtual infinite wall (y = y) is created based on the distance (Y coordinate value) from the door 21 to the key detection point. min) is set. For example, the space between this infinite wall and the current position of the door 21 is defined as the space in which the door 21 can continue to open. However, if this is defined, the problem occurs that the closer an obstacle is to the hinge side of the door 21, the smaller the calculated space in which the door 21 can continue to open. This is an effective safety measure to deal with the fundamental problem that it is difficult for the millimeter wave radar 32 to accurately detect the extent of an obstacle.

[0096] The intersection point (x) of the infinite wall and the open orbit of door 21 (black dashed line) p ,y min ) and door hinge (x H , 0) and the door surface, the angle θ shown in Figure 18 p is calculated as the movable angle using the following (Equation 1).

number

[0097] Next, Fig. 19 is an explanatory diagram of object type discrimination based on the statistical quantity of the distribution of the detection point cloud in this embodiment. In step S52 of Fig. 17, a process is performed to discriminate the type of obstacle based on the distribution pattern of the detection point cloud. Various types are possible, but three discrimination methods are exemplified below. Note that the type of obstacle does not have to be discriminated based on the results of performing these methods individually, and they may be combined as necessary.

[0098] (Type discrimination method 1) Discrimination based on the statistical quantity of the distribution of detection points The distribution pattern of the detection point cloud differs between an obstacle with a small width and depth, such as a metal pole (the gray part in Figure 19(a)), and an obstacle with a large width and depth, such as a staircase (the gray part in Figure 19(b)). For convenience, the following explanation will refer to the former as a simple shape and the latter as a complex shape.

[0099] For simple shapes, the detected points are distributed in a narrow area. On the other hand, for complex shapes, the detected points tend to be distributed over a wider area. Therefore, the two types are distinguished based on the distribution statistics.

[0100] For example, the variance value of the detected point cloud data (V x , V y , V z ) is calculated, where N is the number of detected point cloud data, (x c , y c , z c ) is the center of the distribution.

number

[0101] Then, as shown in the following equation (4), the variance value of the detected point cloud data (for example, V y ) and threshold THD_V y and compare it with the threshold THD_V y If smaller than this, simple geometry, THD_V y If it is larger, it is determined to have a complex shape.

[0102] if (V y <THD_V y ) (Equation 4) # Obstacles are simple shapes else # Obstacles have complex shapes

[0103] (Type discrimination method 2) Discrimination based on the geometric characteristics of the distribution of detected point clouds The difference in distribution of detected point clouds for simple shapes and complex shapes is as explained above, but from a different perspective, the two shapes are distinguished from their geometric features.

[0104] FIG. 20 is an explanatory diagram of object type discrimination based on the geometric characteristics of the distribution of the detection point cloud in this embodiment. For example, the least squares line shown in the following (Equation 5) is found from the distribution pattern of the detection point cloud on the YZ plane. Furthermore, the coefficients c and d of the least squares line can be found by solving (Equation 6). Here, w i is a weighting coefficient, and by adopting the normalized value of the reflected energy at each detection point, it is possible to obtain a least-squares line that reflects the strength of the reflected energy.

number

[0105] Next, using (Equation 7), the unit normal vector NV of the least squares line of (Equation 5) is calculated. o Also, using (Equation 8), the unit normal vector NV of the radar plane is calculated. R = the angle θ formed by (1,0) YZ Calculate.

number

[0106] Then, as shown in equation (9), the angle θ YZ and the threshold interval [THD_θ YZ1 ,THD_θ YZ2 ] and if it is within that range, it is determined to be a simple shape, and if it is outside that range, it is determined to be a complex shape. if (THD_θ YZ1 )<θ YZ ) and (θ YZ <THD_θ YZ2 ) (Formula 9) # Obstacles are simple shapes else # Obstacles have complex shapes

[0107] (Type discrimination method 3) Discrimination based on the reflected energy value of the distribution of detection points A fundamental feature of millimeter-wave radar 32 is that the smaller the radar cross section and the less reflective the material (e.g., plastic resin), the smaller the detected reflected energy value. Conversely, the larger the radar cross section and the less reflective the material (e.g., iron), the larger the detected reflected energy value. According to the above classification, the former corresponds to a simple shape, and the latter corresponds to a complex shape. Therefore, discrimination is made based on the reflected energy value.

[0108] FIG. 21 is an explanatory diagram of object type discrimination based on the reflected energy value of the distribution of a detection point cloud in an embodiment. The discrimination method in this case uses machine learning technology. For example, when processing similar to steps S41 to S44 in FIG. 7 is performed on each piece of detection data, and a feature vector G12 obtained from an obstacle classified as a simple shape (e.g., a metal pole, a traffic cone, a curb, etc.) is compared with a feature vector G11 obtained from an obstacle classified as a complex shape (e.g., a person, a vehicle, a staircase, etc.), differences are found as shown in FIG. 21. Note that the error bars corresponding to each graph indicate standard deviation.

[0109] Fig. 22 is a comparison table showing the object type discrimination performance of multiple machine learning machines in this embodiment. When an attempt was made to discriminate the type of new (unlearned) data using a machine learning machine that had been trained on past data by performing the same process as in step S45 of Fig. 7, it was possible to distinguish between simple and complex shapes with a maximum accuracy rate of 85.4%.

[0110] Returning to FIG. 17, after steps S51 and S52, in step S53, the control unit 56 of the DSP 31 performs processing to combine the results of steps S51 and S52, arbitrarily determine and set the final door movement angle.

[0111] For example, if the obstacle has a simple shape and it is predicted that the damage will be minor if it collides with it, it is possible to determine that the door will operate at the calculated door movement angle. Conversely, if the obstacle has a complex shape and it is predicted that the damage will be serious if it collides with it, it is possible to determine that the door movement angle will be very small (for example, only pop-up (the operation of unlocking the door 21 and freeing it) will be performed).

[0112] In this way, according to the automatic door system S of this embodiment, by using an object detection model generated in advance by machine learning, obstacles around the vehicle 1 can be detected with high accuracy regardless of the surrounding environment such as the road surface.

[0113] Furthermore, a more specific process can be realized in which a region of interest (Fig. 10, Fig. 11) is set using three-dimensional coordinates based on the door and a feature vector showing the distribution shape of the detection point cloud is calculated.

[0114] Furthermore, by opening the door 21 up to the set opening angle, it is possible to avoid collision of the door 21 with an obstacle and also to prevent the door 21 from unnecessarily stopping its opening operation short of the target.

[0115] Furthermore, the vehicle user can open the door 21 by performing a predetermined operation corresponding to a request for automatic opening of the door 21.

[0116] Although the embodiments of the present invention have been described above, the above-described embodiments and modifications are merely examples and are not intended to limit the scope of the invention. The above-described embodiments and modifications can be implemented in various other forms, and various omissions, substitutions, combinations, and modifications can be made without departing from the spirit of the invention. Furthermore, the configurations and shapes of the embodiments and modifications can also be partially interchanged.

[0117] For example, in the above embodiment, the automatic door system S has one ECU 23 (FIG. 2), but is not limited to this. The automatic door system S may have multiple ECUs.

[0118] Furthermore, some of the functions of the DSP 31 (for example, the model generating unit 52) ​​may be provided in the ECU 23.

[0119] Also, in FIG. 20, a least squares straight line is used as an example, but the present invention is not limited to this, and a least squares curve or a least squares surface may also be used.

[0120] Furthermore, the object detection sensor is not limited to a millimeter wave radar, but may be another type of sensor such as an ultrasonic sensor.

[0121] Furthermore, data of a feature vector newly determined by machine learning may be updated as a comparison target for the next comparison.

[0122] In the above-described embodiment, the object detection sensor is installed on the vehicle 10, but the present invention is not limited to this. The object detection sensor can be installed on any mobile object, such as a mobile robot, whose surrounding environment changes from moment to moment as it moves. [Explanation of symbols]

[0123] 1...vehicle, 2...automatic door unit, 3...sensor unit, 4...in-vehicle network, 5...processing unit, 6...storage unit, 21...door, 22...door drive unit, 23...ECU, 31...DSP, 32...millimeter wave radar, 51...acquisition unit, 52...model generation unit, 53...first calculation unit, 54...second calculation unit, 55...estimation unit, 56...control unit, S...automatic door system

Claims

1. an acquisition unit that acquires a plurality of reception results of reflected waves generated when a search wave transmitted from a sensor installed in a door of the vehicle is reflected by objects around the vehicle; During the learning phase, a model generation unit that calculates a detection point cloud as the position of the object based on the plurality of reception results acquired by the acquisition unit, and generates an object detection model by machine learning the relationship between a feature vector indicating a distribution shape of the detection point cloud and information indicating whether the object is an obstacle; and In the estimation phase, a first calculation unit that calculates a detection point cloud as the position of the object based on the plurality of reception results acquired by the acquisition unit; a second calculation unit that calculates a feature vector indicating a distribution shape of the detection point cloud based on the detection point cloud calculated by the first calculation unit; an estimation unit that determines whether the object is an obstacle based on the feature vector calculated by the second calculation unit and the object detection model, and outputs a determination result; the model generation unit converts the detection point cloud into three-dimensional coordinates based on the door on which the sensor is installed, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest, and uses the calculated feature vector as input data; the second calculation unit converts the detection point cloud calculated by the first calculation unit into the three-dimensional coordinate system, sets at least one region of interest in the three-dimensional coordinate system based on the detection point cloud, and calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest.

2. The door is a swing door, The object detection device further includes a control unit that controls a drive unit that opens and closes the door, 2. The object detection device according to claim 1, wherein, when the estimation unit outputs information indicating that the object is an obstacle, the control unit sets an opening movable angle of the door based on position information of the obstacle, and controls the drive unit so that the door opens to the set opening movable angle.

3. 3. The object detection device according to claim 2, wherein the control unit controls the drive unit to open the door to the set movable opening angle based on a request for an automatic opening operation of the door from a user of the vehicle.

4. an acquisition step of acquiring a plurality of reception results of reflected waves generated when a search wave transmitted from a sensor installed in a door of the vehicle is reflected by objects around the vehicle; During the learning phase, a model generation step of calculating a detection point cloud as the position of the object based on the plurality of reception results acquired in the acquisition step, and generating an object detection model by machine learning the relationship between a feature vector indicating the distribution shape of the detection point cloud and information indicating whether the object is an obstacle; and In the estimation phase, a first calculation step of calculating a detection point cloud as the position of the object based on the plurality of reception results acquired in the acquisition step; a second calculation step of calculating a feature vector indicating a distribution shape of the detection point cloud based on the detection point cloud calculated in the first calculation step; an estimation step of determining whether or not the object is an obstacle based on the feature vector calculated in the second calculation step and the object detection model, and outputting a determination result; the model generation step converts the detection point cloud into three-dimensional coordinates based on the door on which the sensor is installed, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest, and uses the calculated feature vector as input data; the second calculation step performs coordinate transformation of the detection point cloud calculated in the first calculation step into the three-dimensional coordinates, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, and calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest.

5. an acquisition step of acquiring a plurality of reception results of reflected waves generated when a search wave transmitted from a sensor installed on a swing door of the vehicle is reflected by objects around the vehicle; During the learning phase, a model generation step of calculating a detection point cloud as the position of the object based on the plurality of reception results acquired in the acquisition step, and generating an object detection model by machine learning the relationship between a feature vector indicating the distribution shape of the detection point cloud and information indicating whether the object is an obstacle; and In the estimation phase, a first calculation step of calculating a detection point cloud as the position of the object based on the plurality of reception results acquired in the acquisition step; a second calculation step of calculating a feature vector indicating a distribution shape of the detection point cloud based on the detection point cloud calculated in the first calculation step; an estimation step of determining whether the object is an obstacle based on the feature vector calculated in the second calculation step and the object detection model, and outputting a determination result; a control step of, when information indicating that the object is an obstacle is output by the estimation step, setting an opening movable angle of the door based on position information of the obstacle, and controlling a drive unit that opens and closes the door so that the door opens to the set opening movable angle, the model generation step converts the detection point cloud into three-dimensional coordinates based on the door on which the sensor is installed, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest, and uses the calculated feature vector as input data; the second calculation step performs coordinate transformation of the detection point cloud calculated in the first calculation step into the three-dimensional coordinates, sets at least one region of interest in the three-dimensional coordinates based on the detection point cloud, and calculates a feature vector indicating a distribution shape of the detection point cloud in the set region of interest.

Images