Vehicle door opening warning method

Abstract

The application discloses a vehicle door opening early warning method, comprising the following steps: S1: acquiring a surround view image of a vehicle, extracting and screening corner points of the surround view image, and obtaining trajectories of the corner points of the surround view image through an optical flow method; S2: setting a scanning sliding window with a predetermined period, obtaining a waveform graph of time and longitudinal displacement and a longitudinal displacement difference value sequence; S3: calculating a reference quantity of a longitudinal displacement change rate, if the reference quantity is less than or equal to a reference quantity threshold value, judging whether the trajectory is a risk trajectory according to a start and end direction of the trajectory; otherwise, judging whether the trajectory is a risk trajectory according to a number of wave crests and wave troughs in the waveform graph, a number of rising edges and falling edges, and the start and end direction of the trajectory; S4: acquiring a number of risk trajectories marked in a predetermined region of the surround view image, if the number of risk trajectories in the predetermined region is greater than a first number threshold value, the vehicle generates an alarm signal; otherwise, the vehicle does not generate an alarm signal. The method can reduce the false alarm probability caused by non-rigid object mixed trajectories.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a vehicle door opening warning method. BACKGROUND

[0002] Improper opening of a vehicle door when the vehicle is parked will cause the oncoming motor vehicle, non-motor vehicle and pedestrian to collide with the door, resulting in vehicle damage and even casualties. The door opening warning function refers to when passengers get on or off the vehicle, if there are objects such as motor vehicles, non-motor vehicles and pedestrians approaching from the rear, the vehicle can timely detect the dangerous target by means of cameras, radars and other sensors, and give a warning through sound or light, so as to avoid the danger. The existing door opening warning device mostly uses radars or cameras, and the scheme using cameras has better cost advantage.

[0003] On a low-cost computing platform, the fisheye camera generated surround view is used for door opening warning detection, and the optical flow algorithm can effectively determine the motion trend information of the rigid object and make timely alarm. However, for the non-rigid object of the pedestrian, the projection transformation in the conversion process from the fisheye image to the surround view image will greatly stretch and distort the objects higher than the ground around the vehicle, and the body movement of the pedestrian may form a complex and large jump trajectory on the surround view, which is easy to induce false alarm in the scene where the pedestrian passes through the rear of the vehicle laterally. SUMMARY

[0004] The present application aims to solve the problem of false alarm triggered by the non-rigid object of the pedestrian behind the vehicle in the prior art. The present application provides a vehicle door opening warning method, which effectively eliminates the false alarm problem caused by the complex trajectory of the non-rigid object while retaining the alarm for the non-rigid object of the pedestrian behind the vehicle.

[0005] The present application discloses a vehicle door opening warning method, comprising the following steps:

[0006] S1: acquiring the surround view images on the left and right sides behind the vehicle, extracting the corner points in the surround view images, and screening the extracted corner points in the surround view images, and obtaining the trajectories of the corner points in the surround view images by the optical flow method;

[0007] S2: obtaining the longitudinal displacement of the corner points at each time in the trajectory by setting a scanning sliding window with a predetermined period, obtaining the waveform graph of time and longitudinal displacement, and obtaining the longitudinal displacement difference value of the corner points in each predetermined period to form a longitudinal displacement difference value sequence;

[0008] S3: calculating the reference quantity of the longitudinal displacement change rate according to the longitudinal displacement difference value sequence, and judging whether the trajectory is a risk trajectory according to the reference quantity; wherein,

[0009] if the reference quantity is less than or equal to the reference quantity threshold value, determining whether the trajectory is a risk trajectory according to a start-to-end direction of the trajectory; wherein if yes, marking the trajectory as a risk trajectory, and if no, marking the trajectory as a non-risk trajectory;

[0010] if the reference quantity is greater than the reference quantity threshold value, determining whether the trajectory is a risk trajectory according to a number of wave crests and a number of wave troughs, a number of rising edges and a number of falling edges in the waveform graph, and a start-to-end direction of the trajectory; wherein if yes, marking the trajectory as a risk trajectory, and if no, marking the trajectory as a non-risk trajectory;

[0011] S4: obtaining a number of the marked risk trajectories in a predetermined region of the surround view image, and determining whether the vehicle generates an alarm signal according to the number of the risk trajectories in the predetermined region; wherein,

[0012] if the number of the risk trajectories in the predetermined region is greater than a first number threshold value, the vehicle generates the alarm signal;

[0013] if the number of the risk trajectories in the predetermined region is less than or equal to the first number threshold value, the vehicle does not generate the alarm signal.

[0014] By using the above technical solution, the surround view images of the left and right sides behind the vehicle are obtained, the corner points in the surround view images are extracted, the corner points in the extracted surround view images are screened, the trajectories of the corner points in the surround view images are obtained by using the optical flow method, the longitudinal displacements of the corner points at each time in the trajectories are obtained by setting a scanning sliding window with a predetermined period, the waveform graph of the time and the longitudinal displacements is obtained, the longitudinal displacements of the corner points in each predetermined period in the trajectory are obtained to form a longitudinal displacement sequence, and two different methods of determining whether the trajectory is a risk trajectory are set according to the size relationship between the reference quantity and the reference quantity threshold value, i.e. according to the fluctuation of the trajectory. If the reference quantity is less than or equal to the reference quantity threshold value, i.e. the trajectory is relatively stable, it is only necessary to determine whether the trajectory is a risk trajectory according to the start-to-end direction of the trajectory, and if the reference quantity is greater than the reference quantity threshold value, i.e. the trajectory fluctuates obviously, it is necessary to determine whether the trajectory is a risk trajectory according to the number of wave crests and the number of wave troughs, the number of rising edges and the number of falling edges in the waveform graph, and the start-to-end direction of the trajectory. In this way, whether the trajectory is a risk trajectory can be determined more accurately, and the false alarm probability caused by the mixed trajectories of non-rigid objects can be reduced. After determining whether the trajectory is a risk trajectory, it is necessary to determine whether the number of the risk trajectories in the predetermined region is greater than a first number threshold value, and if yes, the vehicle generates an alarm signal. In this way, the alarm signal generated can give a safety prompt to the passenger or the driver who is preparing to open the door and get off the vehicle, and the danger can be avoided.

[0015] According to another specific embodiment of the present invention, in the vehicle door opening warning method disclosed in the embodiment of the present invention, in step S3, if the reference quantity is greater than the reference quantity threshold, and the sum of the number of peaks and the number of troughs in the waveform is greater than the second quantity threshold, the number of rising edges is greater than or equal to the number of falling edges, and the beginning and end directions of the trajectory are forward, then the trajectory is a risk trajectory; otherwise, the trajectory is a non-risk trajectory.

[0016] Using the above technical solution, if the reference value is greater than the reference value threshold, it indicates that the trajectory has fluctuated significantly. To reduce false alarms, even if the initial and final directions of the trajectory meet the requirements, it is necessary to combine the relationship between the sum of the number of peaks and troughs in the waveform and the second quantity threshold, as well as the relationship between the number of rising edges and the number of falling edges, to determine whether the trajectory is a risky trajectory. This can correctly issue alarms for pedestrians and non-rigid objects approaching vehicles, effectively reducing the probability of false alarms. Specifically, if the sum of the number of peaks and troughs in the waveform is greater than the second quantity threshold, it indicates that enough motion characteristics of the target object's corner trajectory have been observed, preventing the system from prematurely issuing risk judgments due to occasional fluctuations.

[0017] According to another specific embodiment of the present invention, in the vehicle door opening warning method disclosed in the embodiment of the present invention, if the reference quantity is less than or equal to the reference quantity threshold and the starting and ending directions of the trajectory are forward, the trajectory is a risky trajectory; otherwise, the trajectory is a non-risky trajectory.

[0018] Using the above technical solution, if the reference value is less than or equal to the reference value threshold, it indicates that the trajectory is relatively stable. If the starting and ending directions of the trajectory are forward, it can be directly determined that the trajectory is a risky trajectory. This allows for a quick determination of whether the trajectory is a risky trajectory. If the trajectory is a risky trajectory, the vehicle can promptly issue an alarm signal to warn of rapidly approaching objects and provide safety reminders to passengers, thereby preventing the occurrence of danger.

[0019] According to another specific embodiment of the present invention, in the vehicle door opening warning method disclosed in this embodiment, step S3 involves calculating the mean and variance of the longitudinal displacement difference sequence, and calculating a reference value for the longitudinal displacement change rate according to the following formula:

[0020] Y' var =Y var / Y mean 2

[0021] Among them, Y' var Y is a reference value for the rate of change of longitudinal displacement; var Y is the variance of the longitudinal displacement difference sequence; mean This is the mean of the longitudinal displacement difference sequence.

[0022] The reference quantity of the longitudinal displacement change rate is related to the square of the variance and the mean of the longitudinal displacement difference value sequence, and the reference quantity is used to describe the comprehensive fluctuation intensity of the trajectory. For objects in high-speed motion or low-speed motion, the same reference quantity threshold can accurately divide the trajectory into a fluctuation trajectory and a non-fluctuation trajectory.

[0023] According to another specific embodiment of the present application, the vehicle door opening warning method disclosed by the embodiment of the present application comprises the following steps:

[0024] According to the above technical scheme, the time complexity of the FAST corner point detection calculation is small, and the detection effect is outstanding.

[0025] According to another specific embodiment of the present application, the vehicle door opening warning method disclosed by the embodiment of the present application comprises the following steps:

[0026] According to the above technical scheme, the four-tree equalization algorithm can improve the uniformity of the distribution of the extracted corner points in each frame of the surround view image.

[0027] According to another specific embodiment of the present application, the vehicle door opening warning method disclosed by the embodiment of the present application comprises the following steps:

[0028] S11: After obtaining the current frame of the surround view image, the distance between each corner point extracted from the previous frame of the surround view image and each corner point in the corner point management list is calculated respectively; wherein the current frame of the surround view image is set as the t-th frame of the surround view image, and t is greater than or equal to 3;

[0029] S12: It is judged whether there is a corner point in the previous frame of the surround view image whose distance to the corner point in the corner point management list is less than the distance threshold;

[0030] If yes, the hit count of the corresponding corner point in the corner point management list is increased by 1, and the upper limit of the hit count is M, wherein M is a positive integer;

[0031] If no, the hit count of the corresponding corner point in the corner point management list is decreased by 1, and the lower limit of the hit count is 0;

[0032] S13: It is judged whether there is a corner point in the previous frame of the surround view image whose distance to each corner point in the corner point management list is greater than or equal to the distance threshold;

[0033] If yes, the corresponding corner point is added to the corner point management list, and the hit count of the corresponding corner point is M;

[0034] S14: deleting the corner point with the hit number of 0 in the corner point management list to obtain a new corner point management list, and continuing to obtain the trajectory of the corner point in the current frame surround view image by using the optical flow method.

[0035] By using the above scheme, if there is a corner point in the previous frame surround view image which has a distance less than the distance threshold to the corner point in the corner point management list, the hit number of the corresponding corner point in the corner point management list is increased by 1, if there is no corner point in the previous frame surround view image which has a distance less than the distance threshold to the corner point in the corner point management list, the hit number of the corresponding corner point in the corner point management list is decreased by 1, and if there is a corner point in the previous frame surround view image which has a distance greater than or equal to the distance threshold to each corner point in the corner point management list, the corresponding corner point is added to the corner point management list, and the hit number of the corresponding corner point is M, so that it can be ensured that there is no a large number of points which are too close to each other in the corner point management list, thereby avoiding repeated calculation of the trajectories of the points which are too close to each other, and reducing the memory and calculation overheads.

[0036] According to another specific embodiment of the present application, the vehicle door opening warning method disclosed by the embodiments of the present application comprises the following steps:

[0037] S15: tracking the corner point in the new corner point management list by using the LK optical flow method between the current frame surround view image and the previous frame surround view image, and judging whether the tracking of the corner point is successful or not;

[0038] If the tracking is successful, the corresponding corner point and the trajectory of the corresponding corner point are recorded;

[0039] If the tracking fails, the corresponding corner point and the trajectory of the corresponding corner point in the new corner point management list are deleted;

[0040] The trajectory of the corner point in the current frame surround view image is obtained, and then the step S2 is continued.

[0041] By using the above scheme, the performance of the LK optical flow method for tracking the corner point is relatively optimal, the operation speed is relatively fast, and the trajectory of the corner point in the surround view image can be quickly obtained.

[0042] According to another specific embodiment of the present application, the vehicle door opening warning method disclosed by the embodiments of the present application comprises the following steps:

[0043] The fisheye images of the left and right sides of the rear of the vehicle are obtained by using the fisheye cameras on the vehicle, and the fisheye images of the left and right sides of the rear of the vehicle are formed into the surround view images of the left and right sides of the rear of the vehicle after being spliced.

[0044] The vehicle rear left and right fisheye images are acquired by the fisheye cameras on the left and right sides of the vehicle, so that the surround view images are obtained, and the movement of the objects on the left and right sides of the rear of the vehicle can be monitored in real time.

[0045] According to another specific embodiment of the present application, the vehicle door opening warning method disclosed by the embodiment of the present application has the following advantages.

[0046] The present application has the following advantages.

[0047] The present application provides a vehicle door opening warning method, which acquires surround view images of the left and right sides of the rear of the vehicle, extracts corner points in the surround view images, screens the extracted corner points in the surround view images, and obtains trajectories of the corner points in the surround view images by using an optical flow method. Then, a scanning sliding window with a predetermined period is set, the longitudinal displacement of the corner points at each time in the trajectories is acquired, a waveform graph of time and the longitudinal displacement is obtained, the longitudinal displacement of the corner points in each predetermined period in the trajectories is acquired to form a longitudinal displacement sequence, and the size relationship between a reference quantity and a reference quantity threshold value is used, i.e., two different methods of judging whether the trajectories are risk trajectories are set according to the fluctuation of the trajectories. If the reference quantity is less than or equal to the reference quantity threshold value, i.e., the trajectories are relatively stable, it is only necessary to judge whether the trajectories are risk trajectories according to the start and end directions of the trajectories. If the reference quantity is greater than the reference quantity threshold value, i.e., the trajectories fluctuate obviously, it is necessary to judge whether the trajectories are risk trajectories according to the number of wave crests and wave troughs in the waveform graph, the number of rising edges and falling edges, and the start and end directions of the trajectories. In this way, whether the trajectories are risk trajectories can be determined more accurately, and the false alarm probability caused by the non-rigid object mixed trajectories can be reduced. After judging whether the trajectories are risk trajectories, it is necessary to judge whether the number of risk trajectories in a predetermined region is greater than a first number threshold value. If the number of risk trajectories is greater than the first number threshold value, the vehicle generates an alarm signal. In this way, the alarm signal can give a safety prompt to the passenger or driver who is preparing to open the door and get off the vehicle, and the danger can be avoided. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flow chart of the vehicle door opening warning method of the embodiment of the present application is shown in the figure.

[0049] Figure 2 The surround view image splicing schematic diagram of the vehicle door opening warning method of the embodiment of the present application is shown in the figure.

[0050] Figure 3 The trajectory schematic diagram of the corner points of the rigid object of the vehicle door opening warning method of the embodiment of the present application is shown in the figure.

[0051] Figure 4 The trajectory schematic diagram of the corner points of the non-rigid object of the vehicle door opening warning method of the embodiment of the present application is shown in the figure.

[0052] Figure 5 Waveform diagram of time and longitudinal displacement of the corner point for the vehicle door opening warning method of the embodiment of the present application;

[0053] Figure 6 Flowchart of screening the corner points in the surround view image extracted in the vehicle door opening warning method of the embodiment of the present application. DETAILED DESCRIPTION

[0054] Other advantages and effects of the present application can be easily understood by those skilled in the art from the content disclosed in the specification. Although the description of the present application will be introduced in combination with the preferred embodiments, it does not mean that the features of the present application are limited to the embodiments. On the contrary, the purpose of introducing the present application in combination with the embodiments is to cover other options or modifications which can be extended based on the claims of the present application. In order to provide a deep understanding of the present application, many specific details will be included in the following description. The present application can also be implemented without using these details. In addition, in order to avoid confusion or obscure the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0055] It should be noted that in the specification, similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0056] In the description of the present embodiment, it should be noted that the terms "upper", "lower", "inner", "bottom" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0057] The terms "first", "second", and the like are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0058] In the description of the embodiments, it also needs to be explained that, unless otherwise explicitly specified and limited, the terms "arrange", "connect", "connect" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments can be understood according to the specific circumstances.

[0059] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0060] As shown in the figure, the embodiments of the present application disclose a vehicle door opening warning method, comprising the following steps: Figure 1

[0061] S1: Obtain the surround view images of the left and right sides behind the vehicle, extract the corner points in the surround view images, and screen the extracted corner points in the surround view images, and obtain the trajectories of the corner points in the surround view images by the optical flow method.

[0062] In a specific embodiment, obtaining the surround view images of the left and right sides behind the vehicle comprises: obtaining the fisheye images of the left and right sides behind the vehicle through the vehicle-mounted fisheye cameras on the left and right sides of the vehicle, and forming the surround view images of the left and right sides behind the vehicle after surround stitching of the fisheye images of the left and right sides behind the vehicle.

[0063] As shown in the figure, the surround view images containing only the fields of view of the left and right cameras are generated after surround stitching with the aid of the omnidirectional camera model internal parameter information, distortion parameter information and external parameter information of the vehicle-mounted fisheye cameras on the left and right sides. The longitudinal field of view of the surround view image starts flush with the rearview mirror, and the longitudinal physical length is L meters. The horizontal field of view starts at the edges of the left and right sides of the vehicle body, and the horizontal physical width is W meters on the left and right sides. Figure 2 In the embodiments, the corner points in the surround view images can be extracted by the commonly used corner point detection methods of those skilled in the art, such as Harris detection method, AHI-Tomasi detection method, SUSAN detection method, FAST detection method, etc. The embodiments do not make specific limitations thereon. For example, there is a square in the surround view image, and the extracted corner points can be the four corners of the square.

[0064] In the embodiments, the corner points in the adjacent two frames of surround view images can be tracked by the commonly used optical flow methods of those skilled in the art, such as LK (Lucas-Kanade) optical flow method, HS (Horn Schunck) optical flow method, etc., so as to obtain the trajectories of the corner points in the surround view images. For example,

[0065] Figure 3 ​​This is a schematic diagram of the trajectory of the corner point of a rigid object in the vehicle door opening warning method according to an embodiment of the present invention. Figure 3 In the diagram, black dots represent the corner points of a rigid object, and solid lines represent the trajectories of those corner points. Figure 3 As can be seen, the trajectories of each corner point of the rigid object are parallel to each other and move in the same direction. The dashed lines represent the outline of the rigid object. Figure 4 This is a schematic diagram of the trajectory of the corner point of a non-rigid object in the vehicle door opening warning method according to an embodiment of the present invention. Figure 4 The black dots represent the corner points of non-rigid objects, the solid lines represent the movement trajectories of the corner points of rigid objects, and the dashed lines represent the outlines of non-rigid objects. As can be seen from Figure 4, the trajectories of each corner point of a non-rigid object are intricate. Figure 3 and Figure 4 As shown, the motion of the corner points of a non-rigid object is more complex than that of a rigid object, and the trajectory of the corner points is more complex than that of a rigid object.

[0066] S2: By setting a scanning window with a predetermined period of win, the longitudinal displacement of the corner points at each moment in the trajectory is obtained, resulting in a waveform diagram of time versus longitudinal displacement, such as... Figure 5 As shown. For example, the longitudinal displacement of the corner point at time t is... Obtain the longitudinal displacement difference of the corner points within each predetermined period. Among them, the longitudinal displacement difference of the corner points The calculation formula is: To form a longitudinal displacement difference sequence.

[0067] In this embodiment, if the refresh rate (Frames Per Second, FPS) of the panoramic image is 30, i.e., 30 frames / second, and the predetermined period can be set to 2 frames, i.e., win = 2, then the longitudinal displacement difference of the corner points within each predetermined period is...

[0068] S3: Calculate the reference value for the longitudinal displacement change rate based on the longitudinal displacement difference sequence, and determine whether the trajectory is a risk trajectory based on the reference value; where,

[0069] If the reference value is less than or equal to the reference value threshold, determine whether the trajectory is a risk trajectory based on the start and end directions of the trajectory; if it is, mark the trajectory as a risk trajectory; if not, mark the trajectory as a non-risk trajectory.

[0070] If the reference value is greater than the reference value threshold, determine whether the trajectory is a risk trajectory based on the number of peaks and troughs, the number of rising edges and falling edges, and the beginning and end directions of the trajectory in the waveform diagram; if yes, mark the trajectory as a risk trajectory, otherwise mark the trajectory as a non-risk trajectory.

[0071] It should be noted that in the embodiment, for each wave crest, only when the positive displacement amount from the previous wave trough exceeds the threshold T up , is counted into the rising edge number. Similarly, for each wave trough, only when the negative displacement amount from the previous wave crest exceeds the threshold T down , is counted into the falling edge number. The value of T up may be the pixel number corresponding to 0.5 meters in the physical world on the surround view, and the value of T down may be the pixel number corresponding to 0.1 meters in the physical world on the surround view. In this way, the form and number of rising edges and falling edges are constrained. For example, if there is only one valid, very large positive rising edge in the trajectory, and there are many valid, small falling edges, the rising edge number and the falling edge number need to be determined according to the positive displacement amount of the wave trough, the threshold T up , the negative displacement amount of the wave crest, and T down , so such a trajectory is not necessarily a risk trajectory.

[0072] In the embodiment, the reference amount of the longitudinal displacement change rate can be determined according to the mean and variance of the longitudinal displacement difference value sequence. In the embodiment, the value of the reference amount threshold is in the range of 0.2-0.5, and can be 0.2, 0.3, 0.4, 0.5, or other values in the range of 0.2-0.5. The person skilled in the art can set it according to the actual situation, and the embodiment does not make a specific limitation.

[0073] S4: obtaining the number of marked risk trajectories in a predetermined region of the surround view image, and determining whether the vehicle generates an alarm signal according to the number of risk trajectories in the predetermined region; wherein,

[0074] If the number of risk trajectories in the predetermined region is greater than a first number threshold, the vehicle generates an alarm signal;

[0075] If the number of risk trajectories in the predetermined region is less than or equal to the first number threshold, the vehicle does not generate an alarm signal.

[0076] It should be noted that in the embodiment, the surround view image in step S1 is obtained frame by frame. After each new frame of surround view image is obtained, the vehicle door opening warning method can give a decision on whether to alarm, and the first frame of surround view image of the method is an exception. In addition, before step S2 is performed for the first time, a trajectory of a certain length needs to be obtained, so step S2 needs to be performed after step S1 is repeated several times after a new frame of surround view image is obtained. The person skilled in the art can set the number of repetitions according to the need, and before a trajectory of the required length is obtained, a decision on whether to alarm is not given each time a new frame of surround view image is obtained.

[0077] In this embodiment, the predetermined area can be set as the pixel size corresponding to the 1.5m*3m area in the physical world on the surround view. The first number threshold can be set as 3 or 4, which can be set by those skilled in the art according to the actual situation, and the embodiment does not make specific limitations. If the vehicle generates an alarm signal, the vehicle corresponding to the predetermined area generates an alarm signal, which can specifically give passengers safety prompts through the indicator light and the alarm sound in the vehicle.

[0078] By adopting the above scheme, the surround view images of the left and right sides behind the vehicle are obtained, the corner points in the surround view images are extracted, the trajectories of the corner points in the surround view images are obtained by the optical flow method, the longitudinal displacement of the corner points at each time in the trajectories is obtained by setting a scanning sliding window with a predetermined period, the waveform graph of time and longitudinal displacement is obtained, the longitudinal displacement of the corner points in each predetermined period in the trajectories is obtained to form a longitudinal displacement sequence, and two different methods of judging whether the trajectory is a risk trajectory are set according to the size relationship between the reference quantity and the reference quantity threshold, i.e. according to the fluctuation of the trajectory. If the reference quantity is less than or equal to the reference quantity threshold, i.e. the trajectory is relatively stable, it is only necessary to judge whether the trajectory is a risk trajectory according to the start and end directions of the trajectory, and if the reference quantity is greater than the reference quantity threshold, i.e. the trajectory fluctuates obviously, it is necessary to judge whether the trajectory is a risk trajectory according to the number of wave crests and wave troughs in the waveform graph, the number of rising edges and falling edges, and the start and end directions of the trajectory. In this way, whether the trajectory is a risk trajectory can be determined more accurately, and the false alarm probability caused by the non-rigid object mixed trajectory can be reduced. After judging whether the trajectory is a risk trajectory, it is necessary to judge whether the number of risk trajectories in the predetermined area is greater than the first number threshold, and if it is greater than the first number threshold, the vehicle generates an alarm signal. In this way, the passengers or drivers who are preparing to open the door and get off can be given safety prompts through the generated alarm signal, and the danger can be avoided.

[0079] In a specific embodiment, in step S3, if the reference quantity is greater than the reference quantity threshold, and the sum of the number of wave crests and wave troughs in the waveform graph is greater than the second number threshold, the number of rising edges is greater than or equal to the number of falling edges, and the start and end directions of the trajectory are forward, then the trajectory is a risk trajectory; otherwise, the trajectory is a non-risk trajectory.

[0080] In this embodiment, the value range of the second number threshold is 3-6, which can be specifically 3, 4, 5 or 6, which can be set by those skilled in the art according to the actual situation, and the embodiment does not make specific limitations. The start and end directions of the trajectory mean that the terminal point of the trajectory has a positive displacement relative to the starting point.

[0081] With the technical scheme, if the reference quantity is greater than the reference quantity threshold value, it indicates that the trajectory has obvious fluctuation. In order to reduce the generation of false alarms, even if the start and end directions of the trajectory meet the requirements, the vehicle temporarily delays the sending of the alarm signal. In addition to the start and end directions of the trajectory, the sum of the number of wave crests and the number of wave troughs in the waveform diagram and the relationship with the second quantity threshold value, and the number of rising edges and the number of falling edges are combined to determine whether the trajectory is a risk trajectory. Only in this way, the system can correctly alarm the non-rigid object such as a pedestrian approaching the vehicle, and effectively reduce the probability of false alarms. Since the non-rigid object such as a pedestrian has a low speed and is usually prone to such fluctuating trajectories, appropriate delay of the alarm in this case will not greatly damage the timeliness of the system alarm, and can effectively reduce the probability of false alarms.

[0082] In a specific embodiment, in step S3, if the reference quantity is less than or equal to the reference quantity threshold value, and the start and end directions of the trajectory are forward, the trajectory is a risk trajectory; otherwise, the trajectory is a non-risk trajectory.

[0083] If the reference quantity is less than or equal to the reference quantity threshold value, it indicates that the trajectory is relatively stable. If the start and end directions of the trajectory are forward, the trajectory can be directly determined as a risk trajectory. In this way, the determination of whether the trajectory is a risk trajectory can be made relatively quickly. If the trajectory is a risk trajectory, the passenger can be quickly prompted for safety, so that the danger can be avoided.

[0084] In a specific embodiment, in step S3, the mean and variance of the longitudinal displacement difference value sequence are calculated, and the reference quantity of the longitudinal displacement change rate is calculated according to the following formula:

[0085] Y' var = Y var / Y mean 2

[0086] Y' var is the reference quantity of the longitudinal displacement change rate; Y var is the variance of the longitudinal displacement difference value sequence; and Y mean is the mean of the longitudinal displacement difference value sequence.

[0087] In the embodiment, the reference quantity of the longitudinal displacement change rate is related to the variance and the square of the mean of the longitudinal displacement difference value sequence. The reference quantity is used to describe the comprehensive fluctuation intensity of the trajectory. For objects moving at high speed or low speed, the same reference quantity threshold value can accurately divide the trajectory into a fluctuating trajectory and a non-fluctuating trajectory.

[0088] In an embodiment, in step S1, the corner points in the surround view image are extracted by using the FAST detection method. The FAST detection method is a common method for extracting corner points in an image in the art, and has the characteristics of short calculation time, small complexity, and outstanding detection effect.

[0089] In an embodiment, in step S1, the distribution of the extracted corner points in the surround view image is uniformized by using the quadtree equalization algorithm. The quadtree equalization algorithm can improve the uniformity of the distribution of the corner points, and facilitate the subsequent screening of the corner points and the tracking of the optical flow of the corner points.

[0090] Specifically, the quadtree equalization algorithm is a common technical method in the art. In the embodiment, the target number of corner points in each frame of the surround view image is first determined. The actual number of the extracted corner points in the surround view image can be reduced to the target number by using the quadtree equalization algorithm. For example, if there are multiple corner points in a small range on the surround view image, only one corner point can be retained and the remaining corner points can be deleted, so as to improve the uniformity of the distribution of the corner points.

[0091] In an embodiment, as shown in FIG. 1, in step S1, the screening of the extracted corner points in the surround view image includes the following steps: Figure 6

[0092] S11: After obtaining the current frame of the surround view image, the distance between each corner point extracted in the previous frame of the surround view image and each corner point in the corner point management list is calculated, respectively. It is assumed that the current frame of the surround view image is the t-th frame of the surround view image, and t≥3. The distance can be the Euclidean distance. It should be noted that the first frame of the surround view image and the second frame of the surround view image are not used to obtain the initial corner point management list and the trajectory of the corner points in the first frame of the surround view image and the second frame of the surround view image. Therefore, in step S11, it is required that t≥3.

[0093] It should be noted that the initial corner point management list is established as follows in step S11: first, an empty corner point management list is established, the first frame of the surround view image is obtained, and the corner points in the first frame of the surround view image are extracted; the extracted corner points in the first frame of the surround view image are added to the corner point management list, and the hit count of the corner points in the corner point management list is counted as M, where M is a positive integer; if the FPS of the surround view image is 30, i.e., 30 frames per second, the value of M can be set to 15, which means that 15 frames of the surround view image are used as a corner point screening period.

[0094] ​Further, the second frame of the surround view image is acquired, and a corner point in the second frame of the surround view image is extracted; the corner points in the corner point management list are tracked by using the LK optical flow method between the second frame of the surround view image and the first frame of the surround view image, and it is judged whether the corner points are tracked successfully; if the tracking is successful, the trajectory of the corresponding corner point is recorded; if the tracking fails, the trajectory of the corresponding corner point in the corner point management list and the corresponding corner point in the corner point management list are deleted, and the initial corner point management list and the trajectories of the corner points in the first frame of the surround view image and the second frame of the surround view image are obtained. Then, the third frame of the surround view image is acquired, and the management list is updated according to steps S11-S14.

[0095] S12: it is judged whether there is a corner point in the previous frame of the surround view image, which has a distance less than a distance threshold to the corner point in the corner point management list; if yes, the hit number of the corresponding corner point in the corner point management list is increased by 1, and the upper limit of the hit number is M, wherein M is a positive integer; if no, the hit number of the corresponding corner point in the corner point management list is decreased by 1, and the lower limit of the hit number is 0.

[0096] In the embodiment, if the distance between the corner point in the previous frame of the surround view image and one corner point in the corner point management list is less than the distance threshold, the hit number of the corner point in the corner point management list is increased by 1. After step S13, the hit number of one corner point is increased by 1 at most; and the hit number of one corner point is decreased by 1 at most. The distance threshold can be set as the pixel length in the corresponding surround view image with a distance of 0.5 meters in the physical world, or can be set as the pixel length in the corresponding surround view image with other physical distances, which can be set according to actual needs by those skilled in the art.

[0097] S13: it is judged whether there is a corner point in the previous frame of the surround view image, which has a distance greater than the distance threshold to each corner point in the corner point management list; if yes, the corresponding corner point is added to the corner point management list, and the hit number of the corresponding corner point is M.

[0098] S14: the corner point with the hit number of 0 in the corner point management list is deleted, and a new corner point management list is obtained, and the trajectory of the corner point in the current frame of the surround view image is obtained by using the optical flow method.

[0099] Since repeating the calculation of the trajectories of very close points does not bring improvement to the final performance of the algorithm, it only causes unnecessary memory and calculation overhead. With the above scheme, if there is an angle point in the previous frame of the surround view image that is less than the distance threshold from the angle point in the angle point management list, then the hit count of the corresponding angle point in the angle point management list is incremented by 1, and if there is no angle point in the previous frame of the surround view image that is less than the distance threshold from the angle point in the angle point management list, then the hit count of the corresponding angle point in the angle point management list is decremented by 1; if there is an angle point in the previous frame of the surround view image that is greater than or equal to the distance threshold from each angle point in the angle point management list, then the corresponding angle point is added to the angle point management list, and the hit count of the corresponding angle point is M, which can ensure that there are not a large number of very close points in the angle point management list, thereby avoiding repeated calculation of the trajectories of very close points and reducing memory and calculation overhead.

[0100] In a specific embodiment, in step S1, the LK optical flow method is used to obtain the trajectory of the corner point, including the following steps:

[0101] S15: Between the current frame of the surround view image and the previous frame of the surround view image, the angle point in the new angle point management list is tracked by the LK optical flow method to determine whether the angle point is successfully tracked; wherein the current frame of the surround view image is set as the tthframe of the surround view image; and

[0102] If the tracking is successful, the corresponding angle point and the trajectory of the corresponding angle point are recorded;

[0103] If the tracking fails, the corresponding angle point and the trajectory of the corresponding angle point in the new angle point management list are deleted;

[0104] The trajectory of the angle point in the current frame of the surround view image is obtained, and then step S2 is continued.

[0105] Specifically, the LK optical flow method is used for optical flow tracking and determining whether the angle point is successfully tracked, which is a commonly used method in the art. A deviation value err is used to determine whether the tracking is successful, which represents the sum of the absolute values of the pixel gray scale difference between the WxH picture block around the original position of the angle point and the WxH picture block around the position after tracking the angle point. When err is less than a threshold T optical , it is considered that the tracking is successful, otherwise the tracking fails. In the present embodiment, W and H can both be 11, and the threshold T optical may be 8.

[0106] It should be noted that in the embodiment, before step S2 is performed, it is necessary to first determine whether the trajectory of the corner point included in the obtained current frame surround view image is a valid trajectory, if yes, step S2 is continued to be performed; if no, the current frame surround view image is set as the t+1 frame surround view image, then steps S11-S15 are continued to be performed until the trajectory of the corner point is a valid trajectory, and then step S2 is continued to be performed. The valid trajectory refers to that the trajectory of at least one corner point can be obtained in the continuous k frame surround view images, i.e., the current frame surround view image is the t+k frame surround view image, and the trajectory obtained by continuing to perform steps S11-S15. If the refresh rate (Frame Per Second, FPS) of the surround view image is 30, i.e., 30 frames per second, k can be 5.

[0107] By using the above scheme, the performance of the LK optical flow method for tracking the corner point is better, the operation speed is faster, and the trajectory of the corner point in the surround view image can be quickly obtained.

[0108] In the embodiment, the whole process of the vehicle door opening warning method circulates as follows: if the t-1 frame surround view image and the valid corner point trajectory have been obtained; the t frame surround view image is obtained, if FPS=30, t≥6, i.e., the distance between the corner point of the t-1 frame and the corner point in the corner point management list is calculated, the corner point management list is added or deleted according to the corner point of the t-1 frame, the corner point in the updated corner point management list is tracked by optical flow between the t-1 frame and the t frame, the coordinate position of the corner point in the management list is updated to the t frame coordinate according to the tracking result, the trajectory of the corner point including the t frame surround view image is obtained, the above trajectory is judged by steps S2-S3, and it is decided whether the vehicle outputs the alarm signal after the t frame surround view image is obtained. Then the t+1 frame surround view image is continuously obtained, a new cycle is started, and the above operation is continued to decide whether the vehicle outputs the alarm signal after the t+1 frame surround view image is obtained.

[0109] The application provides a vehicle door opening warning method, which comprises the following steps: acquiring surround view images on the left and right sides behind a vehicle; extracting corner points in the surround view images; screening the extracted corner points in the surround view images; obtaining the trajectories of the corner points in the surround view images by using an optical flow method; obtaining the longitudinal displacement of the corner points at each moment in the trajectories by setting a scanning sliding window with a predetermined period; obtaining the waveform graph of time and longitudinal displacement; obtaining the longitudinal displacement of the corner points in each predetermined period in the trajectories to form a longitudinal displacement sequence; setting two different methods for judging whether the trajectories are risk trajectories according to the size relationship between a reference quantity and a reference quantity threshold value, i.e., according to the fluctuation of the trajectories; if the reference quantity is less than or equal to the reference quantity threshold value, i.e., the trajectory is relatively stable, only the start and end directions of the trajectory are needed to judge whether the trajectory is a risk trajectory; if the reference quantity is greater than the reference quantity threshold value, i.e., the trajectory fluctuates obviously, the number of wave crests and wave troughs in the waveform graph, the number of rising edges and falling edges, and the start and end directions of the trajectory are needed to judge whether the trajectory is a risk trajectory, so that whether the trajectory is a risk trajectory can be determined more accurately, and the false alarm probability caused by non-rigid object mixed trajectories can be reduced. After judging whether the trajectory is a risk trajectory, it is needed to judge whether the number of risk trajectories in a predetermined region is greater than a first number threshold value; if the number of risk trajectories is greater than the first number threshold value, the vehicle generates an alarm signal, so that the alarm signal can give a safety prompt to a passenger or a driver who is preparing to open the door and get off the vehicle, and the occurrence of danger can be avoided.

[0110] Although the application has been illustrated and described with reference to certain preferred embodiments thereof, it should be understood by the skilled in the art that the foregoing is a further detailed description of the application in connection with specific embodiments, and is not intended to limit the application to the specific embodiments. The skilled in the art can make various changes in form and details without departing from the spirit and scope of the application, including making several simple deductions or substitutions.

Claims

1. A vehicle door opening warning method, characterized in that, Includes the following steps: S1: Obtain surround view images of the left and right sides behind the vehicle, extract corner points from the surround view images, filter the extracted corner points from the surround view images, and obtain the trajectory of the corner points from the surround view images using optical flow method; S2: By setting a scanning window with a predetermined period, the longitudinal displacement of the corner point at each moment in the trajectory is obtained, and a waveform diagram of time and longitudinal displacement is obtained; the longitudinal displacement difference of the corner point within each predetermined period is obtained to form a longitudinal displacement difference sequence. S3: Calculate a reference value for the longitudinal displacement change rate based on the longitudinal displacement difference sequence, and determine whether the trajectory is a risk trajectory based on the reference value; wherein, If the reference value is less than or equal to the reference value threshold, and the starting and ending directions of the trajectory are forward, the trajectory is marked as a risk trajectory; otherwise, the trajectory is marked as a non-risk trajectory. If the reference value is greater than the reference value threshold, and the sum of the number of peaks and troughs in the waveform is greater than the second quantity threshold, the number of rising edges is greater than or equal to the number of falling edges, and the starting and ending directions of the trajectory are forward, the trajectory is marked as a risk trajectory; otherwise, the trajectory is marked as a non-risk trajectory. S4: Obtain the number of risk trajectories marked within a predetermined area of ​​the surround view image, and determine whether the vehicle should generate an alarm signal based on the number of risk trajectories within the predetermined area; wherein... If the number of risk trajectories within the predetermined area exceeds a first quantity threshold, the vehicle generates an alarm signal. If the number of risk trajectories within the predetermined area is less than or equal to the first quantity threshold, the vehicle will not generate the alarm signal.

2. The vehicle door opening warning method as described in claim 1, characterized in that, In step S3, the mean and variance of the longitudinal displacement difference sequence are calculated, and the reference value of the longitudinal displacement change rate is calculated according to the following formula: AND' var And var / AND mean 2 Among them, Y' var Y is the reference value for the longitudinal displacement change rate; var Y is the variance of the longitudinal displacement difference sequence; mean The mean of the longitudinal displacement difference sequence is given.

3. The vehicle door opening warning method as described in claim 2, characterized in that, In step S1, the corner points in the panoramic image are extracted using the FAST detection method.

4. The vehicle door opening warning method as described in claim 3, characterized in that, In step S1, a quadtree equalization algorithm is used to homogenize the distribution of corner points in the extracted toroidal image.

5. The vehicle door opening warning method as described in claim 4, characterized in that, In step S1, filtering the corner points in the extracted ring view image includes the following steps: S11: After obtaining the current frame ring view image, calculate the distance between each corner point extracted from the previous frame ring view image and each corner point in the corner point management list; set the current frame ring view image as the t-th frame ring view image, and t≥3; S12: Determine whether there is a corner point in the previous frame of the panoramic image whose distance to the corner point in the corner point management list is less than a distance threshold; If so, the number of times the corresponding corner point is hit in the corner point management list is incremented by 1, and the upper limit of the number of hits is M, where M is a positive integer; If not, the number of times the corresponding corner in the corner management list is hit is reduced by 1, and the lower limit of the number of times it is hit is 0. S13: Determine whether there is a corner in the previous frame of the panoramic image whose distance from each corner in the corner management list is greater than or equal to the distance threshold; If so, the corresponding corner point is added to the corner point management list, and the number of times the corresponding corner point is hit is counted as M; S14: Delete the corner points in the corner point management list that have been hit 0 times, obtain a new corner point management list, and continue to obtain the trajectory of the corner points in the current frame circumferential image through optical flow method.

6. The vehicle door opening warning method as described in claim 5, characterized in that, In step S1, the trajectory of the corner point is obtained using the LK optical flow method, including the following steps: S15: Between the current frame circumsight image and the previous frame circumsight image, perform optical flow tracking on the corner points in the new corner point management list using the LK optical flow method and determine whether the corner points have been successfully tracked. If the tracking is successful, the corresponding corner point and the trajectory of the corresponding corner point are recorded; If tracking fails, delete the corresponding corner point and its trajectory from the new corner point management list; The trajectory of the corner point in the current frame ring view image is obtained, and then step S2 is continued.

7. The vehicle door opening warning method as described in any one of claims 1-6, characterized in that, In step S1, acquiring surround view images of the left and right rear sides of the vehicle includes: Fisheye images of the rear left and right sides of the vehicle are obtained by onboard fisheye cameras on the left and right sides of the vehicle, and the fisheye images of the rear left and right sides of the vehicle are stitched together to form the surround view images of the rear left and right sides of the vehicle.

8. The vehicle door opening warning method as described in any one of claims 1-6, characterized in that, The reference threshold value ranges from 0.2 to 0.5; the first quantity threshold value is 3 or 4; and the second quantity threshold value ranges from 3 to 6.

Images