Vehicle blind spot target detection and tracking method, device, electronic device and storage medium

Abstract

The present invention provides a vehicle blind spot target detection and tracking method, device, electronic device and storage medium. The method comprises: obtaining a blind spot image collected by a fisheye camera; performing rotation target detection on the blind spot image to obtain multiple detection frames; wherein the detection frame comprises a vehicle body detection frame, a front vehicle detection frame and a rear vehicle detection frame; for each vehicle body detection frame, each front vehicle detection frame and a rear vehicle detection frame are matched with the vehicle body detection frame to obtain a front vehicle detection frame or a rear vehicle detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target; for each vehicle target, the two-dimensional position and posture information of the vehicle target is determined based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target; and each vehicle target is tracked based on each two-dimensional position and posture information. The present invention can track vehicle targets in vehicle blind spots, with low calculation amount, fast processing speed and high detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a vehicle blind spot target detection and tracking method, device, electronic equipment and storage medium. Background Art

[0002] Due to the insufficient field of view of the vehicle's rearview mirror, there are always blind spots on both sides of the vehicle, which causes safety hazards to the driver during driving procedures such as changing lanes and merging. In order to improve the safety of the driver, a vehicle blind spot monitoring system is usually set up to assist the driver in monitoring the targets within the blind spot range.

[0003] In order to cover a larger blind spot range, vehicle blind spot monitoring systems often use fisheye cameras with a larger field of view. However, fisheye camera images are severely distorted, and traditional target detection methods are difficult to ensure the measurement accuracy of target position and speed when applied to fisheye camera images, resulting in poor application effects of current blind spot monitoring systems. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, electronic device and storage medium for detecting and tracking a target in a blind spot of a vehicle, so as to solve the problem of improving the accuracy of detecting and tracking a target in a blind spot of a vehicle.

[0005] In a first aspect, an embodiment of the present invention provides a vehicle blind spot target detection and tracking method, comprising:

[0006] Obtain the blind area image collected by the fisheye camera;

[0007] Performing rotation target detection on the blind spot image to obtain multiple detection frames; wherein the detection frames include a vehicle body detection frame, a vehicle front detection frame, and a vehicle rear detection frame;

[0008] For each vehicle body detection frame, each vehicle front detection frame and vehicle rear detection frame is matched with the vehicle body detection frame to obtain the vehicle front detection frame or vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target;

[0009] For each vehicle target, determine the two-dimensional pose information of the vehicle target based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target;

[0010] Based on each two-dimensional pose information, each vehicle target is tracked.

[0011] In a possible implementation, for each vehicle body detection frame, each vehicle front detection frame and vehicle rear detection frame is matched with the vehicle body detection frame to obtain a vehicle front detection frame or a vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set, including:

[0012] For each vehicle body detection frame, the intersection and union ratios of each front detection frame and the rear detection frame with the vehicle body detection frame are calculated to obtain multiple intersection and union ratios. If the maximum value of each intersection and union ratio is greater than a preset threshold, the front detection frame or the rear detection frame corresponding to the largest intersection and union ratio is used as the front detection frame or the rear detection frame corresponding to the vehicle body detection frame, and they form a detection frame set.

[0013] In a possible implementation, the two-dimensional pose information includes a center position, a heading angle, a length, and a width;

[0014] For each vehicle target, the two-dimensional pose information of the vehicle target is determined based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target, including:

[0015] The intersection of the bottom edge of the vehicle body detection frame and the bottom edge of the vehicle front detection frame or the vehicle rear detection frame in the detection frame set corresponding to the first vehicle target is used as the first distance measurement point, the end point of the bottom edge of the vehicle body detection frame is used as the second distance measurement point, and the end point of the bottom edge of the vehicle front detection frame or the vehicle rear detection frame is used as the third distance measurement point; wherein the first vehicle target is any vehicle target;

[0016] The pixel coordinates of the first ranging point, the second ranging point, and the third ranging point are converted into world coordinates based on the intrinsic parameters and extrinsic parameters of the fisheye camera;

[0017] The heading angle and length of the first vehicle target are calculated based on the world coordinates of the first ranging point and the second ranging point, the width of the first vehicle target is calculated based on the world coordinates of the first ranging point and the third ranging point, and the center coordinates of the first vehicle target are calculated based on the world coordinates of the first ranging point, the second ranging point, and the third ranging point.

[0018] In a possible implementation, there are multiple blind spot images; based on each two-dimensional posture information, each vehicle target is tracked, including:

[0019] The Kalman filter algorithm is used to track each vehicle target based on the two-dimensional posture information of each vehicle target in each blind spot image.

[0020] In a possible implementation, rotating target detection is performed on the blind area image to obtain multiple detection frames, including:

[0021] The blind area image is input into the trained rotation target detection model to obtain multiple detection boxes.

[0022] In a possible implementation, the detection frame further includes a pedestrian detection frame, and each pedestrian detection frame corresponds to a pedestrian target;

[0023] After rotating the target detection on the blind area image and obtaining multiple detection frames, the following steps are also included:

[0024] For each pedestrian target, the two-dimensional position information of the pedestrian target is determined based on the coordinates of the pedestrian detection frame corresponding to the pedestrian target.

[0025] In a possible implementation, the heading angle of each pedestrian target is a preset heading angle, the length is a preset length, and the width is a preset width; for each pedestrian target, the two-dimensional pose information of the pedestrian target is determined based on the coordinates of the pedestrian detection frame corresponding to the pedestrian target, including:

[0026] The ground midpoint of the pedestrian detection frame corresponding to the first pedestrian target is used as the distance measurement point; wherein the first pedestrian target is any pedestrian target;

[0027] Based on the intrinsic and extrinsic parameters of the fisheye camera, the pixel coordinates of the ranging point are converted into world coordinates and used as the center position of the first pedestrian target.

[0028] In a second aspect, an embodiment of the present invention provides a vehicle blind spot target detection and tracking device, comprising:

[0029] An acquisition module is used to acquire the blind area image collected by the fisheye camera;

[0030] A detection module is used to perform rotating target detection on the blind spot image to obtain multiple detection frames; wherein the detection frames include a vehicle body detection frame, a front vehicle detection frame, and a rear vehicle detection frame;

[0031] A matching module is used to match each vehicle body detection frame with each vehicle front detection frame and vehicle rear detection frame to obtain a vehicle front detection frame or a vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target;

[0032] A posture recognition module is used to determine the two-dimensional posture information of each vehicle target based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target;

[0033] The tracking module is used to track each vehicle target based on each two-dimensional posture information.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of the method described in the first aspect or any possible implementation manner of the first aspect.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0036] The embodiments of the present invention provide a vehicle blind spot target detection and tracking method, device, electronic device and storage medium. Through rotating target detection and detection frame matching, the two-dimensional position information of each vehicle target in the vehicle blind spot image can be directly determined. There is no need to perform distortion correction on the blind spot image taken by a fisheye camera, and there is no need to solve the three-dimensional parameters of the vehicle target. The vehicle target in the vehicle blind spot can be tracked with low calculation amount, fast processing speed and high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 is a flow chart of a method for detecting and tracking a target in a vehicle blind spot provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of target detection results provided by an embodiment of the present invention;

[0040] Figure 3 It is a structural schematic diagram of a vehicle blind spot target detection and tracking device provided by an embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0044] See also Figure 1, which shows a flow chart of the implementation of the vehicle blind spot target detection and tracking method provided by an embodiment of the present invention, which is described in detail as follows:

[0045] Step 101, obtaining a blind area image captured by a fisheye camera.

[0046] In this embodiment, the fisheye camera installed on the side of the vehicle body is used to collect the image of the blind spot. The blind spot image in this embodiment may not be subjected to distortion correction, but may be subjected to distortion correction to improve the target detection and tracking effect.

[0047] Step 102, performing rotation target detection on the blind spot image to obtain multiple detection frames; wherein the detection frames include a vehicle body detection frame, a vehicle front detection frame, and a vehicle rear detection frame.

[0048] In this embodiment, since the blind spot image has a certain degree of distortion, a deep learning network can be used to detect the rotated target on the captured image and output a rotated target detection frame, which can more accurately reflect the position of the target in the image. For different target types, the output detection frame is different: for pedestrian targets, only one detection frame is output, and for car targets, the body detection frame, as well as the front detection frame or the rear detection frame are output.

[0049] Step 103, for each vehicle body detection frame, each front detection frame and rear detection frame are matched with the vehicle body detection frame to obtain the front detection frame or the rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target.

[0050] In this embodiment, the rotated target detection frames output by the above-mentioned deep learning network are matched, combined and reconstructed to obtain a set of rotated frames for each target; the target frames that need to be combined and reconstructed are mainly detection frames of vehicle targets. The specific implementation method can be to screen and match according to the detection frame category and the IOU of the detection frame to determine multiple detection frames corresponding to the same vehicle target.

[0051] Step 104: for each vehicle target, determine the two-dimensional position information of the vehicle target based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target.

[0052] In this embodiment, the specific positions of various parts of the vehicle target can be determined based on the target frame set obtained in the above steps, combined with the internal and external parameters of the fisheye camera. Since the vehicle target is in contact with the ground and there is no need to detect and track the height changes of the vehicle target, the Z-axis coordinates of the vehicle target can be ignored, so only the XY-axis related posture information such as the center position, heading angle, length, width, etc. of the target needs to be calculated to determine whether there is a vehicle target with a collision risk in the blind spot of the vehicle.

[0053] Step 105: Track each vehicle target based on each two-dimensional position information.

[0054] In this embodiment, by repeating the above steps, the center position, heading angle, length, and width of the vehicle target can be determined from multiple consecutive frames of blind spot images, and then the center position, heading angle, length, and width of the vehicle target can be tracked to determine whether there is a risk of collision with the vehicle.

[0055] The embodiments of the present invention can directly determine the two-dimensional position information of each vehicle target in the vehicle blind spot image through rotating target detection and detection frame matching. There is no need to perform distortion correction on the blind spot image taken by the fisheye camera, nor to solve the three-dimensional parameters of the vehicle target. The vehicle target in the vehicle blind spot can be tracked with low calculation amount, fast processing speed and high detection accuracy.

[0056] In a possible implementation, for each vehicle body detection frame, each vehicle front detection frame and vehicle rear detection frame is matched with the vehicle body detection frame to obtain a vehicle front detection frame or a vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set, including:

[0057] For each vehicle body detection frame, the intersection and union ratios of each front detection frame and the rear detection frame with the vehicle body detection frame are calculated to obtain multiple intersection and union ratios. If the maximum value of each intersection and union ratio is greater than a preset threshold, the front detection frame or the rear detection frame corresponding to the largest intersection and union ratio is used as the front detection frame or the rear detection frame corresponding to the vehicle body detection frame, and they form a detection frame set.

[0058] In this embodiment, to ensure the accuracy of detection frame matching, you can first filter according to the vehicle category in the detection frame label, and then match the body detection frame and the front detection frame / rear detection frame of the same category of vehicles. The specific steps are as follows:

[0059] 1. Filter out the body detection frames and front detection frames / rear detection frames of similar vehicles, such as filtering out the body detection frames and front detection frames / rear detection frames of all cars;

[0060] 2. Calculate the intersection over union (IOU) of each vehicle body detection frame and all vehicle front detection frames / rear detection frames respectively, and select the vehicle front detection frame / rear detection frame with the largest IOU as the candidate frame. If the IOU value of the candidate frame is greater than a preset threshold such as 0.3, it is considered that the vehicle body detection frame and the vehicle front detection frame / rear detection frame are matched successfully, and these two detection frames constitute the rotation frame set of the target.

[0061] In a possible implementation, the two-dimensional pose information includes a center position, a heading angle, a length, and a width;

[0062] For each vehicle target, the two-dimensional pose information of the vehicle target is determined based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target, including:

[0063] The intersection of the bottom edge of the vehicle body detection frame and the bottom edge of the vehicle front detection frame or the vehicle rear detection frame in the detection frame set corresponding to the first vehicle target is used as the first distance measurement point, the end point of the bottom edge of the vehicle body detection frame is used as the second distance measurement point, and the end point of the bottom edge of the vehicle front detection frame or the vehicle rear detection frame is used as the third distance measurement point; wherein the first vehicle target is any vehicle target;

[0064] The pixel coordinates of the first ranging point, the second ranging point, and the third ranging point are converted into world coordinates based on the intrinsic parameters and extrinsic parameters of the fisheye camera;

[0065] The heading angle and length of the first vehicle target are calculated based on the world coordinates of the first ranging point and the second ranging point, the width of the first vehicle target is calculated based on the world coordinates of the first ranging point and the third ranging point, and the center coordinates of the first vehicle target are calculated based on the world coordinates of the first ranging point, the second ranging point, and the third ranging point.

[0066] In this embodiment, for a vehicle target, it is necessary to calculate the positions of multiple points on the vehicle target to determine the positional relationship between the vehicle target and the current vehicle.

[0067] See also Figure 2 , for the case where the detection frame set includes a vehicle body detection frame 1 and a vehicle rear detection frame 2, the intersection of the bottom edge of the vehicle body detection frame 1 and the bottom edge of the vehicle rear detection frame 2 can be selected as the first ranging point 3, the first ranging point 3 represents the lower left vertex of the vehicle rear surface, the end point of the bottom edge of the vehicle body detection frame 1 is selected as the second ranging point 4, the second ranging point 4 represents the lower left vertex of the vehicle front surface, the end point of the bottom edge of the vehicle rear detection frame 2 is selected as the third ranging point 5, the third ranging point 5 represents the lower right vertex of the vehicle rear surface, at this time, the vehicle target is regarded as a cuboid, and the three ranging points represent the three non-coplanar vertices of the cuboid, and the coordinates of the three ranging points can determine multiple data of the cuboid.

[0068] Specifically, after the world coordinates of the three ranging points are calculated according to the intrinsic and extrinsic parameters of the camera, according to the physical meanings of the three ranging points, the heading angle can be calculated according to the vector angle between the first ranging point 3 and the second ranging point 4, the length can be calculated according to the distance between the first ranging point 3 and the second ranging point 4, and the width can be calculated according to the distance between the first ranging point 3 and the third ranging point 5. The midpoints of the coordinates of the first ranging point 3, the second ranging point 4 and the third ranging point 5 in the XY axis directions are taken as the coordinates of the center position of the vehicle target.

[0069] In a possible implementation, there are multiple blind spot images; based on each two-dimensional posture information, each vehicle target is tracked, including:

[0070] The Kalman filter algorithm is used to track each vehicle target based on the two-dimensional posture information of each vehicle target in each blind spot image.

[0071] In this embodiment, the Kalman filter algorithm can merge the past measurement estimation error into the new measurement error to estimate the future error and achieve target tracking. For pedestrian targets, it is only necessary to correct and update the target's center position according to the center position of the pedestrian target in the new frame image. For vehicle targets, the center position, heading angle, length, and width of the target are corrected and updated according to the newly acquired center position, heading angle, length, and width.

[0072] In a possible implementation, rotating target detection is performed on the blind area image to obtain multiple detection frames, including:

[0073] The blind area image is input into the trained rotation target detection model to obtain multiple detection boxes.

[0074] In this embodiment, the rotating target detection model can be constructed using the YOLO network, and training samples can be produced based on vehicle and pedestrian images taken by the fisheye camera and the model can be trained. The rotating target detection model is applied to the vehicle target, and the detection frames corresponding to multiple vehicle parts can be identified, which facilitates the combination of coordinates of multiple parts of the vehicle for posture recognition, thereby solving the problem that the single non-rotating rectangular frame target detection and tracking method cannot accurately measure the target position when applied to the fisheye camera.

[0075] In a possible implementation, the detection frame further includes a pedestrian detection frame, and each pedestrian detection frame corresponds to a pedestrian target;

[0076] After rotating the target detection on the blind area image and obtaining multiple detection frames, the following steps are also included:

[0077] For each pedestrian target, the two-dimensional position information of the pedestrian target is determined based on the coordinates of the pedestrian detection frame corresponding to the pedestrian target.

[0078] In this embodiment, the two-dimensional posture information of a pedestrian target can be used to determine whether the pedestrian target is always within the blind spot of the vehicle. If it is always there, the two-dimensional posture information of the pedestrian target in each frame of the blind spot image can be used to track and predict the movement trajectory of the pedestrian target to determine whether there is a risk of collision between the pedestrian target and the current vehicle.

[0079] In a possible implementation, the heading angle of each pedestrian target is a preset heading angle, the length is a preset length, and the width is a preset width; for each pedestrian target, the two-dimensional pose information of the pedestrian target is determined based on the coordinates of the pedestrian detection frame corresponding to the pedestrian target, including:

[0080] The ground midpoint of the pedestrian detection frame corresponding to the first pedestrian target is used as the distance measurement point; wherein the first pedestrian target is any pedestrian target;

[0081] Based on the intrinsic and extrinsic parameters of the fisheye camera, the pixel coordinates of the ranging point are converted into world coordinates and used as the center position of the first pedestrian target.

[0082] In this embodiment, pedestrian targets have the characteristics of small size, small volume difference and slow movement speed compared with vehicle targets. Therefore, for pedestrian targets, the ground midpoint of the rotating target frame can be directly selected as the ranging pixel point, and the center position of the target can be calculated according to the intrinsic and extrinsic parameters of the camera. The heading angle is taken as a preset heading angle, such as 0, the length is set to a fixed preset length, such as 0.5, and the width is set to a fixed preset width, such as 0.5.

[0083] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0084] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0085] Figure 3 The following is a schematic diagram showing the structure of a vehicle blind spot target detection and tracking device provided by an embodiment of the present invention. For ease of explanation, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0086] like Figure 3 As shown, the vehicle blind spot target detection and tracking device 3 includes:

[0087] An acquisition module 31 is used to acquire a blind area image captured by a fisheye camera;

[0088] The detection module 32 is used to perform rotating target detection on the blind spot image to obtain multiple detection frames; wherein the detection frames include a vehicle body detection frame, a front vehicle detection frame and a rear vehicle detection frame;

[0089] A matching module 33 is used to match each vehicle body detection frame with each vehicle front detection frame and vehicle rear detection frame to obtain a vehicle front detection frame or a vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target;

[0090] A posture recognition module 34 is used to determine, for each vehicle target, the two-dimensional posture information of the vehicle target based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target;

[0091] The tracking module 35 is used to track each vehicle target based on each two-dimensional position information.

[0092] In a possible implementation, the matching module 33 is specifically configured to:

[0093] For each vehicle body detection frame, the intersection and union ratios of each front detection frame and the rear detection frame with the vehicle body detection frame are calculated to obtain multiple intersection and union ratios. If the maximum value of each intersection and union ratio is greater than a preset threshold, the front detection frame or the rear detection frame corresponding to the largest intersection and union ratio is used as the front detection frame or the rear detection frame corresponding to the vehicle body detection frame, and they form a detection frame set.

[0094] In a possible implementation, the two-dimensional pose information includes a center position, a heading angle, a length, and a width;

[0095] The posture recognition module 34 is specifically used for:

[0096] The intersection of the bottom edge of the vehicle body detection frame and the bottom edge of the vehicle front detection frame or the vehicle rear detection frame in the detection frame set corresponding to the first vehicle target is used as the first distance measurement point, the end point of the bottom edge of the vehicle body detection frame is used as the second distance measurement point, and the end point of the bottom edge of the vehicle front detection frame or the vehicle rear detection frame is used as the third distance measurement point; wherein the first vehicle target is any vehicle target;

[0097] The pixel coordinates of the first ranging point, the second ranging point, and the third ranging point are converted into world coordinates based on the intrinsic parameters and extrinsic parameters of the fisheye camera;

[0098] The heading angle and length of the first vehicle target are calculated based on the world coordinates of the first ranging point and the second ranging point, the width of the first vehicle target is calculated based on the world coordinates of the first ranging point and the third ranging point, and the center coordinates of the first vehicle target are calculated based on the world coordinates of the first ranging point, the second ranging point, and the third ranging point.

[0099] In a possible implementation, there are multiple blind area images; the tracking module 35 is specifically used for:

[0100] The Kalman filter algorithm is used to track each vehicle target based on the two-dimensional posture information of each vehicle target in each blind spot image.

[0101] In a possible implementation, the detection module 32 is specifically configured to:

[0102] The blind area image is input into the trained rotation target detection model to obtain multiple detection boxes.

[0103] In a possible implementation, the detection frame further includes a pedestrian detection frame, and each pedestrian detection frame corresponds to a pedestrian target;

[0104] The posture recognition module 34 is also used for:

[0105] After rotating the target detection on the blind spot image to obtain multiple detection frames, for each pedestrian target, the two-dimensional position information of the pedestrian target is determined based on the coordinates of the pedestrian detection frame corresponding to the pedestrian target.

[0106] In a possible implementation, the heading angle of each pedestrian target is a preset heading angle, the length is a preset length, and the width is a preset width; the posture recognition module 34 is specifically used for:

[0107] The ground midpoint of the pedestrian detection frame corresponding to the first pedestrian target is used as the distance measurement point; wherein the first pedestrian target is any pedestrian target;

[0108] Based on the intrinsic and extrinsic parameters of the fisheye camera, the pixel coordinates of the ranging point are converted into world coordinates and used as the center position of the first pedestrian target.

[0109] The embodiments of the present invention can directly determine the two-dimensional position information of each vehicle target in the vehicle blind spot image through rotating target detection and detection frame matching. There is no need to perform distortion correction on the blind spot image taken by the fisheye camera, nor to solve the three-dimensional parameters of the vehicle target. The vehicle target in the vehicle blind spot can be tracked with low calculation amount, fast processing speed and high detection accuracy.

[0110] Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps in the above-mentioned vehicle blind spot target detection and tracking method embodiments are implemented, such as Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 3 Functionality of the modules / units 31 to 35 shown.

[0111] Exemplarily, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 42 in the electronic device 4. For example, the computer program 42 may be divided into Figure 3 Modules / units 31 to 35 are shown.

[0112] The electronic device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0113] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. The memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0115] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned vehicle blind spot target detection and tracking method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0122] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A vehicle blind spot target detection and tracking method, characterized in that: include: Obtain the blind area image collected by the fisheye camera; Performing rotation target detection on the blind spot image to obtain multiple detection frames; wherein the detection frames include a vehicle body detection frame, a vehicle front detection frame, and a vehicle rear detection frame; For each vehicle body detection frame, each vehicle front detection frame and vehicle rear detection frame is matched with the vehicle body detection frame to obtain the vehicle front detection frame or vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target; For each vehicle target, determine the two-dimensional pose information of the vehicle target based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target; Track each vehicle target based on each two-dimensional pose information; The two-dimensional posture information includes center position, heading angle, length and width; The method of determining, for each vehicle target, two-dimensional pose information of the vehicle target based on the coordinates of each detection frame in a detection frame set corresponding to the vehicle target, includes: The intersection of the bottom edge of the vehicle body detection frame and the bottom edge of the vehicle front detection frame or the vehicle rear detection frame in the detection frame set corresponding to the first vehicle target is used as the first distance measurement point, the end point of the bottom edge of the vehicle body detection frame is used as the second distance measurement point, and the end point of the bottom edge of the vehicle front detection frame or the vehicle rear detection frame is used as the third distance measurement point; wherein the first vehicle target is any vehicle target; Convert the pixel coordinates of the first ranging point, the second ranging point, and the third ranging point into world coordinates based on the intrinsic parameters and extrinsic parameters of the fisheye camera; The heading angle and length of the first vehicle target are calculated based on the world coordinates of the first ranging point and the second ranging point, the width of the first vehicle target is calculated based on the world coordinates of the first ranging point and the third ranging point, and the center coordinates of the first vehicle target are calculated based on the world coordinates of the first ranging point, the second ranging point, and the third ranging point.

2. The vehicle blind spot target detection and tracking method according to claim 1, characterized in that: For each vehicle body detection frame, each vehicle front detection frame and vehicle rear detection frame is matched with the vehicle body detection frame to obtain a vehicle front detection frame or a vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set, including: For each vehicle body detection frame, the intersection and union ratios of each front detection frame and the rear detection frame with the vehicle body detection frame are calculated to obtain multiple intersection and union ratios. If the maximum value of each intersection and union ratio is greater than a preset threshold, the front detection frame or the rear detection frame corresponding to the largest intersection and union ratio is used as the front detection frame or the rear detection frame corresponding to the vehicle body detection frame, and they form a detection frame set.

3. The vehicle blind spot target detection and tracking method according to claim 1, characterized in that: The blind spot images are multiple; and the tracking of each vehicle target based on each two-dimensional posture information includes: The Kalman filter algorithm is used to track each vehicle target based on the two-dimensional posture information of each vehicle target in each blind spot image.

4. The vehicle blind spot target detection and tracking method according to claim 1, characterized in that: The rotating target detection is performed on the blind area image to obtain multiple detection frames, including: The blind area image is input into a trained rotating target detection model to obtain multiple detection frames.

5. The vehicle blind spot target detection and tracking method according to claim 1, characterized in that: The detection frame also includes a pedestrian detection frame, each pedestrian detection frame corresponds to a pedestrian target; After performing rotation target detection on the blind area image to obtain multiple detection frames, the method further includes: For each pedestrian target, the two-dimensional position information of the pedestrian target is determined based on the coordinates of the pedestrian detection frame corresponding to the pedestrian target.

6. The vehicle blind spot target detection and tracking method according to claim 5, characterized in that: The heading angle of each pedestrian target is a preset heading angle, the length is a preset length, and the width is a preset width; The method of determining, for each pedestrian target, two-dimensional pose information of the pedestrian target based on the coordinates of a pedestrian detection frame corresponding to the pedestrian target, includes: The ground midpoint of the pedestrian detection frame corresponding to the first pedestrian target is used as a distance measurement point; wherein the first pedestrian target is any pedestrian target; Based on the intrinsic parameters and extrinsic parameters of the fisheye camera, the pixel coordinates of the ranging point are converted into world coordinates and used as the center position of the first pedestrian target.

7. A vehicle blind spot target detection and tracking device, characterized in that: include: An acquisition module is used to acquire the blind area image collected by the fisheye camera; A detection module, used to perform rotation target detection on the blind spot image to obtain multiple detection frames; wherein the detection frames include a vehicle body detection frame, a front vehicle detection frame, and a rear vehicle detection frame; A matching module is used to match each vehicle body detection frame with each vehicle front detection frame and vehicle rear detection frame to obtain a vehicle front detection frame or a vehicle rear detection frame corresponding to the vehicle body detection frame, and form a detection frame set; wherein each detection frame set corresponds to a vehicle target; A posture recognition module is used to determine the two-dimensional posture information of each vehicle target based on the coordinates of each detection frame in the detection frame set corresponding to the vehicle target; A tracking module is used to track each vehicle target based on each two-dimensional posture information; The two-dimensional posture information includes center position, heading angle, length and width; The posture recognition module is specifically used for: The intersection of the bottom edge of the vehicle body detection frame and the bottom edge of the vehicle front detection frame or the vehicle rear detection frame in the detection frame set corresponding to the first vehicle target is used as the first distance measurement point, the end point of the bottom edge of the vehicle body detection frame is used as the second distance measurement point, and the end point of the bottom edge of the vehicle front detection frame or the vehicle rear detection frame is used as the third distance measurement point; wherein the first vehicle target is any vehicle target; Convert the pixel coordinates of the first ranging point, the second ranging point, and the third ranging point into world coordinates based on the intrinsic parameters and extrinsic parameters of the fisheye camera; The heading angle and length of the first vehicle target are calculated based on the world coordinates of the first ranging point and the second ranging point, the width of the first vehicle target is calculated based on the world coordinates of the first ranging point and the third ranging point, and the center coordinates of the first vehicle target are calculated based on the world coordinates of the first ranging point, the second ranging point, and the third ranging point.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images