Target region prediction method and related apparatus, electronic device and storage medium

Abstract

The application discloses a target region prediction method and related device, electronic equipment and storage medium, wherein the target region prediction method comprises: acquiring a video image and a radar point cloud image collected by a camera device and a radar device at a target time respectively; determining a first object region of a target object in the video image, and determining a second object region of the target object in the radar point cloud image, and establishing a region matching relationship between the first object region and the second object region; comparing the region matching relationship with a recorded reference region matching relationship to obtain a comparison result; and determining a target region of the target object at the target time based on the comparison result. The above scheme can accurately predict the target region. In particular, when a millimeter wave radar is selected as the radar device, the millimeter wave radar has a very high detection probability for moving objects, and when combined with a camera, the target region prediction accuracy can be further improved in scenes such as congestion, vehicle concentration and poor light.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a target region prediction method and related device, electronic equipment and storage medium. BACKGROUND

[0002] The advantage of the radar device is that it can accurately obtain the spatial information and motion speed information of the target, and can detect the target far away and is not affected by the environment such as light; the advantage of the camera device is that it can obtain high-accuracy target recognition information, and the video can accurately detect all targets in the medium and short distance. Therefore, the detection equipment integrating the radar device and the camera device is paid more and more attention.

[0003] However, although the camera device and the radar device can complement each other and expand the target detection scene when detecting the target at the same time, high-precision target recognition accuracy cannot be obtained at any time, for example, the vehicles are too dense during the rush hour, the vehicles are blocked, and the light is poor. In view of this, how to accurately predict the target region as much as possible becomes a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a target region prediction method and related device, electronic equipment and storage medium, which can accurately predict the target region as much as possible.

[0005] In order to solve the above technical problem, the first aspect of the present application provides a target region prediction method, comprising: acquiring a video image and a radar point cloud image collected by a camera device and a radar device at a target time; wherein the video image and the radar point cloud image include a target object; determining a first object region of the target object in the video image, and determining a second object region of the target object in the radar point cloud image, and establishing a region matching relationship between the first object region and the second object region; comparing the region matching relationship with a recorded reference region matching relationship to obtain a comparison result; wherein the reference region matching relationship is the region matching relationship at a time before the target time; the comparison result includes whether the region matching relationship is consistent with the reference region matching relationship; determining a target region of the target object at the target time based on the comparison result.

[0006] To solve the above technical problems, the second aspect of the present application provides a target region prediction device, comprising a data acquisition module, a matching relationship establishment module, a comparison module and a target region determination module. The data acquisition module is configured to acquire a video image and a radar point cloud image collected by a camera device and a radar device at a target time, wherein the video image and the radar point cloud image comprise a target object. The matching relationship establishment module is configured to determine a first object region of the target object in the video image, determine a second object region of the target object in the radar point cloud image, and establish a region matching relationship between the first object region and the second object region. The comparison module is configured to compare the region matching relationship with a recorded reference region matching relationship to obtain a comparison result, wherein the reference region matching relationship is a region matching relationship at a time prior to the target time, and the comparison result comprises whether the region matching relationship is consistent with the reference region matching relationship. The target region determination module is configured to determine a target region of the target object at the target time based on the comparison result.

[0007] To solve the above technical problems, the third aspect of the present application provides an electronic device comprising a processor and a memory, wherein the memory and the processor are coupled to each other, and the processor is configured to execute program instructions stored in the memory to implement the target region prediction method in the first aspect.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a computer readable storage medium storing program instructions capable of being executed by a processor, wherein the program instructions are configured to implement the target region prediction method in the first aspect.

[0009] In the above scheme, the first object region of the target object collected by the camera device at the target time and the second object region of the target object collected by the radar device are obtained to obtain the region matching relationship at the target time, and the region matching relationship and the reference matching relationship are compared to determine the target region of the target object at the target time according to the comparison result. The comparison result comprehensively considers the region matching relationship of the target object in the video and the radar at the target time and the time prior to the target time, and can accurately determine whether the target objects at the target time and the time prior to the target time are the same target object according to whether the region matching relationship is consistent. For the target object that fails to be successfully matched at the time prior to the target time, historical data at the time prior to the target time and the object region confirmed at the target time can be further referred to for reasonable analysis and prediction, so that the target region can be as accurately as possible predicted under limited detection conditions. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of an embodiment of the target region prediction method of the present application;

[0011] Figure 2 is a schematic diagram of the scenes of traffic monitoring in the traffic jam and the congestion;

[0012] Figure 3 is a flowchart of an embodiment of the target object coincidence degree acquisition method;

[0013] Figure 4 is a schematic diagram of mapping the first object region to the radar coordinate system;

[0014] Figure 5 is a schematic diagram of displaying the second object region on the basis of Figure 4

[0015] Figure 6 is a flowchart of an embodiment of step S14 in the target region prediction method; Figure 1

[0016] Figure 7 is a flowchart of another embodiment of the target region prediction method of the present application;

[0017] Figure 8 is a framework diagram of an embodiment of the target region prediction device of the present application;

[0018] Figure 9 is a framework diagram of an embodiment of the electronic device of the present application;

[0019] Figure 10 is a framework diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0020] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0021] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, but not for the sake of limitation.

[0022] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is merely an associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" in this paper means two or more than two.

[0023] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the target region prediction method of the present application. Specifically, the target region prediction method in the present embodiment can include the following steps:

[0024] Step S11: Obtain the video image and the radar point cloud image collected by the camera device and the radar device at the target time, respectively. ​​

[0025] In this embodiment, the video image and the radar point cloud image include a target object. The radar device is particularly suitable for millimeter wave radar. The millimeter wave radar can actively emit electromagnetic waves and receive signals of the same frequency. For moving objects or objects with a large RCS (Radar Cross Section), there is a very high detection probability, and for stationary objects, there is a lower detection probability (the detection probability is not zero). The millimeter wave radar can work 24 hours a day and is less affected by weather, so it can effectively cooperate with the camera.

[0026] In one implementation scenario, the video image is obtained by a camera device at a target time, and then obtained by analyzing video data. The analysis method for analyzing the video data can be a specific algorithm such as target recognition and target tracking, which is not limited here. The radar point cloud image is obtained by a radar device at a target time, and then obtained by analyzing radar data. The analysis method for analyzing the radar data can be a radar point cloud clustering method, a radar point cloud feature extraction method, for example, a density-based clustering method with noise (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), which is not limited here.

[0027] Further, the target object is set according to the actual application scenario, for example, see Figure 2 , Figure 2 is a schematic diagram of a traffic monitoring application scenario, the left side is a smooth scene, and the right side is a congested scene. In this application scenario, the target object is a vehicle, and the diagram includes vehicle A, vehicle B, vehicle C, and vehicle D. Similarly, in a shopping mall crowd monitoring application scenario, the target object is a pedestrian. The object area of the target object represents the range of the target object in the video data and the radar data. In some specific application scenarios, in order to facilitate calculation and analysis, a regular graphic area (such as a rectangular area, a circular area, etc.) similar to the object area can be directly used to represent the object area.

[0028] Step S12: Determine the first object area of the target object in the video image, and determine the second object area of the target object in the radar point cloud image, and establish a region matching relationship between the first object area and the second object area.

[0029] In one implementation scenario, the region matching relationship between the first object area and the second object area can be established according to the relative position of the target object to the static reference in the video image and the radar point cloud image and the size information thereof.

[0030] In a specific implementation scenario, the target object E and the static reference F are included in the video image, and the length of the static reference F in the video coordinate system is L1, and the target object E is located 4 times the length of the static reference (4*L1) above the static reference F; the target object G and the static reference F are included in the radar point cloud image, and the length of the static reference F in the radar coordinate system is L2, and the target object G is located 4 times the length of the static reference (4*L2) above the static reference F. At this time, the region matching relationship can be obtained as the target object E and the target object G matching.

[0031] In another implementation scenario, the target object overlap degree can be obtained based on the first object region and the second object region, and the region matching relationship of the target moment can be obtained based on the maximum target object overlap degree. Therefore, by calculating the target object overlap degree and determining the matching relationship according to the maximum target object overlap degree, the quantitative calculation and analysis of the entire region matching relationship of the target object can be performed, and the accuracy of the region matching relationship can be improved.

[0032] Further, please refer to Figure 3 , Figure 3 is a flowchart of an embodiment of obtaining a target object overlap degree. Specifically, the obtaining of the target object overlap degree in the above embodiment can include the following steps:

[0033] Step S1201: Constructing a data mapping relationship between the camera device and the radar device.

[0034] The video data of the camera device and the radar data of the radar device are located in different coordinate systems, and in order to realize the correspondence of the first object region and the second object region, the data mapping relationship between the camera device and the radar device needs to be constructed.

[0035] In a specific implementation scenario, a camera internal and external parameter calibration method, a four-point calibration method, or the like can be used to construct the data mapping relationship between the camera device and the radar device.

[0036] Step S1202: Mapping the first object region of the target moment based on the data mapping relationship to obtain the first region of the target object mapped to the radar coordinate system.

[0037] For ease of understanding, taking the target object as a vehicle, the object region as a rectangle, and the number of target objects as 1 as an example, the description is made. Please refer to Figure 4 , Figure 4 is a schematic diagram of mapping the first object region to the radar coordinate system. As Figure 4As shown, the left side is the first object region, and the right side is the radar coordinate system. Since the object region is rectangle D, only the coordinates of four end points (①, ②, ③, ④) are needed to determine the object region. In order to improve the mapping efficiency when mapping based on the data mapping relationship, the diagonal points ① and ③ are selected for mapping, and then the rectangle region D is redrawn according to the diagonal points ① and ③ to obtain the first region, so as to realize the mapping of the whole object region. Of course, diagonal points ② and ④ can also be selected for mapping, or all end points can be selected for mapping, which is not specifically limited here. Further, please refer to Figure 4 , Figure 5 , Figure 5 is a schematic diagram of the second object region based on Figure 4 . As shown in Figure 5 , the rectangle object region A is drawn in the radar coordinate system to obtain the second object region. The length and width of the rectangle region A are wR and hR, and the length and width of the rectangle region D are hV and wV.

[0038] Step S1203: obtaining the target object coincidence degree based on the first region and the second object region.

[0039] In one implementation scenario, the target object coincidence degree can be obtained by calculating the overlapping area of the first region and the second object region, and then based on the ratio of the overlapping area to the smaller value of the areas of the first region and the second object region. Therefore, the target object coincidence degree can be intuitively reflected by the ratio of the areas, and the target object coincidence degree can be more accurately obtained through quantitative calculation.

[0040] Further, the area of the first region is denoted as S v , the area of the second object region is denoted as S r , the overlapping area is denoted as S O , and the target object coincidence degree is denoted as Dis. At this time, the target object coincidence degree can be represented by formula (1):

[0041]

[0042] In other implementation scenarios, similarly, the target object coincidence degree can also be obtained by calculating the overlapping area of the first region and the second object region, and then based on the ratio of the overlapping area to the larger value of the areas of the first region and the second object region; or the target object coincidence degree can also be obtained by calculating the overlapping area of the first region and the second object region, and then based on the ratio of the overlapping area to the total coverage area of the first region and the second object region. The calculation method of the target object coincidence degree is not specifically limited here.

[0043] In the above scheme, by constructing a data mapping relationship between the camera device and the radar device, the first object region is mapped to the radar coordinate system, and the target object coincidence degree of the two object regions is calculated in the same coordinate system, thereby greatly improving the accuracy and convenience of the target object coincidence degree calculation.

[0044] It should be noted that when the number of target objects is greater than 1, when one target object is mapped to the radar coordinate system to obtain a first region, there can be a second object region of multiple target objects in the radar point cloud map that coincides with the first region. At this time, the region matching relationship can be determined according to the maximum target object coincidence degree. The maximum target object coincidence degree indicates that the coincident region of the first region and the second object region is the largest. Therefore, the corresponding target object in the video image and the corresponding target object in the radar point cloud map are the most likely to be the same target object. In this application, they can be considered as the same target object, and the coincident region can be taken as the second object region of the target object.

[0045] Step S13: Comparing the region matching relationship with a recorded reference region matching relationship to obtain a comparison result.

[0046] In one implementation scenario, the reference matching relationship is the region matching relationship of a previous time of the target time, and the comparison result is whether the region matching relationship and the reference region matching relationship are consistent. It can be understood that the region matching relationship indicates whether the target objects corresponding to the first object region and the second object region are the same target object. Therefore, the consistency of the region matching relationships of the previous time and the current time can indicate that the target objects corresponding to the region matching relationships of the previous time and the current time are the same target object.

[0047] In one specific implementation scenario, in order to meet the specific application requirement of monitoring, the region matching relationship needs to be obtained in real time.

[0048] In another specific implementation scenario, in an application scenario with low real-time requirement for information, the region matching relationship can be obtained intermittently at intervals. The interval can be 1 second, 5 seconds, or other specific time, which is not limited herein.

[0049] Step S14: Determining a target region of a target object at a target time based on the comparison result.

[0050] Please refer to Figure 6 , Figure 6 is Figure 1 a flowchart of one embodiment of step S14 in

[0051] Step S141: Based on the comparison result, obtaining a first object set, a second object set, and a third object set, respectively.

[0052] It needs to be understood that the first object set is constructed based on target objects with consistent region matching relationship between the before time and the after time, the second object set is constructed based on target objects of the before time with inconsistent region matching relationship between the before time and the after time, and the third object set is constructed based on target objects of the target time with inconsistent region matching relationship between the before time and the after time. Further, as mentioned above, the consistent region matching relationship between the before time and the after time indicates that the target objects corresponding to the region matching relationship between the before time and the after time are the same target object, and the first object set is the target object set determined as the same target object; the inconsistent region matching relationship between the before time and the after time indicates that the target objects corresponding to the region matching relationship between the before time and the after time are not the same target object, the target objects corresponding to the region matching relationship of the before time are constructed as the second object set, and the target objects corresponding to the region matching relationship of the target time are constructed as the third object set.

[0053] In one implementation scenario, the target association matrix can be constructed based on the comparison result, and the first object set, the second object set and the third object set can be obtained by deconstructing based on the target association matrix. Therefore, the comparison result can be intuitively reflected in the form of the matrix, and the deconstruction can be more quickly completed, and then the object sets are obtained.

[0054] Further, the method of constructing the target association matrix can include: in response to the region matching relationship of the ith target object of the target time being consistent with the region matching relationship of the jth target object of the before time, determining the element in the ith row and the jth column of the target association matrix as a first value; and in response to the region matching relationship of the ith target object of the target time being inconsistent with the region matching relationship of the jth target object of the before time, determining the element in the ith row and the jth column of the target association matrix as a second value; wherein the first value is different from the second value. Therefore, the region matching relationship between the target objects of the before time and the after time can be directly determined according to the value of each element in the matrix by constructing the target association matrix through the above method, and the method is more convenient and efficient.

[0055] For the constructed target association matrix, elements corresponding to the first value in the target association matrix can be selected, and target objects associated with the selected elements can be included in the first object set; columns in which all elements in the column are the second value in the target association matrix can be selected, and target objects of the before time associated with the selected columns can be included in the second object set; rows in which all elements in the row are the second value in the target association matrix can be selected, and target objects of the target time associated with the selected rows can be included in the third object set. Therefore, the target association matrix can be quickly deconstructed by observing the values of the elements in the target association matrix and simply analyzing the rows and columns, and the first object set, the second object set and the third object set required can be obtained.

[0056] In one specific implementation scenario, the first value can be set as 1 and the second value can be set as 0. The region matching relationship of the four target objects at the previous time instant is denoted as and The region matching relationship of the four target objects at the target time instant is denoted as and The comparison results are listed in the following table:

[0057]

[0058] The table is further simplified as follows:

[0059] 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0

[0060] The simplified table is constructed into a 4*4 target correlation matrix as follows:

[0061]

[0062] It can be seen that only the values of elements M 11 and M 22 in the matrix are 1, indicating that the region matching relationship of the first target object at the target time instant is consistent with that of the first target object at the previous time instant, and the region matching relationship of the second target object at the target time instant is consistent with that of the second target object at the previous time instant. At this time, as described above, it can be determined that the first target object at the target time instant and the first target object at the previous time instant are the same target object, and the second target object at the target time instant and the second target object at the previous time instant are the same target object, which can be included in the first object set. Further, it can be seen that the elements in the third and fourth columns of the matrix are all 0, indicating that the third target object and the fourth target object at the previous time instant do not match the region matching relationship of any target object at the target time instant, which can be included in the second object set. Similarly, it can also be seen that the elements in the third and fourth rows of the matrix are all 0, indicating that the third target object and the fourth target object at the target time instant do not match the region matching relationship of any target object at the previous time instant, which can be included in the third object set.

[0063] In another specific implementation scenario, the target correlation matrix can also be split into sub-matrices to represent the first object set, the second object set and the third object set. For example, the matrix M is split into the following three sub-matrices:

[0064]

[0065]

[0066]

[0067] Wherein, the sub-matrix M1 represents that the region matching relationship of the first target object at the target time is consistent with the region matching relationship of the first target object at the previous time, and the region matching relationship of the second target object at the target time is consistent with the region matching relationship of the second target object at the previous time, that is, the first object set; the sub-matrix M2 represents that the third target object and the fourth target object at the previous time fail to match the region matching relationship of any target object at the target time, and there is a possibility of matching with any unmatched target object at the target time, that is, the second object set; the matrix M3 represents that the third target object and the fourth target object at the target time fail to match the region matching relationship of any target object at the previous time, and there is a possibility of matching with any unmatched target object at the previous time, that is, the third object set.

[0068] In another implementation scenario, the data set can also be constructed based on the comparison result, and then the first object set, the second object set and the third object set can be obtained by using the traversal method.

[0069] Step S142: performing region prediction on the target objects in the second object set to obtain a predicted region.

[0070] In one implementation scenario, the motion speed of the target object can also be obtained from the video image or the radar point cloud image, and then the displacement distance can be obtained based on the time difference between the previous time and the target time and the motion speed of the target object in the second object set, and the predicted region can be obtained based on the displacement distance and the second object region of the target object in the second object set. Therefore, the possible displacement distance can be accurately predicted according to the speed and the time, and then the predicted region can be accurately obtained according to the displacement distance and the region information.

[0071] In one specific implementation scenario, the motion speed of the i-th target object at the previous time is The time difference between the previous time and the target time is Δt, and the second object region of the i-th target object in the second object set is At this time, the predicted region of the i-th target object at the previous time which fails to match the region matching relationship is It can be represented by formula (2):

[0072]

[0073] In another implementation scenario, the second object region of the target object in the second object set and the second object region of the target object in the third object set can be traversed to obtain a maximum distance difference between the two in the radar coordinate system, and a predicted region of the target object in the second object set can be obtained according to the maximum distance difference and the second object region of the target object in the second object set.

[0074] In one implementation scenario, after step S142 is performed and before step S143 is performed, the predicted region can be updated to delete part of the impossible predicted region.

[0075] Further, the predicted region can be updated based on the detection ranges of the camera device and the radar device, and / or the predicted region can be updated based on the second object region of the target object in the third object set and the maximum region deviation. Therefore, updating the predicted region in advance before performing the subsequent steps can help to improve the efficiency of the subsequent work on the one hand, and deleting the impossible predicted region can help to improve the accuracy of the subsequent determination of the target region on the other hand.

[0076] In one implementation scenario, updating the predicted region based on the detection ranges of the camera device and the radar device can include: deleting the predicted region in response to the predicted region exceeding the detection range of the camera device or the radar device; and retaining the predicted region in response to the predicted region not exceeding the detection range of the camera device and the radar device. Therefore, by judging whether the predicted region exceeds the detection range of the camera device or the radar device, and then deciding to retain or delete the current predicted region, the impossible predicted region can be directly and explicitly deleted, which is conducive to the subsequent analysis based on the predicted region.

[0077] In one specific implementation scenario, the detection range of the camera device is The detection range of the radar device is The predicted region of the i th target object that fails to match the region matching relationship at the previous moment is When When formula (3) or formula (4) is satisfied, the predicted region is deleted.

[0078]

[0079]

[0080] When When formula (5) and formula (6) are both satisfied, the predicted region is retained.

[0081]

[0082]

[0083] In another implementation scenario, updating the prediction region based on the object region of the target object in the third object set and the maximum region deviation can include: determining a region deviation range of the prediction region based on the second object region of the target object in the third object set and the maximum region deviation, deleting the prediction region in response to the prediction region exceeding the region deviation range, and retaining the prediction region in response to the prediction region not exceeding the region deviation range. It should be understood that the maximum region deviation represents the maximum deviation that can occur to the target object. Therefore, based on the object region of the target object that fails to match the region matching relationship at the target time and taking the maximum region deviation into full consideration, the region deviation range at the target time can be obtained, and then the prediction region can be screened according to the region deviation range, which can accurately delete the impossible prediction region and is beneficial to subsequent analysis based on the prediction region.

[0084] In a specific implementation scenario, the maximum region deviation is Δp, the object region of the i th target object that fails to match the region matching relationship at the target time is the prediction region of the i th target object that fails to match the region matching relationship at the previous time is When When formula (7) is satisfied, the prediction region is retained, otherwise the prediction region is deleted.

[0085]

[0086] Step S143: obtaining a target region of the target object at the target time based on the prediction region, the first object set, and the third object set.

[0087] It should be noted that the prediction region in step S143 can be the prediction region directly predicted in step S142, or in some implementation scenarios, the updated prediction region can be used as a new prediction region after the prediction region is updated.

[0088] In the above scheme, the first object set that has successfully matched the region matching relationship, the second object set that fails to match the region matching relationship at the previous time, and the second object set that fails to match the region matching relationship at the target time are split based on the comparison result, the target object in the second object set is region-predicted to obtain a prediction region, and finally the target region of the target object at the target time is obtained based on the prediction region, the first object set, and the third object set. Therefore, in addition to the region information of the target object at the target time, the second object region of the target object that fails to match at the previous time is also predicted, so that the target region of the target object at the target time is more comprehensive and accurate.

[0089] In one implementation scenario, the prediction region set can be constructed according to the prediction region. The target region of the target object at the target time is also expressed in the form of a set, denoted as a target region set. The sum of the prediction region set, the first object set and the third object set is the target region set.

[0090] In one specific implementation scenario, the prediction region set is U curEsttrg , the first object set is U curtrg , and the third object set is U curnstrg . At this time, the target region set U outtrg can be expressed by formula (8):

[0091] U outtrg = U curEsttrg + U curtrg + U curntrg ……(8)

[0092] In another implementation scenario, the target region of the target object at the target time is also expressed in the form of a set. At this time, the prediction region can also be clustered to obtain a clustered region set. Similarly, the sum of the clustered region set, the first object set and the third object set is denoted as the target region set.

[0093] In the above scheme, the first object region of the target object at the target time collected by the camera device and the second object region of the target object collected by the radar device are obtained to obtain the region matching relationship at the target time. The region matching relationship and the reference matching relationship are compared, and the target region of the target object at the target time is determined according to the comparison result. The comparison result considers the region matching relationship of the target object at the target time and before the target time in the video and the radar, and can accurately determine whether the target objects at the target time and before the target time are the same target object according to whether the region matching relationship is consistent. For the target object that fails to be successfully matched at the previous time, the historical data at the previous time and the object region confirmed at the target time can be further referred to for reasonable analysis and prediction, so that the target region can be predicted as accurately as possible under limited detection conditions.

[0094] Please refer to Figure 7 , Figure 7 is a flowchart of another embodiment of the target region prediction method of the present application. Specifically, the target region prediction method in the present embodiment can include the following steps:

[0095] Step S201: initialization.

[0096] When the target region prediction method starts to execute, initialization is first performed, including setting a threshold value, self-checking of the camera device, self-checking of the radar device, etc.

[0097] Step S202: camera internal and external parameters / four-point marking.

[0098] Corresponding to the "establishing a data mapping relationship between the camera device and the radar device" in the foregoing embodiments, the specific implementation can refer to the foregoing embodiments, which will not be described here again.

[0099] Step S203: video target detection.

[0100] The number and position of the target are detected by using an artificial intelligence method such as deep learning, and each target is assigned a unique identity information.

[0101] Step S204: video target tracking.

[0102] The video target is tracked to ensure that the same target has the same identity information.

[0103] Step S205: video target frame extraction.

[0104] According to the position of the target frame in the video, the upper left point and the lower right point of the target frame are extracted.

[0105] Step S206: radar target detection.

[0106] The detection data of the radar device is obtained.

[0107] Step S207: clustering.

[0108] The number of targets and the point cloud data corresponding to each target are obtained by using a density clustering method.

[0109] Step S208: radar target region extraction.

[0110] The rectangular outer contour of each radar target point cloud data is determined.

[0111] Step S209: time synchronization.

[0112] Based on the timestamp, paired radar and video data can be obtained.

[0113] Step S210: fuse the radar target and the video target as the fusion result of the current frame.

[0114] Corresponding to the "establishing a region matching relationship between the first object region and the second object region" in the foregoing embodiments, the specific implementation can refer to the foregoing embodiments, which will not be described here again.

[0115] Step S211: compare the previous frame fusion result and the current frame fusion result and establish a correlation matrix.

[0116] The comparison of the region matching relationship with the recorded reference region matching relationship and the construction of the target correlation matrix based on the comparison result can refer to the foregoing embodiments and will not be described here again.

[0117] Step S212: disassemble the correlation matrix.

[0118] The construction of the first object set, the second object set and the third object set based on the target correlation matrix can refer to the foregoing embodiments and will not be described here again.

[0119] Step S213: position estimation of the target in the unassociated list in the previous frame.

[0120] The region prediction of the target object in the second object set can refer to the foregoing embodiments and will not be described here again.

[0121] Step S214: select the best estimation result as the actual target of the current frame.

[0122] The construction of the target region of the target object at the target time based on the predicted region, the first object set and the third object set can refer to the foregoing embodiments and will not be described here again.

[0123] In the foregoing scheme, the first object region of the target object collected by the camera device at the target time and the second object region of the target object collected by the radar device are used to obtain the region matching relationship at the target time, and the region matching relationship and the reference matching relationship are compared, and the target region of the target object at the target time is determined according to the comparison result. The comparison result comprehensively considers the region matching relationship of the target object at the target time and at the previous time in the video and the radar, and can accurately determine whether the target objects at the previous time and the target time are the same target object according to whether the region matching relationship is consistent. For the target object that fails to be successfully matched at the previous time, the historical data at the previous time and the object region confirmed at the target time can be further referred to for reasonable analysis and prediction, so that the target region can be as accurately as possible predicted under limited detection conditions.

[0124] Please refer to Figure 8 , Figure 8is a schematic diagram of a framework of an embodiment of the target region prediction apparatus 80. Specifically, the target region prediction apparatus 80 comprises a data acquisition module 81, a matching relationship establishment module 82, a comparison module 83, and a target region determination module 84. Further, the data acquisition module 81 is configured to acquire a video image and a radar point cloud image collected by a camera device and a radar device at a target time, respectively; wherein the video image and the radar point cloud image comprise a target object; the matching relationship establishment module 82 is configured to determine a first object region of the target object in the video image, determine a second object region of the target object in the radar point cloud image, and establish a region matching relationship between the first object region and the second object region; the comparison module 83 is configured to compare the region matching relationship with a recorded reference region matching relationship to obtain a comparison result; wherein the reference region matching relationship is a region matching relationship at a time prior to the target time; the comparison result comprises whether the region matching relationship is consistent with the reference region matching relationship; and the target region determination module 84 is configured to determine a target region of the target object at the target time based on the comparison result.

[0125] In the above scheme, the region matching relationship at the target time is obtained by the first object region of the target object collected by the camera device and the second object region of the target object collected by the radar device at the target time, and the region matching relationship and the reference matching relationship are compared, and the target region of the target object at the target time is determined according to the comparison result. The comparison result comprehensively considers the region matching relationship of the target object in the video and the radar at the target time and the time prior to the target time, and can accurately determine whether the target objects at the time prior to the target time and the target time are the same target object according to whether the region matching relationship is consistent. For the target object that fails to be successfully matched at the time prior to the target time, the historical data at the time prior to the target time and the object region confirmed at the target time can be further referred to for reasonable analysis and prediction, so that the target region can be as accurately as possible predicted under limited detection conditions.

[0126] In some disclosed embodiments, the target region determination module 84 comprises an object set acquisition unit and a target region prediction unit. The object set acquisition unit is configured to acquire a first object set, a second object set, and a third object set based on the comparison result, respectively; wherein the first object set is constructed based on target objects whose region matching relationships at any two times prior to and after the target time are consistent, the second object set is constructed based on target objects at the time prior to the target time whose region matching relationships at any two times prior to and after the target time are inconsistent, and the third object set is constructed based on target objects at the target time whose region matching relationships at any two times prior to and after the target time are inconsistent; the target region prediction unit is configured to perform region prediction on the target objects in the second object set to obtain a predicted region; and the target region determination module 84 is configured to obtain the target region of the target object at the target time based on the predicted region, the first object set, and the third object set.

[0127] Therefore, in addition to the target object area information of the target moment, the second object area of the target object which fails to match at the previous moment is predicted, so that the target area of the target object at the target moment is more comprehensive and accurate.

[0128] In some disclosed embodiments, the target area determination module 84 further includes an association matrix construction unit. The association matrix construction unit is configured to construct a target association matrix based on the comparison result. The object set acquisition unit is further configured to deconstruct the target association matrix based on the target association matrix to obtain the first object set, the second object set and the third object set.

[0129] Therefore, the comparison result can be intuitively reflected in the form of a matrix, and the deconstruction can be more quickly completed, and thus the object sets are obtained.

[0130] In some disclosed embodiments, the association matrix construction unit is configured to determine an element in the i-th row and the j-th column of the target association matrix as a first numerical value in response to the region matching relationship of the i-th target object at the target moment being consistent with the region matching relationship of the j-th target object at the previous moment, and determine the element in the i-th row and the j-th column of the target association matrix as a second numerical value in response to the region matching relationship of the i-th target object at the target moment being inconsistent with the region matching relationship of the j-th target object at the previous moment; and the first numerical value is different from the second numerical value.

[0131] Therefore, the target association matrix constructed by the above method can intuitively determine whether the region matching relationship of the target object at the previous moment is consistent with that at the target moment according to the numerical value of each element in the matrix, and is more convenient and efficient.

[0132] In some disclosed embodiments, the object set acquisition unit is further configured to select an element corresponding to the first numerical value in the target association matrix, and include the target object associated with the selected element into the first object set; select a column in which all elements in the column are the second numerical value in the target association matrix, and include the target object at the previous moment associated with the selected column into the second object set; and select a row in which all elements in the row are the second numerical value in the target association matrix, and include the target object at the target moment associated with the selected row into the third object set.

[0133] Therefore, only the numerical value of the element in the target association matrix needs to be observed, and a simple analysis according to the row and the column can quickly deconstruct the target association matrix to obtain the required first object set, the second object set and the third object set.

[0134] In some disclosed embodiments, the data acquisition module 81 is further configured to acquire the moving speed of the target object, and the target region prediction unit further comprises a displacement calculation subunit. The displacement calculation subunit is configured to obtain a displacement distance based on a time difference between the previous time and the target time and the moving speed of the target object in the second object set; and the target region prediction unit is configured to obtain the predicted region based on the displacement distance and the object region of the target object in the second object set.

[0135] Therefore, the displacement distance that is likely to occur can be accurately predicted according to the speed and the time, and then the predicted region can be accurately obtained according to the displacement distance and the region information.

[0136] In some disclosed embodiments, the target region determination module 84 further comprises a target region updating unit. The target region updating unit is configured to update the predicted region based on the detection ranges of the camera device and the radar device; and / or, update the predicted region based on the second object region of the target object in the third object set and the maximum region deviation, wherein the maximum region deviation represents the maximum deviation that is likely to occur for the target object. The target region determination module 84 is further configured to obtain the target region of the target object at the target time based on the updated predicted region, the first object set and the third object set.

[0137] Therefore, updating the predicted region in advance is helpful to improve the efficiency of subsequent work on the one hand, and to delete the impossible predicted region on the other hand, which is conducive to the accuracy of subsequent determination of the target region.

[0138] In some disclosed embodiments, the target region updating unit is further configured to delete the predicted region in response to the predicted region exceeding the detection range of the camera device or the radar device; and retain the predicted region in response to the predicted region not exceeding the detection range of the camera device and the radar device.

[0139] Therefore, by judging whether the predicted region exceeds the detection range of the camera device or the radar device, and then deciding to retain or delete the current predicted region, the impossible predicted region can be directly and explicitly deleted, which is conducive to the analysis based on the predicted region subsequently.

[0140] In some disclosed embodiments, the target region updating unit is further configured to determine a region deviation range of the predicted region based on the object region of the target object in the third object set and the maximum region deviation; delete the predicted region in response to the predicted region exceeding the region deviation range; and retain the predicted region in response to the predicted region not exceeding the region deviation range.

[0141] Therefore, on the basis of the second object region of the target object failing to match the region matching relationship at the target moment and fully considering the maximum region deviation, the region deviation range at the target moment can be obtained, and then the prediction region is screened according to the region deviation range, so that the impossible prediction region can be accurately deleted, which is beneficial to subsequent analysis based on the prediction region.

[0142] In some disclosed embodiments, the matching relationship establishing module 82 comprises a coincidence degree calculation unit. The coincidence degree calculation unit is configured to obtain the target object coincidence degree based on the first object region and the second object region. The matching relationship establishing module 82 is configured to obtain the region matching relationship at the target moment based on the maximum target object coincidence degree.

[0143] Therefore, by calculating the target object coincidence degree and determining the matching relationship according to the maximum target object coincidence degree, the entire region matching relationship of the target object can be quantitatively calculated and analyzed, and the accuracy of the region matching relationship is improved.

[0144] In some disclosed embodiments, the coincidence degree calculation unit comprises a mapping relationship construction subunit and a first region acquisition subunit. The mapping relationship construction subunit is configured to construct the data mapping relationship between the camera device and the radar device. The first region acquisition subunit is configured to map the first object region at the target moment based on the data mapping relationship to obtain the first region of the target object mapped to the radar coordinate system. The coincidence degree calculation unit is configured to obtain the target object coincidence degree based on the first region and the second object region.

[0145] Therefore, by constructing the data mapping relationship between the camera device and the radar device, the first object region is mapped to the radar coordinate system, and the target object coincidence degree of the two object regions is calculated in the same coordinate system, which greatly improves the accuracy and convenience of the target object coincidence degree calculation.

[0146] In some disclosed embodiments, the coincidence degree calculation unit is further configured to calculate the overlapping region area of the first region and the second object region, and obtain the target object coincidence degree based on the ratio of the overlapping region area to the smaller value of the area of the first region and the area of the second object region.

[0147] Therefore, the target object coincidence degree can be intuitively reflected by the ratio of the areas, and the target object coincidence degree can be more accurately obtained by quantitative calculation.

[0148] Please refer to Figure 9 , Figure 9 is a frame diagram of an embodiment of the electronic device 90. Specifically, the electronic device 90 comprises a processor 901 and a memory 902, the memory 902 is coupled to the processor 901, and the processor 901 is configured to execute the program instructions stored in the memory 902 to implement the steps in any embodiment of the target region prediction method.

[0149] Specifically, the processor 901 can also be called a CPU (Central Processing Unit). The processor 901 can be an integrated circuit chip having a processing capability of signals. The processor 901 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. In addition, the processor 901 can be implemented by a plurality of circuit chips collectively.

[0150] In the above scheme, the region matching relationship at the target time is obtained by the first object region of the target object collected by the camera device at the target time and the second object region of the target object collected by the radar device, and the region matching relationship and the reference matching relationship are compared, and the target region of the target object at the target time is determined according to the comparison result. The comparison result considers the region matching relationship of the target object at the target time and the previous time in the video and the radar, and can accurately determine whether the target objects at the previous time and the target time are the same target object according to whether the region matching relationship is consistent. For the target object that fails to match successfully at the previous time, the historical data at the previous time and the object region confirmed at the target time can be further referred to for reasonable analysis and prediction, so that the target region can be predicted as accurately as possible under limited detection conditions.

[0151] Please refer to Figure 10 , Figure 10 is a schematic diagram of the framework of an embodiment of the computer readable storage medium 10. In this embodiment, the computer readable storage medium 10 stores processor executable program instructions 1001 for executing the steps in the above target region prediction method embodiment.

[0152] The computer readable storage medium 10 can specifically be a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, etc. a medium that can store program instructions, or also can be a server that stores the program instructions, the server can send the stored program instructions to other devices for running, or also can run the stored program instructions.

[0153] In the scheme, the first object region of the target object collected by the camera device at the target moment and the second object region of the target object collected by the radar device are used to obtain the region matching relationship at the target moment, and the region matching relationship and the reference matching relationship are compared, and the target region of the target object at the target moment is determined according to the comparison result. The comparison result considers the region matching relationship of the target object at the target moment and at the previous moment in the video and the radar, and can accurately determine whether the target objects at the previous moment and the target moment are the same target object according to whether the region matching relationship is consistent. For the target object that fails to be successfully matched at the previous moment, the object region at the target moment can be further analyzed and predicted by referring to the historical data at the previous moment and the object region confirmed at the target moment. Therefore, the target region can be accurately predicted as much as possible under limited detection conditions.

[0154] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic, and the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0155] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0156] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0157] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0158] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as agreeing to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious mark / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload the personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

Claims

1. A target region prediction method characterized by, The method comprises: acquiring a video image and a radar point cloud image collected by a camera device and a radar device at a target time; wherein the video image and the radar point cloud image comprise a target object; determining a first object region of the target object in the video image and a second object region of the target object in the radar point cloud image, and establishing a region matching relationship between the first object region and the second object region; comparing the region matching relationship with a recorded reference region matching relationship to obtain a comparison result; wherein the reference region matching relationship is a region matching relationship at a previous time of the target time; the comparison result comprises whether the region matching relationship is consistent with the reference region matching relationship; based on the comparison result, acquiring a first object set, a second object set and a third object set; wherein the first object set is constructed based on the target objects whose region matching relationships at any two of the previous and subsequent times are consistent, the second object set is constructed based on the target objects at the previous time whose region matching relationships at any two of the previous and subsequent times are inconsistent, and the third object set is constructed based on the target objects at the target time whose region matching relationships at any two of the previous and subsequent times are inconsistent; performing region prediction on the target objects in the second object set to obtain a predicted region; based on the predicted region, the first object set and the third object set, obtaining a target region of the target object at the target time.

2. The method of claim 1, wherein, The method based on the comparison result, acquiring a first object set, a second object set and a third object set, comprises: based on the comparison result, constructing a target correlation matrix; based on the target correlation matrix, deconstructing to obtain the first object set, the second object set and the third object set.

3. The method of claim 2, wherein, The method based on the comparison result, constructing a target correlation matrix, comprises: in response to the region matching relationship of the i-th target object at the target time being consistent with the region matching relationship of the j-th target object at the previous time, determining that the element in the i-th row and the j-th column of the target correlation matrix is a first value; and in response to the region matching relationship of the i-th target object at the target time being inconsistent with the region matching relationship of the j-th target object at the previous time, determining that the element in the i-th row and the j-th column of the target correlation matrix is a second value; wherein the first value is different from the second value.

4. The method of claim 3, wherein, The method based on the target correlation matrix, deconstructing to obtain the first object set, the second object set and the third object set, comprises: selecting elements corresponding to the first value in the target correlation matrix, and including the target objects associated with the selected elements into the first object set; selecting columns in the target correlation matrix where all elements in the columns are the second value, and including the target objects at the previous time associated with the selected columns into the second object set; Selecting a row in the target association matrix in which all elements are the second value, and including the target object associated with the selected row at the target time into the third object set.

5. The method of claim 1, wherein, The method further includes obtaining a motion speed of the target object, and performing region prediction on the target object in the second object set to obtain a predicted region, including: obtaining a displacement distance based on a time difference between the previous time and the target time and the motion speed of the target object in the second object set; obtaining the predicted region based on the displacement distance and the second object region of the target object in the second object set.

6. The method of claim 1, wherein, After the region prediction on the target object in the second object set to obtain a predicted region, and before obtaining the target region of the target object at the target time based on the predicted region, the first object set and the third object set, the method further includes: updating the predicted region based on the detection ranges of the camera device and the radar device; and / or, updating the predicted region based on the second object region of the target object in the third object set and a maximum region deviation, wherein the maximum region deviation represents a maximum deviation that the target object can have; obtaining the target region of the target object at the target time based on the predicted region, the first object set and the third object set, including: obtaining the target region of the target object at the target time based on the updated predicted region, the first object set and the third object set.

7. The method of claim 6, wherein, updating the predicted region based on the detection ranges of the camera device and the radar device, including: deleting the predicted region in response to the predicted region exceeding the detection range of the camera device or the radar device; retaining the predicted region in response to the predicted region not exceeding the detection range of the camera device and the radar device.

8. The method of claim 6, wherein, updating the predicted region based on the second object region of the target object in the third object set and a maximum region deviation, including: determining a region deviation range of the predicted region based on the second object region of the target object in the third object set and a maximum region deviation; deleting the predicted region in response to the predicted region exceeding the region deviation range; retaining the predicted region in response to the predicted region not exceeding the region deviation range.

9. The method of claim 1, wherein, establishing the region matching relationship between the first object region and the second object region, including: obtaining a target object coincidence degree based on the first object region and the second object region; obtaining the region matching relationship at the target time based on the maximum target object coincidence degree.

10. The method of claim 9, wherein, obtaining a target object coincidence degree based on the first object region and the second object region, including: constructing a data mapping relationship between the camera device and the radar device; mapping the first object region at the target time based on the data mapping relationship to obtain a first region in which the target object is mapped to a radar coordinate system; Obtain the target object coincidence degree based on the first region and the second object region.

11. The method of claim 10, wherein, The obtaining of the target object coincidence degree based on the first region and the second object region comprises: Calculating the area of the overlapping region of the first region and the second object region; Obtaining the target object coincidence degree based on the ratio of the area of the overlapping region to the smaller one of the area of the first region and the area of the second object region.

12. A target region prediction apparatus characterized by comprising: Comprise: The data acquisition module is used for obtaining video images and radar point cloud images collected by a camera device and a radar device at a target time; wherein the video images and the radar point cloud images comprise a target object; The matching relationship establishment module is used for determining a first object region of the target object in the video images and determining a second object region of the target object in the radar point cloud images, and establishing a region matching relationship between the first object region and the second object region; The comparison module is used for comparing the region matching relationship with a recorded reference region matching relationship to obtain a comparison result; wherein the reference region matching relationship is a region matching relationship at a previous time of the target time; the comparison result comprises whether the region matching relationship is consistent with the reference region matching relationship; The target region determination module is used for obtaining a first object set, a second object set and a third object set based on the comparison result; wherein the first object set is constructed based on the target objects whose region matching relationships at any two of the previous time and the target time are consistent, the second object set is constructed based on the target objects at the previous time whose region matching relationships at any two of the previous time and the target time are inconsistent, and the third object set is constructed based on the target objects at the target time whose region matching relationships at any two of the previous time and the target time are inconsistent; the target region of the target object at the target time is obtained based on the predicted region, the first object set and the third object set.

13. An electronic device, comprising: The processor and the memory are coupled with each other, and the processor is used for executing program instructions stored in the memory to realize the target region prediction method in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The program instructions stored in the memory can be run by the processor, and the program instructions are used for realizing the target region prediction method in any one of claims 1-11.

Images