Image processing method, device and apparatus

Abstract

The present application relates to the field of millimeter wave radar and video calibration technology, and in particular to an image processing method, device and apparatus for providing an image processing solution that can obtain higher precision. In an embodiment of the present application, each frame of radar point cloud image in a radar point cloud image sequence collected by a radar device and each frame of video image in a video image sequence collected by a camera device are respectively subjected to target detection, and the target object in each frame of radar point cloud image and each frame of video image are identified. Based on the position information of the target object in each frame of radar point cloud image and each frame of video image, and a pre-constructed optimization objective function, the transformation matrix between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device is determined; based on the transformation matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence are fused.

Description

Technical Field

[0001] The present application relates to the technical field of millimeter-wave radar and video calibration, and in particular to an image processing method, equipment and device. Background Art

[0002] Traffic is the driving force behind a city's operations. Everyone in the city cannot live without traffic. With the increasing number of vehicles, obtaining accurate road traffic information is an important prerequisite for road traffic detection.

[0003] Currently, a common method of obtaining road traffic information in life is to capture video images through a camera and analyze the captured video images to obtain vehicle information in road traffic. However, since the environmental information collected by the camera itself is relatively single, the accuracy of the vehicle information is affected. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, device and apparatus to provide an image processing solution that can achieve higher precision.

[0005] In a first aspect, the present application provides an image processing method, the method comprising:

[0006] Performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, respectively, to identify the target object in each frame of the radar point cloud image and each frame of the video image;

[0007] Determining a conversion matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, where the position in the image is a position in the video image or a position in the radar point cloud image;

[0008] According to the conversion matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence are fused.

[0009] In a second aspect, the present application provides an image processing device, the device comprising at least one processor and at least one memory;

[0010] The memory stores program code, and when the program code is executed by the processor, the following process is implemented:

[0011] Performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, respectively, to identify the target object in each frame of the radar point cloud image and each frame of the video image;

[0012] Determining a conversion matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, where the position in the image is a position in the video image or a position in the radar point cloud image;

[0013] According to the conversion matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence are fused.

[0014] In a third aspect, the present application provides an image processing device, comprising:

[0015] an identification module for performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, and identifying the target object in each frame of the radar point cloud image and each frame of the video image;

[0016] a determination module, configured to determine a transformation matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on position information of the target object in each frame of the radar point cloud image, position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in a horizontal coordinate direction and a second weight coefficient in a vertical coordinate direction, the first weight coefficient corresponding to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponding to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, wherein the position in the image is a position in the video image or a position in the radar point cloud image;

[0017] A fusion module is used to fuse each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the conversion matrix.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to perform the image processing method described in the first aspect above.

[0019] In a fifth aspect, the present application provides a computer program product, including a computer program:

[0020] When the computer program is executed by a processor, the image processing method as described in the first aspect above is implemented.

[0021] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0022] Since the embodiment of the present application performs target detection on each frame of radar point cloud image collected by the radar device and each frame of video image collected by the camera device, respectively, identifies the target object in each frame of radar point cloud image and each frame of video image, and determines the conversion matrix between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device based on the position information of the target object in each frame of radar point cloud image and each frame of video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, and the first weight coefficient corresponds to the horizontal coordinate of the projection position of the target object and the position in the image The first weight coefficient and the second weight coefficient that affect the offset value of the target object's projection position in the horizontal direction and the offset value of the vertical direction in the target image are added to the optimization objective function, and the obtained transformation matrix can independently affect the fusion accuracy in the horizontal and vertical directions during image fusion, making the image fusion solution more flexible and able to obtain a variety of image fusion results with different accuracy requirements. Based on the image fusion result, vehicle information detection can be better performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1This is a schematic diagram of an application scenario of an optional image processing method according to an embodiment of the present application;

[0025] Figure 2 This is a schematic diagram of an application scenario of an optional image processing method according to an embodiment of the present application;

[0026] Figure 3 This is an image processing flow chart of an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a radar point cloud image containing a target object according to an embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of a detection point pairing result according to an embodiment of the present application;

[0029] Figure 6 This is a schematic diagram of an image after fusion processing according to an embodiment of the present application;

[0030] Figure 7 This is an overall flow chart of image processing according to an embodiment of the present application;

[0031] Figure 8 This is a structural diagram of an image processing device according to an embodiment of the present application;

[0032] Figure 9 This is a structural diagram of an image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] To make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] Moreover, in the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0035] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0036] The following describes an application scenario of an optional image processing method provided in an embodiment of the present application with reference to the accompanying drawings.

[0037] like Figure 1 As shown, an optional application scenario of the image processing method in an embodiment of the present application includes an image processing device 10 and a terminal device 11 with a user interface.

[0038] The image processing device 10 includes a radar device 101 and a camera device 102;

[0039] The image processing device 10 collects a radar point cloud image sequence of the road environment through the radar device 101, and collects a video image sequence of the road environment through the camera device 102; performs target detection on each frame of the radar point cloud image sequence and each frame of the video image sequence, identifies the target object in each frame of the radar point cloud image, and the target object in each frame of the video image; determines the conversion between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function Matrix; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to the offset value between the projection position of the target object and the position in the image in the horizontal coordinate direction, the second weight coefficient corresponds to the offset value between the projection position of the target object and the position in the image in the vertical coordinate direction, and the position in the image is the position in the video image or the position in the radar point cloud image; according to the transformation matrix, each frame of the radar point cloud image in the radar point cloud image sequence and each frame of the video image in the video image sequence are fused, and the fused image data and the detection result of the target object are sent to the terminal device 11.

[0040] The terminal device 11 is used to display the fused image data and the detection results of the target object to the user.

[0041] like Figure 2 As shown, an optional application scenario of the image processing method in an embodiment of the present application includes an image processing device 10, a terminal device 11 with a user interface, and a server 12;

[0042] The image processing device 10 may be a roadside device installed along a road, or may be a detection device installed on a vehicle.

[0043] The image processing device 10 includes a radar device 101 and a camera device 102;

[0044] The image processing device 10 collects a radar point cloud image sequence of the road environment through the radar device 101 and sends the radar point cloud image sequence of the road environment to the server 12 .

[0045] The camera device 102 captures a video image sequence of the road environment and sends the video image sequence of the road environment to the server 12 .

[0046] The server 12 determines a conversion matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projection position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projection position of the target object and the position in the image, and the position in the image is a position in the video image or a position in the radar point cloud image; according to the conversion matrix, each frame of the radar point cloud image in the radar point cloud image sequence and each frame of the video image in the video image sequence are fused; based on the fused image data, a vehicle is detected in the road environment, and the fused image data and the detection result of the target object are sent to the terminal device 11.

[0047] The terminal device 11 is used to display the fused image data and the detection results of the target object to the user.

[0048] The server 12 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0049] The terminal device 11 is an electronic device used by the user, which may be a mobile phone, a computer, an intelligent voice interaction device, a smart home, a car terminal, etc.

[0050] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 or Figure 2The application scenario shown can also be used in other possible application scenarios, which are not limited in the embodiments of the present application.

[0051] like Figure 3 The following is a flowchart of an image processing method provided by this application. The specific steps are as follows:

[0052] Step S301: Target detection is performed on each frame of a radar point cloud image sequence collected by a radar device and each frame of a video image sequence collected by a camera device, respectively, to identify a target object in each frame of the radar point cloud image and each frame of the video image.

[0053] Among them, the road environment collected by the radar equipment and the road environment collected by the camera device should be the same road environment.

[0054] In an optional implementation, before step S301 , the radar device and the camera device may be pre-processed separately.

[0055] For example, a user may determine a position offset of the radar device based on its position relative to the center lane in the road environment. The user may then adjust the radar device based on the position offset and the angular deflection angle of the radar device so that the radar device is aligned with the center of the center lane in the road environment and the road environment image in the radar point cloud image sequence captured by the radar device is positive. Based on the adjusted radar device, the user may configure the number of lanes and vehicle width corresponding to the radar device according to the actual road environment, completing preprocessing of the radar device so that each frame of the radar point cloud image sequence contains the configured lane line information. For the camera device, the user may configure the lane lines corresponding to the camera device according to the actual road environment, completing preprocessing of the camera device so that each frame of the video image sequence contains the configured lane line information. The lane line information may be the position information of each lane line in the radar point cloud image and the video image.

[0056] After the embodiment of the present application completes the preprocessing of the radar equipment and the camera device, the installation and debugging personnel can send instructions to the vehicle detection equipment on the road to trigger the image processing equipment on the road to start the radar calibration process.

[0057] Step S302: Determine a transformation matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function.

[0058] The optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction. The first weight coefficient corresponds to the offset value between the projection position of the target object and the position in the image in the horizontal coordinate direction. The second weight coefficient corresponds to the offset value between the projection position of the target object and the position in the image in the vertical coordinate direction. The position in the image is the position in the video image or the position in the radar point cloud image.

[0059] It should be noted that a target object may include one or more detection points.

[0060] In some embodiments, the above step S302 specifically includes the following steps A1-A3:

[0061] A1. Pairing each frame of radar point cloud image in a radar point cloud image sequence and each frame of video image in a video image sequence according to the same acquisition time to determine multiple image sets.

[0062] One image set includes one frame of radar point cloud image and one frame of video image, and the radar point cloud image and video image in one image set are collected at the same time.

[0063] A2. For any image set, pair the target object in the radar point cloud image in the image set with the target object in the video image in the image set.

[0064] Optionally, the target object is a vehicle on the road.

[0065] In an optional embodiment, a radar point cloud image sequence is determined based on point cloud data of the road environment collected by the radar equipment, and target detection is performed on each frame of the radar point cloud image through a deep learning network to identify vehicles in each frame of the radar point cloud image; target detection is performed on each frame of the video image sequence through a deep learning network to identify vehicles in each frame of the video image, and vehicles on the road in each frame of the image video are marked with a rectangular frame.

[0066] like Figure 4 Shown is any frame of radar point cloud image in a radar point cloud image sequence of a road environment collected by a radar device, wherein the frame of radar point cloud image includes a vehicle and can show the user the vehicle's position on the road, the vehicle's running speed, and the vehicle's direction of movement, where the arrow indicates the vehicle's direction of movement.

[0067] During implementation, the point cloud position of the target object in the radar point cloud image in the image set is paired with the center point position of the lower edge of the rectangular detection box where the target object is located in the video image in the image set.

[0068] For any image set, a target object in the radar point cloud image and a target object in the corresponding matching video image are determined as a detection point pairing set.

[0069] In an embodiment of the present application, each vehicle in the radar point cloud image can be paired with each vehicle in the video image based on the coordinates of each vehicle in the radar point cloud image, and the relationship between the coordinates of each vehicle in the radar point cloud image and the lane line information configured in the radar point cloud image; as well as the coordinates of each vehicle in the video image, and the relationship between the coordinates of each vehicle in the video image and the lane line information configured in the video image, to obtain multiple detection point pairing sets.

[0070] Among them, the relationship between the coordinates of each vehicle in the radar point cloud image and the lane line information configured in the radar point cloud image is to determine which two lane lines each vehicle in the radar point cloud image is between based on the coordinates of each vehicle in the radar point cloud image and the lane line information configured in the radar point cloud image. The process of the photographic device determining the relationship between the coordinates of each vehicle in the video image and the configured lane line information is the same as the above and will not be repeated here.

[0071] For example, in an image set, if two vehicles are identified in the radar point cloud image collected by the radar device, based on the coordinates of the two vehicles in the radar point cloud image, and the relationship between the coordinates of each vehicle in the radar point cloud image and the lane line information configured in the radar point cloud image, it can be determined that vehicle No. 1 exists in lane 1 and vehicle No. 2 exists in lane 2; if two vehicles are identified in the video image collected by the camera device, based on the coordinates of the two vehicles in the video image, and the relationship between the coordinates of each vehicle in the video image and the lane line information configured in the video image, it can be determined that vehicle No. 3 exists in lane 1 and vehicle No. 4 exists in lane 2, then vehicle No. 1 in the radar point cloud image is paired with vehicle No. 3 in the video image, and vehicle No. 2 in the radar point cloud image is paired with vehicle No. 4 in the video image to determine the corresponding detection point pairing set.

[0072] like Figure 5 As shown in FIG. 1 , a schematic diagram of a detection point pairing result in an embodiment of the present application is shown in FIG. Figure 5 The left picture shows a frame of video image captured by the camera device, in which the vehicles on the road are marked with rectangular frames; Figure 5 The right figure shows the radar point cloud image collected by the radar equipment in the same image set as the video image. By pairing the detection points in the video image with the detection points in the radar point cloud image, three detection point pairing sets are determined, where each detection point pairing set is in Figure 6 It is represented by a double-arrow curve.

[0073] Optionally, in addition to the above-mentioned process of automatically pairing the various detection points of the target object, the radar point cloud image and video image in the image set can also be displayed to the user in the display interface of the terminal, and the user can manually pair the various detection points of the target object in the radar point cloud image with the various detection points of the target object in the video image according to the position information of the various detection points of the target object in the radar point cloud image and the video image.

[0074] A3. Determine the transformation matrix between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device based on the position information of the target object in the radar point cloud image after pairing in each image set, the position information of the target object in the video image, and a pre-constructed optimization objective function.

[0075] In the implementation, for the i-th detection point pairing set p in multiple detection point pairing sets, i , which includes the coordinates of the detection points in the radar point cloud image as [x i ,y i ], and the coordinates of the detection points in the video image are [u i , v i ].

[0076] In some embodiments, the transformation matrix is ​​a second transformation matrix for transforming the second coordinate system into the first coordinate system. The step of determining the second transformation matrix specifically includes B1-B2:

[0077] B1. For any image set, project the target object in the video image in the image set onto the radar point cloud image in the image set according to the initial second transformation matrix to obtain the projected position of the target object; and determine the offset value in the horizontal and vertical directions between the projected position of the target object and the position of the target object in the radar point cloud image.

[0078] The initial second conversion matrix is ​​a 3×3 matrix composed of unknown parameters. It should be noted that since a more accurate image processing effect needs to be obtained, the initial second conversion matrix needs to be a 3×3 matrix.

[0079] The projection position of the target object is obtained by formula 1; wherein, the video projection coordinate is the projection position of the target object in the video image projected onto the radar point cloud image in the image set.

[0080] [x ′ ;y ′ ; 1] = F(h V2R ,p v )——Formula 1

[0081] Among them, [x ′ ;y′ ; 1] represents the video projection coordinates, F represents the coordinate conversion function, p v is the homogeneous coordinate [u; v; 1] corresponding to the coordinate of the target object in the video image, h V2R represents the initial second transformation matrix.

[0082] In an optional implementation, the offset value in the horizontal coordinate direction and the offset value in the vertical coordinate direction between the projection position of the target object and the position of the target object in the video image are determined by calculating the Euclidean distance.

[0083] B2. Input the projection position of the target object determined by each image set and the position of the target object in the video image into a pre-constructed optimization objective function in the horizontal coordinate direction and the vertical coordinate direction; and determine the second transformation matrix when the function value corresponding to the optimization objective function is minimum.

[0084] It should be noted that, for the sake of convenience, the second transformation matrix determined when the function value corresponding to the optimization objective function is minimized is called the target second transformation matrix.

[0085] The optimization objective function of the pre-constructed second transformation matrix is ​​shown in Formula 2:

[0086]

[0087] Among them, h * V2R represents the target second transformation matrix, x i with y i Denotes the detection point pairing set p i The homogeneous coordinates [x i ;y i ;1], x i ′ with y i ′ Denotes the detection point set p i The corresponding video projection coordinates of the detection points in the video image [x i ;y i ′ ; 1], where D 2 (x i , x i ′ ) represents x i with x i ′ The Euclidean distance between 2 (y i ,y i ′ ) represents yi with y i ′ The Euclidean distance between them, w1 represents the first weight coefficient in the horizontal direction, w2 represents the second weight coefficient in the vertical direction, p i represents the i-th detection point pairing set in multiple detection point pairing sets, ∑ p∈pi () indicates the accumulation of the values ​​corresponding to all paired sets of detection points, and argmin indicates the operation of determining the variable value when the function value corresponding to the optimization objective function is the minimum.

[0088] During implementation, the user controls the projection accuracy of the target object in the video image in the x-axis coordinate direction and the y-axis coordinate direction in the first coordinate system corresponding to the radar device through the preset first weight coefficient and the second weight coefficient; if higher projection accuracy is required in the x-axis coordinate direction and / or the y-axis coordinate direction, the corresponding preset weight coefficient is increased.

[0089] In an optional implementation, w1 corresponding to the x-axis coordinate direction may be increased to improve the projection accuracy in the x-axis coordinate direction, thereby making the projection of the target object in the x-axis coordinate direction more accurate.

[0090] By constructing a numerical optimization solver, the optimization objective function of the second transformation matrix is ​​solved. The specific process is as follows:

[0091] Optionally, the embodiment of the present application can construct a numerical optimization solver through common numerical optimization algorithms such as gradient descent, quasi-Newton method, BFGS algorithm, etc. to obtain the target second transformation matrix.

[0092] The embodiment of the present application explains the process of obtaining the target second transformation matrix through a gradient descent algorithm.

[0093] Let the loss function L(h) = w1·D 2 (x i , x i ′ )+w2·D 2 (y i ,y i ′ ), loss function gradient Since h is a 3×3 matrix with 8 degrees of freedom, as shown in Formula 3:

[0094]

[0095] Wherein, based on the initial second conversion matrix, Formula 4 may be used as an initial value to determine a candidate second conversion matrix corresponding to the initial value.

[0096] In an optional implementation, h can be determined by formula 4 V2RAs the initial value of h.

[0097] [x;y;1]=h V2R [u; v; 1]——Formula 4

[0098] Among them, [u; v; 1] and [x; y; 1] are homogeneous coordinates. [x; y; 1] represents the homogeneous coordinates corresponding to the coordinates of the detection point in the radar point cloud image in the detection point pairing set, and [u; v; 1] represents the homogeneous coordinates of the coordinates of the detection point in the video image in the same detection point pairing set. h can be obtained by formula 4. V2R .

[0099] Since [u; v; 1] and [x; y; 1] are both homogeneous coordinates, then h R2V and h V2R Each has 8 degrees of freedom, and at least 4 detection point matching pairs are required to determine h V2R .

[0100] By using Formula 5, h is iteratively updated to determine the second transformation matrix corresponding to the minimum value of L(h).

[0101] h n+1 =h n -α·g(h n )——Formula 5

[0102] Among them, h n is the candidate second transformation matrix before updating, h n+1 represents the updated candidate second transformation matrix, α represents the iteration step size, g(h n ) indicates that the gradient of the loss function is in h n Take the value at.

[0103] According to the candidate second transformation matrix before and after the update, the corresponding loss function value is calculated respectively. When L(h n+1 ) <L(h n ), h n+1 The value of h * And save, continue to execute formula 6, and so on, until the optimization termination condition is met, and the final h * The target second transformation matrix when it is minimized.

[0104] The optimization termination condition is at least one of the following conditions:

[0105] Condition 1: The maximum number of iterations is reached.

[0106] Condition 2: L(h n+1 )>L(h n ) exceeds the preset number of times.

[0107] It should be noted that when L(h n+1 )>L(h n ) exceeds the preset number of times, it is considered that L(h n+1 ) no longer decreases.

[0108] In an optional embodiment, the conversion matrix is ​​a first conversion matrix for converting the first coordinate system into the second coordinate system, and the step of determining the first conversion matrix specifically includes C1-C2:

[0109] C1. For any image set, project the target object in the radar point cloud image in the image set onto the video image in the image set based on the initial first transformation matrix to obtain the projected position of the target object. Also determine the offset between the projected position of the target object and the position of the target object in the video image in the horizontal and vertical directions.

[0110] The initial first conversion matrix is ​​a 3×3 matrix composed of unknown parameters. It should be noted that, in order to obtain a more accurate image processing effect, the initial second conversion matrix needs to be a 3×3 matrix.

[0111] The projection position of the target object is obtained by formula 6, wherein the radar projection coordinate is the projection position of the target object in the radar point cloud image projected onto the video image in the image set.

[0112] [u′;v′;1]=F(h R2V ,p r )——Formula 6

[0113] Where [u′; v′; 1] represents the radar projection coordinates, F represents the coordinate conversion function, and p r is the homogeneous coordinate [x; y; 1] corresponding to the coordinates of the target object in the radar point cloud image, h R2V Represents the initial first transformation matrix.

[0114] In an optional embodiment, the offset value D in the horizontal coordinate direction between the projection position of the target object and the position of the target object in the video image is determined by calculating the Euclidean distance. 2 (u i ,u i ′ ), and the offset value D in the ordinate direction 2 (v i , v i ′ ).

[0115] C2. Input the offset values ​​of the projection position of the target object and the position of the target object in the video image in the horizontal direction and the vertical direction determined by each image set into a pre-constructed optimization objective function; and determine the first transformation matrix when the function value corresponding to the optimization objective function is minimum.

[0116] It should be noted that, for the sake of convenience, the first transformation matrix determined when the function value corresponding to the optimization objective function is minimized is called the target first transformation matrix.

[0117] According to the radar projection coordinates, the optimization objective function of the first conversion matrix is ​​constructed, see Formula 7:

[0118]

[0119] p∈pi

[0120] Among them, h * R2V represents the first transformation matrix, u i With v i Denotes the detection point pairing set p i The homogeneous coordinates corresponding to the position information of the detection point in the video image [u i ;v i ;1] in the information, u i ′ With v i ′ Denotes the detection point set p i The radar projection coordinates of the detection points in the radar point cloud image [u i ′ ;v i ′ ; 1], where D(u i ,u i ′ ) represents u i with u i ′ The Euclidean distance between i , v i ′ ) means v i With v i ′ The Euclidean distance between them, w1 represents the first weight coefficient in the horizontal direction, w2 represents the second weight coefficient in the vertical direction, p i represents the i-th detection point pairing set in multiple detection point pairing sets, ∑ p∈pi () indicates the accumulation of the values ​​corresponding to all paired sets of detection points, and argmin indicates the operation of determining the variable value when the function value corresponding to the optimization objective function is minimized.

[0121] Optionally, the embodiment of the present application can construct a numerical optimization solver through common numerical optimization algorithms such as gradient descent, quasi-Newton method, BFGS algorithm, etc. to obtain the target first transformation matrix.

[0122] The solution process is similar to the process of determining the target second transformation matrix, and will not be repeated here.

[0123] Optionally, after obtaining the target transformation matrix, the transformation matrix can be no longer calculated, and the radar calibration process can be completed based on the target transformation matrix; as time goes by, the vehicle detection equipment may be moved by external forces. At this time, the calibration accuracy of the radar point cloud image and the video image may not meet the use requirements of the image processing equipment. The installation and debugging personnel need to remotely send instructions to the image processing equipment to trigger the image processing equipment on the road to start the radar calibration process.

[0124] Step S303: fusing the radar point cloud images of each frame in the radar point cloud image sequence and the video images of each frame in the video image sequence according to the conversion matrix.

[0125] Among them, according to the target transformation matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence can be fused, specifically including the following two cases:

[0126] Case 1: The transformation matrix is ​​the first transformation matrix that transforms the first coordinate system into the second coordinate system:

[0127] For the radar point cloud image and the video image in any image set, according to the first transformation matrix, some or all of the pixels in the radar point cloud image are projected into the video image for fusion processing.

[0128] Optionally, if there is at least one detection point pairing set in the image set, the embodiment of the present application projects the pixel points in the detection point pairing set corresponding to the radar point cloud image into the video image, and fuses the information corresponding to the pixel points in the radar point cloud image into the video image.

[0129] Case 2: The transformation matrix is ​​the second transformation matrix that transforms the second coordinate system into the first coordinate system:

[0130] For the radar point cloud image and video image in any image set, according to the second transformation matrix, some or all of the pixels in the video image are projected into the radar point cloud image for fusion processing.

[0131] Optionally, if there is at least one detection point pairing set in the image set, the embodiment of the present application projects the pixel points in the detection point pairing set corresponding to the video image into the radar point cloud image, and fuses the information corresponding to the pixel points in the video image into the radar point cloud image.

[0132] For example, the information corresponding to the pixel points in the video image can be data such as the category attributes of the target vehicle, vehicle color, license plate information, driver's facial information, etc., and some or all of the above information can be fused into the radar point cloud image.

[0133] Based on the fused image data, the speed, movement direction, category attributes, license plate number and other data of the target vehicle can be detected.

[0134] In the process of determining the transformation matrix, the embodiment of the present application increases the weights of the projections corresponding to the x-axis coordinate direction and the y-axis coordinate direction, thereby simultaneously satisfying the projection accuracy of the x-axis coordinate direction and the projection accuracy of the y-axis coordinate direction corresponding to different weights. Among them, if the projection accuracy in the x-axis coordinate direction is low, the vehicle may be projected into the wrong lane line, and if the projection accuracy in the y-axis coordinate direction is low, the projected position may have an error of the front and back distance from the correct position. For example, if the coordinates of the vehicle captured by the camera device are projected into the first coordinate system corresponding to the radar device, in practice, by increasing the projection accuracy in the x-axis coordinate direction, the coordinates of the vehicle in the video image can be projected into the correct lane among the multiple lanes contained in the radar point cloud image, and by adjusting the accuracy in the x-axis coordinate direction and the y-axis coordinate direction, the projection accuracy effect required by the user can be flexibly met.

[0135] According to the process of vehicle detection on the road, the embodiment of the present application can apply the corresponding vehicle detection equipment to road traffic, and can realize the functions of counting the traffic flow in road traffic, identifying the license plates of speeding vehicles, and detecting and capturing speeding lane changes of vehicles.

[0136] By connecting the terminal to the vehicle detection device through communication, the user can view the fused image data and the detection results of the target object on the terminal. For example, the user connects the mobile phone to the vehicle detection device through communication, and views the fused image data on the mobile phone, or displays the fused image data on the display. Figure 6 As shown, an image after fusion processing in an embodiment of the present application, wherein the gray rectangle corresponds to our vehicle. If our vehicle needs to change lanes to the east at the intersection ahead, the user can change lanes based on the speed and movement direction of other vehicles detected.

[0137] Based on the traffic flow statistics of vehicles in road traffic, the average traffic flow speed at traffic intersections can be estimated, and the number of vehicles passing through within a period of time can be counted. If the average traffic flow speed estimation at traffic intersections and the number of vehicles passing through within a period of time are linked with traffic lights, a smart traffic system can be created to reduce the red light time at intersections with heavy traffic flow and increase the red light time when there are more pedestrians and fewer vehicles, thereby improving the traffic efficiency at intersections.

[0138] like Figure 7 As shown in FIG, an overall flow chart of vehicle detection on a road according to an embodiment of the present application, the specific steps are as follows:

[0139] Step S701: Preprocess the radar equipment and the camera device.

[0140] Step S702: Perform target detection on the video image sequence.

[0141] Optionally, an embodiment of the present application detects a target object in a video image sequence through a deep learning network, determines the target category and position information of the target object, marks the target object with a rectangular frame and marks the corresponding category attribute.

[0142] Step S703: Track the target object in the video image.

[0143] During implementation, the target object detected in step S702 is continuously tracked, and the target ID of the same target object is kept unchanged.

[0144] Step S704: Perform target detection on the radar point cloud image sequence.

[0145] Optionally, after processing the input radar point cloud data, a deep learning network is used to detect target objects within the radar monitoring range to determine the target object's location information, movement direction, speed, radar reflection area and other data.

[0146] Step S705: Track the target object in the radar point cloud image.

[0147] During implementation, the target object detected in step S705 is continuously tracked, and the target ID of the same target object is kept unchanged.

[0148] Step S706: Time-align the video images in the video image sequence and the lightning point cloud images in the radar point cloud image sequence.

[0149] The data of the target object obtained by the aligned video image and the lightning point cloud image are data at the same time.

[0150] Step S707 : Determine a corresponding detection point pairing set according to each detection point of the target object in the video image and each detection point of the target object in the radar point cloud image.

[0151] A detection point pairing set includes position information of a detection point in a radar point cloud image and position information of a detection point in a corresponding video image.

[0152] Step S708: Determine whether the number of detection point pairing sets is greater than 4. If yes, execute step S711; if not, execute step S709.

[0153] Step S709: Determine whether the number of detection point pair sets is less than 4. If yes, execute step S716; if not, execute step S710.

[0154] Step S710: Determine the initial conversion matrix and fuse the radar point cloud image and the video image. The initial conversion matrix can be determined by the homography matrix:

[0155] [u;v;1]=h R2V [x;y;1]

[0156] [x;y;1]=h V2R [u;v;1]

[0157] Among them, [x; y; 1] represents the homogeneous coordinates corresponding to the coordinates of the detection point in the radar point cloud image in the detection point pairing set, and [u; v; 1] represents the homogeneous coordinates of the coordinates of the detection point in the video image in the same detection point pairing set. The initial first transformation matrix h is obtained. R2V and the initial second transformation matrix h V2R .

[0158] The initial transformation matrix is ​​solved by Gaussian elimination, which is an algorithm in linear algebraic programming that can be used to solve a system of linear equations.

[0159] Step S711: construct optimization objective functions of the conversion matrices respectively.

[0160] Step S712: Build a numerical optimization solver.

[0161] Step S713: perform operations by iteratively optimizing the matrix.

[0162] Step S714: Determine whether the optimization termination condition is reached. If yes, execute step S715; if no, execute step S713.

[0163] Step S715: Determine the target transformation matrix and perform fusion processing on the radar point cloud image and the video image.

[0164] Step S716: The conversion matrix cannot be determined, and an error is output.

[0165] like Figure 8 As shown, an embodiment of the present application provides an image processing device 80, the vehicle detection device 80 includes at least one processor 801 and at least one memory 802;

[0166] The memory 802 stores program code. When the program code is executed by the processor 801, the processor performs the following process:

[0167] Performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, respectively, to identify the target object in each frame of the radar point cloud image and each frame of the video image;

[0168] Determining a conversion matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, where the position in the image is a position in the video image or a position in the radar point cloud image;

[0169] According to the conversion matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence are fused.

[0170] Optionally, the processor 801 is specifically configured to:

[0171] Pairing each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the same acquisition time to determine a plurality of image sets; one image set includes one frame of radar point cloud image and one frame of video image, and the radar point cloud images and video images in one image set are acquired at the same time;

[0172] For any image set, pairing a target object in a radar point cloud image in the image set with a target object in a video image in the image set;

[0173] Based on the position information of the target objects in the radar point cloud images after pairing in each image set, the position information in the video images, and a pre-constructed optimization objective function, a transformation matrix between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device is determined.

[0174] Optionally, the target object includes multiple detection points, and the processor 801 is specifically configured to:

[0175] The point cloud position of the target object in the radar point cloud image in the image set is paired with the center point position of the lower edge of the rectangular detection box where the target object is located in the video image in the image set.

[0176] Optionally, the coordinate conversion parameter is a first conversion matrix for converting the first coordinate system into the second coordinate system, and the processor 801 is specifically configured to:

[0177] For any image set, project a target object in the radar point cloud image in the image set onto a video image in the image set according to an initial first transformation matrix to obtain a projected position of the target object; and determine an offset value between the projected position of the target object and the position of the target object in the video image in the horizontal and vertical directions;

[0178] The projection position of the target object determined by each image set and the position of the target object in the video image are offset in the horizontal direction and the vertical direction, and input into the pre-constructed optimization objective function; and determine the first transformation matrix when the function value corresponding to the optimization objective function is minimum.

[0179] Optionally, the coordinate transformation parameter is a second transformation matrix for transforming the second coordinate system into the first coordinate system, and the processor 801 is specifically configured to:

[0180] For any image set, projecting a target object in a video image in the image set onto a radar point cloud image in the image set according to an initial second transformation matrix to obtain a projected position of the target object; and determining an offset value in the horizontal coordinate direction and an offset value in the vertical coordinate direction between the projected position of the target object and the position of the target object in the radar point cloud image;

[0181] The projection position of the target object determined by each image set and the position of the target object in the video image are offset in the horizontal direction and the vertical direction, and input into the pre-constructed optimization objective function; and determine the second transformation matrix when the function value corresponding to the optimization objective function is minimum.

[0182] Optionally, the conversion matrix is ​​a first conversion matrix for converting the first coordinate system into the second coordinate system, and the processor 801 is specifically configured to:

[0183] For any radar point cloud image and video image in an image set, according to the first transformation matrix, some or all of the pixels in the radar point cloud image are projected into the video image for fusion processing.

[0184] Optionally, the conversion matrix is ​​a second conversion matrix for converting the second coordinate system into the first coordinate system, and the processor 801 is specifically configured to:

[0185] For any radar point cloud image and video image in an image set, according to the second transformation matrix, some or all of the pixels in the video image are projected into the radar point cloud image for fusion processing.

[0186] like Figure 9 As shown, an embodiment of the present application provides an image processing device 90, which can be specifically applied to the vehicle detection equipment in the above embodiment, including:

[0187] Identification module 901, configured to perform target detection on each frame of a radar point cloud image sequence acquired by a radar device and each frame of a video image sequence acquired by a camera device, and identify the target object in each frame of the radar point cloud image and each frame of the video image;

[0188] Determination module 902 determines a transformation matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in a horizontal coordinate direction and a second weight coefficient in a vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, where the position in the image is a position in the video image or a position in the radar point cloud image;

[0189] The fusion module 903 is configured to fuse the radar point cloud images of each frame in the radar point cloud image sequence and the video images of each frame in the video image sequence according to the conversion matrix.

[0190] Optionally, the determining module 902 is specifically configured to:

[0191] Pairing each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the same acquisition time to determine a plurality of image sets; one image set includes one frame of radar point cloud image and one frame of video image, and the radar point cloud images and video images in one image set are acquired at the same time;

[0192] For any image set, pairing a target object in a radar point cloud image in the image set with a target object in a video image in the image set;

[0193] Based on the position information of the target objects in the radar point cloud images after pairing in each image set, the position information in the video images, and a pre-constructed optimization objective function, a transformation matrix between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device is determined.

[0194] Optionally, the determining module 902 is specifically configured to:

[0195] The point cloud position of the target object in the radar point cloud image in the image set is paired with the center point position of the lower edge of the rectangular detection box where the target object is located in the video image in the image set.

[0196] Optionally, the coordinate conversion parameter is a first conversion matrix for converting the first coordinate system into the second coordinate system, and the determining module 902 is specifically configured to:

[0197] For any image set, project a target object in the radar point cloud image in the image set onto a video image in the image set according to an initial first transformation matrix to obtain a projected position of the target object; and determine an offset value between the projected position of the target object and the position of the target object in the video image in the horizontal and vertical directions;

[0198] The projection position of the target object determined by each image set and the position of the target object in the video image are offset in the horizontal direction and the vertical direction, and input into the pre-constructed optimization objective function; and determine the first transformation matrix when the function value corresponding to the optimization objective function is minimum.

[0199] Optionally, the coordinate conversion parameter is a second conversion matrix for converting the second coordinate system into the first coordinate system, and the determining module 902 is specifically configured to:

[0200] For any image set, projecting a target object in a video image in the image set onto a radar point cloud image in the image set according to an initial second transformation matrix to obtain a projected position of the target object; and determining an offset value in the horizontal coordinate direction and an offset value in the vertical coordinate direction between the projected position of the target object and the position of the target object in the radar point cloud image;

[0201] The projection position of the target object determined by each image set and the position of the target object in the video image are offset in the horizontal direction and the vertical direction, and input into the pre-constructed optimization objective function; and determine the second transformation matrix when the function value corresponding to the optimization objective function is minimum.

[0202] Optionally, the conversion matrix is ​​a first conversion matrix for converting the first coordinate system into the second coordinate system, and the determining module 902 is specifically configured to:

[0203] For any radar point cloud image and video image in an image set, according to the first transformation matrix, some or all of the pixels in the radar point cloud image are projected into the video image for fusion processing.

[0204] Optionally, the conversion matrix is ​​a second conversion matrix for converting the second coordinate system into the first coordinate system, and the determining module 902 is specifically configured to:

[0205] For any radar point cloud image and video image in an image set, according to the second transformation matrix, some or all of the pixels in the video image are projected into the radar point cloud image for fusion processing.

[0206] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0207] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0208] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0210] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: The method comprises: Performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, respectively, to identify the target object in each frame of the radar point cloud image and each frame of the video image; Determining a conversion matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, where the position in the image is a position in the video image or a position in the radar point cloud image; According to the conversion matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence are fused.

2. The method according to claim 1, wherein The determining, based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function, a transformation matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device includes: Pairing each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the same acquisition time to determine a plurality of image sets; one image set includes one frame of radar point cloud image and one frame of video image, and the radar point cloud images and video images in one image set are acquired at the same time; For any image set, pairing a target object in a radar point cloud image in the image set with a target object in a video image in the image set; Based on the position information of the target objects in the radar point cloud images after pairing in each image set, the position information in the video images, and a pre-constructed optimization objective function, a transformation matrix between the first coordinate system corresponding to the radar device and the second coordinate system corresponding to the camera device is determined.

3. The method according to claim 2, wherein Pairing the target object in the radar point cloud image in the image set with the target object in the video image in the image set includes: The point cloud position of the target object in the radar point cloud image in the image set is paired with the center point position of the lower edge of the rectangular detection box where the target object is located in the video image in the image set.

4. The method according to claim 2, wherein The conversion matrix is ​​a first conversion matrix for converting the first coordinate system into the second coordinate system; Determining coordinate conversion parameters between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on position information of the paired target objects in the radar point cloud images and the video images in each image set, and a pre-constructed optimization objective function, includes: For any image set, project a target object in the radar point cloud image in the image set onto a video image in the image set according to an initial first transformation matrix to obtain a projected position of the target object; and determine an offset value between the projected position of the target object and the position of the target object in the video image in the horizontal and vertical directions; The projection position of the target object determined by each image set and the position of the target object in the video image are offset in the horizontal direction and the vertical direction, and input into the pre-constructed optimization objective function; and determine the first transformation matrix when the function value corresponding to the optimization objective function is minimum.

5. The method according to claim 2, wherein The conversion matrix is ​​a second conversion matrix for converting the second coordinate system into the first coordinate system; The determining, based on the position information of the target object in the radar point cloud image in each image set and the position information of the target object in the video image, and a pre-constructed optimization objective function, coordinate conversion parameters between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device includes: For any image set, projecting a target object in a video image in the image set onto a radar point cloud image in the image set according to an initial second transformation matrix to obtain a projected position of the target object; and determining an offset value in the horizontal coordinate direction and an offset value in the vertical coordinate direction between the projected position of the target object and the position of the target object in the radar point cloud image; The projection position of the target object determined by each image set and the position of the target object in the video image are offset in the horizontal direction and the vertical direction, and input into the pre-constructed optimization objective function; and determine the second transformation matrix when the function value corresponding to the optimization objective function is minimum.

6. The method according to claim 1, wherein The conversion matrix is ​​a first conversion matrix for converting the first coordinate system into the second coordinate system; The fusing of each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the conversion matrix includes: For any radar point cloud image and video image in an image set, according to the first transformation matrix, some or all of the pixels in the radar point cloud image are projected into the video image for fusion processing.

7. The method according to claim 1, wherein The conversion matrix is ​​a second conversion matrix for converting the second coordinate system into the first coordinate system; The fusing of each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the conversion matrix includes: For any radar point cloud image and video image in an image set, according to the second transformation matrix, some or all of the pixels in the video image are projected into the radar point cloud image for fusion processing.

8. The method according to claim 6 or 7, wherein: The part of pixel points is the pixel points in the area where the target object is located in the image.

9. An image processing device, characterized in that The device includes at least one processor and at least one memory; The memory stores program code, and when the program code is executed by the processor, the processor performs the following process: Performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, respectively, to identify the target object in each frame of the radar point cloud image and each frame of the video image; Determining a conversion matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on the position information of the target object in each frame of the radar point cloud image, the position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in the horizontal coordinate direction and a second weight coefficient in the vertical coordinate direction, the first weight coefficient corresponds to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponds to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, where the position in the image is a position in the video image or a position in the radar point cloud image; According to the conversion matrix, each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence are fused.

10. An image processing device, characterized in that: The device comprises: an identification module for performing target detection on each frame of a radar point cloud image sequence collected by the radar device and each frame of a video image sequence collected by the camera device, and identifying the target object in each frame of the radar point cloud image and each frame of the video image; a determination module, configured to determine a transformation matrix between a first coordinate system corresponding to the radar device and a second coordinate system corresponding to the camera device based on position information of the target object in each frame of the radar point cloud image, position information of the target object in each frame of the video image, and a pre-constructed optimization objective function; wherein the optimization objective function includes a first weight coefficient in a horizontal coordinate direction and a second weight coefficient in a vertical coordinate direction, the first weight coefficient corresponding to an offset value in the horizontal coordinate direction between the projected position of the target object and the position in the image, and the second weight coefficient corresponding to an offset value in the vertical coordinate direction between the projected position of the target object and the position in the image, wherein the position in the image is a position in the video image or a position in the radar point cloud image; A fusion module is used to fuse each frame of radar point cloud image in the radar point cloud image sequence and each frame of video image in the video image sequence according to the conversion matrix.

Images