A method and system for stitching vehicle images taken by a fisheye camera

Abstract

The present invention discloses a method for splicing vehicle images taken by a fisheye camera, comprising: S1. obtaining a sequence of fisheye camera image frames; S2. performing correction processing on all images; S3. detecting the corrected images to obtain a vehicle detection frame and a wheel detection frame; S4. tracking and identifying the vehicle and the wheel; S5. determining the vehicle splicing state with reference to the vehicle trajectory and the coordinates of the splicing line; S6. cutting the image in the splicing process when the vehicle splicing state is in the splicing process for splicing. The present invention calculates the splicing displacement of the front and rear frames of the vehicle with reference to the identification and tracking of the vehicle axis, and compensates for the splicing displacement in combination with the external parameter calibration coefficient, so as to obtain a stable splicing image, and the splicing speed is fast, and the camera operation with a high frame rate can be supported. The present invention can obtain high-quality splicing images, support high-frame rate splicing, and can also adapt to complex scenes such as congested driving, high-speed driving, and unstable ambient light.

Description

Technical Field

[0001] The invention belongs to the technical field of image stitching, and in particular relates to a method and system for stitching vehicle images taken by a fisheye camera. Background Art

[0002] With the rapid development of the highway industry, the mileage of traffic in China is increasing, and the rapid advancement of smart highways and transportation power projects has become an obvious trend. The demand for toll evasion inspection, entrance overload control, and unmanned construction of highway toll stations is also increasing. The vehicle type recognition device has the function of intelligently identifying highway toll vehicle types and accurately classifying them. It is the core functional component of smart highways and is usually used to assist vehicle identity authentication and accurate toll collection. The device integrates a panoramic fisheye camera and a head and tail license plate recognition camera. It relies on image video acquisition, image stitching, image analysis and other technologies to realize the dynamic combination of vehicle license plates and vehicle types. Among them, image stitching technology is one of the important functional modules of the above equipment.

[0003] However, due to the lack of rich texture on the vehicle surface, some features of some textures are not very distinguishable (local feature pixels at multiple different coordinate points), which are easily affected by light, causing the underlying features or key point recognition to change easily. Therefore, the recognition effect of using traditional image feature matching methods is not stable enough and cannot meet the needs well in practical applications. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the prior art. The present invention provides a method and system for stitching vehicle images taken by a fisheye camera. The present invention calculates the stitching displacement of the front and rear frames of the vehicle with reference to the vehicle axis identification and tracking, and compensates for the stitching displacement in combination with the external parameter calibration coefficient, so as to obtain a stable stitching image, and the stitching speed is fast, and it can support high frame rate camera operation. The present invention can obtain high-quality stitching images, support high frame rate stitching, and can also adapt to complex scenes such as congested driving, high-speed driving, and unstable ambient light.

[0005] In order to achieve the expected effect, the present invention adopts the following technical solutions:

[0006] The present invention discloses a method for splicing vehicle images taken by a fisheye camera, comprising:

[0007] S1. Obtain a fisheye camera image frame sequence;

[0008] S2. Correct all images;

[0009] S3. Detect the corrected image to obtain a vehicle detection frame and a wheel detection frame;

[0010] S4. Track and identify the vehicle and wheels;

[0011] S5. Determine the vehicle splicing status by referring to the vehicle trajectory and the splicing line coordinates;

[0012] S6. Cutting the image of the vehicle in the stitching process state for stitching.

[0013] Furthermore, the S2 specifically includes:

[0014] The camera internal parameters and distortion coefficients are calculated using Zhang Zhengyou calibration method;

[0015] The mapping matrix is ​​obtained through the camera internal parameters and distortion coefficients;

[0016] Using the mapping matrix, each image is remapped to obtain a normal visual effect image after eliminating distortion.

[0017] Furthermore, the S3 specifically includes: detecting the corrected image based on the yolov5 algorithm to obtain a vehicle detection frame and a wheel detection frame.

[0018] Furthermore, the detecting the corrected image based on the yolov5 algorithm to obtain the vehicle detection frame and the wheel detection frame specifically includes:

[0019] For the corrected image, the 1 / 32, 1 / 16, and 1 / 8 output layers of the yolov5 algorithm are used to output the vehicle detection frame, and the 1 / 4 output layer of the yolov5 algorithm is used to output the wheel detection frame.

[0020] Furthermore, the S4 specifically includes:

[0021] The Kalman filter method is used to track and identify the vehicle. Based on the pairing relationship between the vehicle and the wheel, the Kalman filter method is used to track and identify the wheel.

[0022] Furthermore, the S5 specifically includes:

[0023] When it is detected that the front end of the vehicle is located at the first side of the splicing line, it is determined that the splicing state of the vehicle is not started;

[0024] When it is detected that the front of the vehicle is located at the second side of the splicing line and the rear of the vehicle is located at the first side of the splicing line, it is determined that the splicing state of the vehicle is being spliced;

[0025] When it is detected that the rear end of the vehicle is located at the second side of the splicing line, it is determined that the splicing state of the vehicle is completed;

[0026] The first side and the second side are opposite sides of the splicing line.

[0027] Furthermore, the S6 specifically includes: cutting the central areas on both sides of the central line of the image whose vehicle stitching state is in the stitching process for stitching;

[0028] The total width of the central area is: S = (A / 2 + B / 2), where A is the wheel center displacement between the current frame and the previous frame, B is the wheel center displacement between the current frame and the next frame, and the cutting area on the right and left sides of the center line is A / 2.

[0029] Furthermore, the method also includes: optimizing the splicing effect based on the dynamic scaling factor.

[0030] Furthermore, the optimization of the splicing effect based on the dynamic scaling factor specifically includes:

[0031] The dynamic scaling factor of the wheel center is calculated by combining the pixel width of the wheel image and the horizontal coordinate of the wheel center.

[0032] According to the dynamic scaling coefficient of the wheel center, the cutting and splicing displacement is calculated;

[0033] The splicing effect is optimized by cutting and splicing displacement.

[0034] The present invention also discloses a system for splicing vehicle images taken by a fisheye camera, comprising:

[0035] An acquisition module, used to acquire a fisheye camera image frame sequence;

[0036] A stitching module is used to stitch the vehicle images taken by the fisheye camera according to any of the above methods.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a method and system for splicing vehicle images taken by a fisheye camera. The present invention is based on high-precision axis detection and adopts a local attention mechanism to improve the detection accuracy of axis coordinates; the present invention adopts a combination of image augmentation and shallow feature enhancement, so that the wheel can still be stably detected under conditions of occlusion or poor illumination, thereby improving the stability of wheel detection; the present invention is based on a reference Kalman filtering method, combined with reasonable spatiotemporal logic, to obtain a stable wheel ID and wheel-vehicle binding relationship, and based on multi-frame vehicle wheel coordinates, can effectively classify various vehicles including carriers or special-shaped vehicles; the present invention is based on a vehicle splicing scheme based on vehicle separation and high-precision wheel center tracking, and can also derive a reversing classification and reversing splicing scheme. The present invention calculates the splicing displacement of the front and rear frames of the vehicle with reference to vehicle axis identification and tracking, and compensates for the splicing displacement in combination with an external parameter calibration coefficient, so as to obtain a stable splicing image, and the splicing speed is fast, which can support high frame rate camera operation. The present invention can obtain high-quality stitching images, supports high frame rate stitching, and can also adapt to complex scenes such as congested driving, high-speed driving, and unstable ambient light. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 The present invention is a schematic diagram of a fisheye camera correction method provided by an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of target detection / target tracking results of a first single-frame image provided by an embodiment of the present invention.

[0041] Figure 3 It is a schematic diagram of target detection / target tracking results of a second single-frame image provided by an embodiment of the present invention.

[0042] Figure 4 It is a schematic diagram of target detection / target tracking results of a third single-frame image provided by an embodiment of the present invention.

[0043] Figure 5 It is a schematic diagram of target detection / target tracking results of the fourth single-frame image provided by an embodiment of the present invention.

[0044] Figure 6 It is a schematic diagram of a splicing state provided by an embodiment of the present invention.

[0045] Figure 7 It is a schematic diagram of cutting and splicing provided by an embodiment of the present invention.

[0046] Figure 8 It is a schematic diagram of the horizontal axis distribution of a wheel track provided by an embodiment of the present invention.

[0047] Fig. 9 The present invention provides a flow chart of a method for stitching vehicle images taken by a fisheye camera according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figures 1 to 9 The present invention discloses a method for stitching vehicle images taken by a fisheye camera, comprising:

[0050] S1. Obtain a fisheye camera image frame sequence;

[0051] S2. Correct all images;

[0052] Preferably, S2 specifically includes:

[0053] Through Zhang Zhengyou calibration method, the camera internal parameters and distortion coefficients are calculated; a chessboard calibration plate is required here;

[0054] A mapping matrix is ​​obtained through camera intrinsic parameters and distortion coefficients; in the mapping matrix, each pixel coordinate in the new image corresponds one-to-one to the pixel coordinate in the old image;

[0055] Using the mapping matrix, each image is remapped to obtain a normal visual effect image after eliminating distortion, such as Figure 1 shown.

[0056] S3. Detect the corrected image to obtain a vehicle detection frame and a wheel detection frame;

[0057] Preferably, the S3 specifically includes: detecting the corrected image based on the yolov5 algorithm to obtain a vehicle detection frame and a wheel detection frame.

[0058] Furthermore, the detecting the corrected image based on the yolov5 algorithm to obtain the vehicle detection frame and the wheel detection frame specifically includes:

[0059] For the corrected image, the 1 / 32, 1 / 16, and 1 / 8 output layers of the yolov5 algorithm are used to output the vehicle detection frame, and the 1 / 4 output layer of the yolov5 algorithm is used to output the wheel detection frame. This step is to improve the detection accuracy of the wheel center.

[0060] The present invention is based on high-precision axis detection and adopts a local attention mechanism to improve the detection accuracy of the axis coordinates; the present invention adopts a combination of image enhancement and shallow feature enhancement to enable the wheel to be stably detected under conditions of occlusion or poor lighting, thereby improving the stability of wheel detection.

[0061] S4. Track and identify the vehicle and wheels;

[0062] In a preferred embodiment, the S4 specifically includes:

[0063] The Kalman filter method is used to track and identify the vehicle. Based on the pairing relationship between the vehicle and the wheel, the Kalman filter method is used to track and identify the wheel. During the tracking and identification process, the next frame position prediction operation is generally performed; for the prediction of the next frame coordinates of the vehicle, the conventional Kalman filter method is used; for the prediction of the next frame of the wheel, the Kalman filter corrected based on the movement of the reference vehicle detection frame is used.

[0064] Specifically, the process of target tracking based on the target frame is introduced below:

[0065] First frame: All targets in the image are new targets; a tracking chain is established for each target [with a unique tracking ID];

[0066] From the second frame to the nth frame, the target in the image will calculate the similarity with the target in the tracking chain (predict the target represented by this tracking chain to obtain its coordinates in the current frame, and then detect the predicted coordinates); calculate the pixel degree of the target frame of the current frame and the predicted target of each tracking chain one by one; if there is a tracking chain with high enough similarity, join this tracking chain; if the similarity is not enough, establish a new tracking chain;

[0067] In addition, the present invention uses an improved IoU. Intersection over Union (IoU) is a standard for measuring the accuracy of detecting corresponding objects in a specific data set. IoU is a simple measurement standard, and any task that obtains a prediction range (bounding boxes) in the output can be measured using IoU. The usual IoU calculates the intersection area. The present invention is simplified and only calculates the intersection of line segments. The reason for the simplification is: if a vehicle is photographed from the side in a scene, the vehicle in the picture only moves laterally (moves in the x-direction); the simplification is beneficial to reducing the amount of calculation and eliminating the additional error introduced by the target box jumping along the y-axis.

[0068] For example, [x1, x2] represents the left and right coordinates of the current target detection box, and [xp1, xp2] represents the left and right coordinates of the predicted box.

[0069] Crossover rate = (min(x2,xp2)-max(x1,xp1)) / (max(x2,xp2)-min(x1,xp1))

[0070] It is worth noting that the present invention predicts the coordinates of the current frame based on the coordinate values ​​of the previous frame in a target tracking chain. The specific tracking logic is: first, track the vehicle; then refer to the vehicle tracking information to track the wheels. Then simplify the coordinate prediction, and only predict the horizontal axis displacement of the vehicle; use the Kalman filter method to predict the vehicle detection frame predicted by the current frame coordinates; while using the Kalman filter method to predict the movement, introduce the movement of the vehicle detection frame for correction, and then obtain the current frame coordinates to predict the wheel detection frame. The specific operations are as follows:

[0071] First, the wheel and the vehicle are paired. The pairing method is: if the center point of the wheel detection frame is between the left end and the right end of the vehicle detection frame, then the wheel belongs to the vehicle.

[0072] Then, since the vehicle has been tracked, the displacement of the vehicle in the current frame and the previous frame can be calculated, and the vehicle displacement can be detected and obtained.

[0073] Finally, the movement of the wheel is predicted to obtain the predicted position of the wheel. Preferably, wheel displacement = Kalman filter * 0.2 + vehicle displacement * 0.8.

[0074] The present invention is based on a Kalman filter method with reference, combined with reasonable spatiotemporal logic, to obtain stable wheel IDs and wheel-vehicle binding relationships, and can effectively classify various vehicles including carriers or special-shaped vehicles based on multi-frame vehicle wheel coordinates.

[0075] S5. Determine the vehicle splicing status by referring to the vehicle trajectory and the splicing line coordinates;

[0076] Specifically, there are three states of vehicle splicing: splicing not started, splicing in progress, and splicing completed.

[0077] In one embodiment, the S5 specifically includes:

[0078] When it is detected that the front end of the vehicle is located at the first side of the splicing line, it is determined that the splicing state of the vehicle is not started;

[0079] When it is detected that the front of the vehicle is located at the second side of the splicing line and the rear of the vehicle is located at the first side of the splicing line, it is determined that the splicing state of the vehicle is being spliced;

[0080] When it is detected that the rear end of the vehicle is located at the second side of the splicing line, it is determined that the splicing state of the vehicle is completed;

[0081] The first side (eg, the left side) and the second side (eg, the right side) are opposite sides of the splicing line.

[0082] S6. Cutting the image of the vehicle in the stitching process state for stitching.

[0083] In a preferred embodiment, the S6 specifically includes: cutting the central areas on both sides of the central line of the image whose vehicle stitching state is in the stitching process for stitching;

[0084] The total width of the central area is: S = (A / 2 + B / 2), where A is the wheel center displacement between the current frame and the previous frame, B is the wheel center displacement between the current frame and the next frame, and the cutting area on the right and left sides of the center line is A / 2.

[0085] Specifically, the central area of ​​the fisheye camera's correction image has lower distortion and a more correct image. Therefore, the central area of ​​each frame is cut for stitching. The distortion near the center line is the smallest, so the areas on both sides of the center line are taken. The cut range of the start and end frames of the stitching needs to be determined while considering the coordinates of the vehicle detection frame.

[0086] Furthermore, the method also includes: optimizing the splicing effect based on the dynamic scaling factor.

[0087] Specifically, the reason for using dynamic scaling factors to compensate for stitching is that due to the large field of view of the fisheye camera, the corrected image will still have a certain scale of deformation. The actual manifestation is that the pixel size of the same wheel imaged close to the center line is larger than the pixel size of the imaged far from the center line. During the stitching process, the stitching sub-blocks are all cut from the center line of the picture. However, the wheel position in most frames is not at the center, so there will be a certain deviation in the scale of the cutting range calculated based on these frames. The size of the deviation is related to the distance of the wheel position from the center line.

[0088] In a preferred embodiment, the optimization of the splicing effect based on the dynamic scaling factor specifically includes:

[0089] The dynamic scaling factor of the wheel center is calculated by combining the pixel width of the wheel image and the horizontal coordinate of the wheel center.

[0090] According to the dynamic scaling coefficient of the wheel center, the cutting and splicing displacement is calculated;

[0091] The splicing effect is optimized by cutting and splicing displacement.

[0092] Specifically, before the device is run, the dynamic scaling factor is initialized and assigned based on external parameter calibration to establish the dynamic scaling factor at different positions on the horizontal axis. Because the corrected image still has a little distortion, and the camera plane and the side of the vehicle are not completely parallel, the pixel size corresponding to the same vehicle running to different positions will be different. The steps for calculating the dynamic scaling factor are as follows:

[0093] Mathematical formula for establishing dynamic scaling factor: k = 1-s1*d-s2*d*d-s3*d*d*d; d = abs(mx);

[0094] Note: x represents the horizontal axis coordinate of the image pixel, m represents the midpoint position of the horizontal axis of the image pixel (m is a known number); d represents the distance between m and x, that is, the distance from the left side of a certain x to the center position; k represents the dynamic scaling factor; s1, s2, s3 represent model coefficients, which need to be solved. Once the solution is completed, the mathematical modeling of the dynamic scaling coefficient is completed. Preferably, after extensive sampling, the values ​​of s1, s2, and s3 are obtained by the least squares method.

[0095] It is worth noting that after observation, the image of the same object in the corrected image shows a trend of being larger in the middle and smaller on both sides, so k = 1, which means that the middle gradually becomes smaller towards both sides. Since the reduction is a nonlinear relationship, the first, second, and third powers can be used as the reduction amount.

[0096] For example, using wheels as reference objects, when a car passes through the picture, the wheels will leave a set of driving tracks, and are distributed along the horizontal axis of the image, such as Figure 8 shown.

[0097] Based on the target detection results described above, the pixel width of the wheel at each position can be calculated;

[0098] The k value of the wheel at each position = the current position width / the center wheel width;

[0099] The x value of the wheel at each position = the horizontal coordinate of the wheel center;

[0100] Therefore, a sequence of [wheel width-horizontal axis coordinate values] (or called "kx" values) can be obtained.

[0101] In order to make the sampled data more balanced, 200 wheels are sampled to obtain 200 sequences of "kx" values. Each sequence includes the "kx" values ​​of the same wheel at different positions, and usually each sequence is sampled from 10 to 30. Therefore, there are a total of 2000 to 6000 "kx" values. Use 2000 to 6000 "kx" values ​​to substitute into the formula k = 1-s1*d-s2*d*d-s3*d*d*d, and based on the least squares method, s1, s2, and s3 can be solved.

[0102] Vehicle stitching is done by cutting the middle part of each frame and stitching the middle parts of multiple frames. The cutting displacement of the middle part is measured based on the wheel center displacement. Since the wheel center coordinates and wheel center displacement are not always in the middle of the picture, the dynamic scaling factor k of the wheel center position is calculated. Since the dynamic scaling factor of the wheel center is 1, the wheel cutting displacement = wheel center displacement / k.

[0103] In addition, during the operation, each wheel will generate a tracking sequence, which contains the scale of the wheel at each horizontal coordinate. Therefore, during the operation, the dynamic scaling factor will be dynamically adjusted based on the scale transformation law of the wheel sequence.

[0104] The above embodiment is introduced by taking the vehicle traveling from left to right as an example. The principle of the vehicle traveling from right to left is similar and will not be described in detail.

[0105] In a preferred embodiment, the present invention also includes vehicle splicing in a reversing scenario:

[0106] Determine the reversing state based on vehicle and wheel tracking results;

[0107] After detecting the reversing state, record the wheel ID and coordinates at the time of reversing;

[0108] If the vehicle does not drive out of the splicing line when reversing, the splicing task will be retained; after the vehicle is detected to move forward again, it will enter the pre-splicing state and compare it with the previously recorded reversing mark;

[0109] If the vehicle passes the coordinates of the recorded time again, continue splicing;

[0110] If the vehicle reverses out of the splicing line, the current splicing task will be deleted, and the splicing task will be re-established when the vehicle moves forward again.

[0111] The present invention calculates the splicing displacement of the front and rear frames of the vehicle by referring to the vehicle axis identification and tracking, and compensates for the splicing displacement by combining the external parameter calibration coefficient, so as to obtain a stable splicing image with a fast splicing speed, and can support the operation of a high frame rate camera. The present invention can obtain high-quality splicing images, support high frame rate splicing, and can also adapt to complex scenes such as congested driving, high-speed driving, and unstable ambient light.

[0112] The present invention also discloses a system for splicing vehicle images taken by a fisheye camera, comprising:

[0113] An acquisition module, used to acquire a fisheye camera image frame sequence;

[0114] A stitching module is used to stitch the vehicle images taken by the fisheye camera according to any of the above methods.

[0115] The system embodiments can be implemented one-to-one with the aforementioned method embodiments, and will not be described in detail here.

[0116] Based on the same inventive concept, the present invention also discloses an electronic device, which may include: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute a method for splicing vehicle images taken by a fisheye camera, including:

[0117] S1. Obtain a fisheye camera image frame sequence;

[0118] S2. Correct all images;

[0119] S3. Detect the corrected image to obtain a vehicle detection frame and a wheel detection frame;

[0120] S4. Track and identify the vehicle and wheels;

[0121] S5. Determine the vehicle splicing status by referring to the vehicle trajectory and the splicing line coordinates;

[0122] S6. Cutting the image of the vehicle in the stitching process state for stitching.

[0123] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] On the other hand, an embodiment of the present invention further provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer can execute a method for stitching vehicle images taken by a fisheye camera provided in the above-mentioned method embodiments, including:

[0125] S1. Obtain a fisheye camera image frame sequence;

[0126] S2. Correct all images;

[0127] S3. Detect the corrected image to obtain a vehicle detection frame and a wheel detection frame;

[0128] S4. Track and identify the vehicle and wheels;

[0129] S5. Determine the vehicle splicing status by referring to the vehicle trajectory and the splicing line coordinates;

[0130] S6. Cutting the image of the vehicle in the stitching process state for stitching.

[0131] In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for stitching a vehicle image taken by a fisheye camera provided in the above embodiments is implemented, including:

[0132] S1. Obtain a fisheye camera image frame sequence;

[0133] S2. Correct all images;

[0134] S3. Detect the corrected image to obtain a vehicle detection frame and a wheel detection frame;

[0135] S4. Track and identify the vehicle and wheels;

[0136] S5. Determine the vehicle splicing status by referring to the vehicle trajectory and the splicing line coordinates;

[0137] S6. Cutting the image of the vehicle in the stitching process state for stitching.

[0138] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for stitching vehicle images taken by a fisheye camera, characterized in that: include: S1. Obtain a fisheye camera image frame sequence; S2. Correct all images; S3. Detect the corrected image to obtain a vehicle detection frame and a wheel detection frame; S4. Track and identify the vehicle and wheels; S5. Determine the vehicle splicing status by referring to the vehicle trajectory and the splicing line coordinates; S6. Cutting the image of the vehicle in the stitching process for stitching; S7. Optimize the stitching effect based on the dynamic scaling factor, including: S71. Calculate the dynamic scaling factor of the wheel center by combining the pixel width of the wheel image and the horizontal coordinate of the wheel center, specifically including: Sample several wheels respectively, and obtain corresponding "kx" value sequences respectively, each sequence includes "kx" values ​​of the same wheel at different positions; Substitute these "kx" value sequences into the dynamic scaling coefficient model formula, obtain the corresponding model coefficients based on the least square method, and further obtain the dynamic scaling coefficient of the wheel center; The dynamic scaling coefficient model formula includes: k = 1 - s1*d - s2*d*d - s3*d*d*d; d = abs(mx); Among them, x represents the horizontal axis coordinate of the image pixel, that is, the horizontal coordinate of the wheel center at each position; m represents the midpoint position of the horizontal coordinate of the image pixel, and m is a known number; d represents the distance between m and x; k represents the dynamic scaling factor, and the k value of the wheel at each position = the current position width / center wheel width; s1, s2, and s3 represent model coefficients; S72. According to the dynamic scaling coefficient of the wheel center, the cutting and splicing displacement is calculated; the cutting and splicing displacement of the wheel = the wheel center displacement / k; S73. Optimize the splicing effect by cutting and splicing displacement.

2. The method for stitching vehicle images taken by a fisheye camera as claimed in claim 1, characterized in that: The S2 specifically includes: The camera internal parameters and distortion coefficients are calculated using Zhang Zhengyou calibration method; The mapping matrix is ​​obtained through the camera internal parameters and distortion coefficients; Using the mapping matrix, each image is remapped to obtain a normal visual effect image after eliminating distortion.

3. The method for stitching vehicle images taken by a fisheye camera as claimed in claim 1, characterized in that: The S3 specifically includes: detecting the corrected image based on the yolov5 algorithm to obtain a vehicle detection frame and a wheel detection frame.

4. The method for stitching vehicle images taken by a fisheye camera as claimed in claim 1, characterized in that: The method of detecting the corrected image based on the yolov5 algorithm to obtain the vehicle detection frame and the wheel detection frame specifically includes: For the corrected image, the 1 / 32, 1 / 16, and 1 / 8 output layers of the yolov5 algorithm are used to output the vehicle detection frame, and the 1 / 4 output layer of the yolov5 algorithm is used to output the wheel detection frame.

5. The method for stitching vehicle images taken by a fisheye camera as claimed in claim 1, It is characterized in that The S4 specifically includes: The Kalman filter method is used to track and identify the vehicle. Based on the pairing relationship between the vehicle and the wheel, the Kalman filter method is used to track and identify the wheel.

6. The method for stitching vehicle images taken by a fisheye camera as claimed in claim 1, It is characterized in that The S5 specifically includes: When it is detected that the front end of the vehicle is located at the first side of the splicing line, it is determined that the splicing state of the vehicle is not started; When it is detected that the front of the vehicle is located at the second side of the splicing line and the rear of the vehicle is located at the first side of the splicing line, it is determined that the splicing state of the vehicle is being spliced; When it is detected that the rear end of the vehicle is located at the second side of the splicing line, it is determined that the splicing state of the vehicle is completed; The first side and the second side are opposite sides of the splicing line.

7. The method for stitching vehicle images taken by a fisheye camera as claimed in claim 1, characterized in that: The S6 specifically includes: cutting the central areas on both sides of the central line of the image in the vehicle stitching state in the stitching process for stitching; The total width of the central area is: S = (A / 2 + B / 2), where A is the wheel center displacement between the current frame and the previous frame, B is the wheel center displacement between the current frame and the next frame, and the cutting area on the right and left sides of the center line is A / 2.

8. A system for stitching vehicle images taken by a fisheye camera, characterized in that: include: An acquisition module, used to acquire a fisheye camera image frame sequence; A stitching module, used for stitching vehicle images taken by a fisheye camera according to any method described in claims 1-7.

Images