All-round splicing method and system adaptive to cable winding and unwinding vehicle

Abstract

The invention discloses an all-round splicing method and system adaptive to a cable winding and unwinding vehicle, and relates to the technical field of monitoring equipment. The method comprises the steps that fisheye cameras and binocular cameras are fixed at a plurality of preset positions of a cable winding and unwinding vehicle, and image and depth information at the same moment is collected; performing weighted fusion on the fisheye camera image and the depth map to obtain a fused image; feature points of the residual image and the fused image are extracted and matched, and a coordinate conversion relation is determined; splicing the images into a top view image, adjusting colors and fusing overlapped areas to generate a two-dimensional all-round view image; and inputting the top view image into the three-dimensional sphere model, outputting a three-dimensional look-around image, and displaying the three-dimensional look-around image in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monitoring equipment, in particular to a surround view splicing method and system suitable for a cable winding and unwinding vehicle. BACKGROUND

[0002] Engineering vehicles are the main force of open-pit mine engineering. Their appearance has multiplied the progress of open-pit mine engineering and greatly reduced the labor cost. However, the working environment of engineering vehicles is complex, and the vehicle body is large, which has a large blind area and it is difficult to take into account all around. Due to the large size and weight of the engineering vehicle, once it collides with a car, serious consequences often occur. The working environment of engineering vehicles in open-pit mines is harsh, and the lack of driver operating level and vehicle performance can cause engineering vehicles to collide and overturn, etc., threatening the safety of people's lives and property. Therefore, how to ensure the safety of the engineering site is a problem that engineering vehicle manufacturers and researchers need to solve urgently.

[0003] In the prior art, due to the rapid development of machine vision technology and the continuous improvement of camera hardware performance, visual auxiliary safety protection systems are increasingly valued, and many achievements have been applied to cars. However, in the application of engineering vehicles, the existing panoramic surround view image systems basically have splicing seams, and it is difficult to achieve seamless splicing, which affects the visual effect.

[0004] Therefore, there is an urgent need for an image surround view splicing method that can be applied to engineering vehicles to eliminate the splicing seams of the spliced images of engineering vehicles. SUMMARY

[0005] Therefore, it is necessary to provide a surround view splicing method and system suitable for a cable winding and unwinding vehicle in view of the above technical problems.

[0006] The present application adopts the following technical solutions: A plurality of fisheye cameras and binocular cameras are fixed at a plurality of predetermined positions of the cable winding and unwinding vehicle; A plurality of cable winding and unwinding vehicle images at the same time are collected by the plurality of fisheye cameras and binocular cameras; and a scene depth map of the cable winding and unwinding vehicle at the same time is collected by the binocular cameras; The images of the cable reel collected by the two fisheye cameras at the same position as the preset position of the binocular camera and the two scene depth maps are weighted and fused to obtain a cable reel fusion image; feature points of the remaining multiple cable reel images and the cable reel fusion image are extracted and matched to obtain a coordinate position conversion relationship between the cable reel images and the cable reel fusion image; based on the coordinate position conversion relationship, the multiple cable reel images are moved to the same coordinate system and spliced to obtain a cable reel overhead splicing image; the brightness histogram information of each cable reel image is extracted to adjust the color of the cable reel overhead splicing image; the overlapping area in the cable reel overhead splicing image after the color is adjusted is fused to obtain a two-dimensional surround view splicing image of the cable reel; and the cable reel overhead splicing image after the color is adjusted is input into a preset three-dimensional spherical model to output a three-dimensional surround view splicing image of the cable reel. The two-dimensional surround view splicing image and the three-dimensional surround view splicing image at each moment are displayed in real time.

[0007] Preferably, the optical axis direction of the binocular camera is arranged at an acute angle with the ground; and the viewing angle of the fisheye camera is 180 degrees.

[0008] Preferably, the multiple fisheye cameras and the binocular camera are fixed at multiple preset positions of the vehicle, specifically including: The six fisheye cameras are respectively fixed at the center of the vehicle head, the center of the vehicle tail, the front 25% quantile point of the left side of the vehicle body, the rear 25% quantile point of the left side of the vehicle body, the front 25% quantile point of the right side of the vehicle body, and the rear 25% quantile point of the right side of the vehicle body. The two binocular cameras are respectively fixed at the center of the vehicle head and the center of the vehicle tail.

[0009] Preferably, the images of the cable reel collected by the two fisheye cameras at the same position as the preset position of the binocular camera and the scene depth maps are weighted and fused to obtain a cable reel fusion image, specifically including: Based on the position conversion relationship between the fisheye camera and the binocular camera, the fisheye camera coordinate system in which the cable reel image is located and the binocular camera coordinate system in which the scene depth map is located are converted to the same image coordinate system; The pixel distances based on the two scene depth maps and the pixel distances between the two cable reel images collected by the two fisheye cameras at the same position as the preset position of the binocular camera are weighted and fused to obtain a weighted fused pixel distance, and the formula is: ; In the formula, D final is the weighted fused pixel distance, D stereo is the pixel distance of the two scene depth maps,D momo a pixel distance between images of the cable reel collected by the fisheye camera at the same position as the preset position of the binocular camera, w 1 is a pixel distance weight between two scene depth maps, w 2 is a pixel distance weight between images of the cable reel collected by the fisheye camera at the same position as the preset position of the binocular camera; based on the weighted fused pixel distance, the images of the cable reel collected by the fisheye camera at the same position as the preset position of the binocular camera and the scene depth map are fused pixel by pixel to obtain a cable reel fusion image.

[0010] Preferably, feature points of the remaining plurality of cable reel images and the cable reel fusion image are extracted and matched to obtain a coordinate position conversion relationship between each cable reel image and the cable reel fusion image, specifically including: convolve the two-dimensional Gaussian kernel function with the cable reel image to construct a cable reel scale space, and detect candidate extreme points through a Difference of Gaussian (DOG) space; based on a Taylor expansion of the DOG function, sub-pixel level positioning is performed on the candidate extreme points, and edge responses are removed through a Hessian matrix to obtain feature points of the plurality of cable reel images; the directions and gradients of the pixel points in the designated field of the feature points of the cable reel image are counted; and based on the directions and gradients of the pixel points, a feature point descriptor of the cable reel image is generated; match the feature point descriptors in each adjacent two cable reel images to obtain a matching point pair; calculate a rotation matrix and a translation vector between each adjacent two cable reel images according to the matching point pair; the rotation matrix and the translation vector are the coordinate position conversion relationship of the plurality of cable reel images.

[0011] Preferably, based on the directions and gradients of the pixel points, a feature point descriptor of the cable reel image is generated, specifically including: select a neighborhood window of a designated size centered on the feature point, and divide the neighborhood window into a plurality of sub-regions; calculate a gradient direction histogram in each sub-region; each gradient direction histogram contains a plurality of direction intervals, forming a multi-dimensional feature vector; concatenate the multi-dimensional feature vectors of each sub-region and normalize to obtain a feature point descriptor of each cable reel image.

[0012] A surround view splicing method suitable for a cable reel, characterized in that it comprises: An image acquisition module is configured to fix a plurality of fisheye cameras and binocular cameras at a plurality of preset positions of a cable winding and unwinding vehicle respectively, acquire a plurality of images of the cable winding and unwinding vehicle at the same time through the plurality of fisheye cameras and binocular cameras, and acquire a scene depth map of the cable winding and unwinding vehicle at the same time through the binocular cameras; An image stitching module is configured to fuse images of the cable winding and unwinding vehicle acquired by two fisheye cameras at the same preset position as the binocular cameras and two scene depth maps to obtain a fused image of the cable winding and unwinding vehicle, extract feature points of the remaining plurality of images of the cable winding and unwinding vehicle and the fused image of the cable winding and unwinding vehicle, and perform feature point matching to obtain a coordinate position conversion relationship between the images of the cable winding and unwinding vehicle and the fused image of the cable winding and unwinding vehicle, move the plurality of images of the cable winding and unwinding vehicle to the same coordinate system based on the coordinate position conversion relationship, and stitch the images to obtain an overhead stitched image of the cable winding and unwinding vehicle, extract brightness histogram information of each image of the cable winding and unwinding vehicle to adjust the color of the overhead stitched image of the cable winding and unwinding vehicle, fuse overlapping areas in the overhead stitched image of the cable winding and unwinding vehicle after the color is adjusted to obtain a two-dimensional surround view stitched image of the cable winding and unwinding vehicle, and input the overhead stitched image of the cable winding and unwinding vehicle after the color is adjusted to a preset three-dimensional spherical model to output a three-dimensional surround view stitched image of the cable winding and unwinding vehicle; A display module is configured to display the two-dimensional surround view stitched image and the three-dimensional surround view stitched image in real time.

[0013] Preferably, the system further comprises an obstacle detection module and a trajectory smoothing module. The obstacle detection module is configured to identify obstacle pixels based on the three-dimensional surround view stitched image and convert image coordinates of the obstacle pixels into three-dimensional coordinates. The trajectory smoothing module is configured to perform Kalman filtering or particle filtering on the three-dimensional coordinates of the obstacle in consecutive frames to obtain a smoothed obstacle motion trajectory.

[0014] Preferably, the display module is further configured to superimpose and display the obstacle motion trajectory and real-time distance information.

[0015] The above-mentioned at least one technical solution adopted by the present application can achieve the following beneficial effects: In the surround view splicing method provided by the application, through the combination layout of six fisheye cameras and front and rear binocular cameras, the overhead splicing is integrated with depth information-texture information after one-time calibration, which eliminates the splicing seam and distortion of the traditional surround view system, and can expand the field of view with high-precision depth in the near field and high-resolution texture in the far field. The first surround view image obtained by fusion realizes seamless splicing of the overhead view around the vehicle body. By mapping and fusing the binocular depth information, the second surround view image highlights the depth information in the near field and the texture information in the far field, further eliminates the splicing seam caused by brightness and color difference, significantly improves the visual effect, and reduces the misjudgment. BRIEF DESCRIPTION OF DRAWINGS

[0016] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0017] Figure 1 The method flowchart of the surround view splicing method provided by the application is shown in the figure; Figure 2 The image coordinate system diagram of the panoramic surround view provided by the surround view splicing method of the application is shown in the figure; Figure 3 The feature point matching algorithm step diagram of the surround view splicing method provided by the application is shown in the figure; Figure 4 The main direction of the key point of the surround view splicing method provided by the application is shown in the figure; Figure 5 The algorithm flowchart of the surround view image formation of the surround view splicing method provided by the application is shown in the figure; Figure 6 The coordinate system conversion diagram of the surround view splicing method provided by the application is shown in the figure; Figure 7 The fisheye camera monitoring area diagram of the surround view splicing method provided by the application is shown in the figure; Figure 8 The splicing effect diagram of the surround view splicing method provided by the application is shown in the figure; Figure 9 The system diagram of the surround view splicing system provided by the application is shown in the figure. DETAILED DESCRIPTION

[0018] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the specification, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0019] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0020] Figure 1 The structure diagram of the surround view splicing method suitable for the cable reel vehicle in the present application specifically comprises The plurality of fisheye cameras and binocular cameras are fixed at a plurality of preset positions of the cable reel vehicle respectively; The plurality of cable reel vehicle images at the same time are collected by the plurality of fisheye cameras and binocular cameras, and the scene depth map of the cable reel vehicle at the same time is collected by the binocular camera; The cable reel vehicle images collected by the two fisheye cameras at the same preset positions as the binocular camera and the two scene depth maps are fused by weighting to obtain a cable reel vehicle fusion image; the feature points of the remaining plurality of cable reel vehicle images and the cable reel vehicle fusion image are extracted and matched to obtain the coordinate position conversion relationship between the cable reel vehicle images and the cable reel vehicle fusion image; based on the coordinate position conversion relationship, the plurality of cable reel vehicle images are moved to the same coordinate system and spliced to obtain a cable reel vehicle overhead splicing image; the brightness histogram information of each cable reel vehicle image is extracted to adjust the color of the cable reel vehicle overhead splicing image; the overlapping area in the cable reel vehicle overhead splicing image after adjusting the color is fused to obtain a two-dimensional surround view splicing image of the cable reel vehicle; and the cable reel vehicle overhead splicing image after adjusting the color is input into a preset three-dimensional spherical model to output a three-dimensional surround view splicing image of the cable reel vehicle; The two-dimensional surround view splicing image and the three-dimensional surround view splicing image at each time are displayed in real time.

[0021] Optionally, the optical axis direction of the binocular camera is arranged at an acute angle with the ground; and the viewing angle of the fisheye camera is 180 degrees.

[0022] Optionally, the plurality of fisheye cameras and binocular cameras are fixed at a plurality of preset positions of the vehicle respectively, specifically comprising: The six fisheye cameras are fixed at the center of the vehicle head, the center of the vehicle tail, the front 25% quantile point of the left side of the vehicle body, the rear 25% quantile point of the left side of the vehicle body, the front 25% quantile point of the right side of the vehicle body, and the rear 25% quantile point of the right side of the vehicle body of the cable reel vehicle respectively; Two binocular cameras are fixed at the center of the front and the center of the rear of the vehicle.

[0023] Optionally, the cable reel images collected by the two fisheye cameras with the same preset position as the binocular cameras and the scene depth maps are weightedly fused to obtain a cable reel fusion image, specifically including: Based on the position conversion relationship between the fisheye cameras and the binocular cameras, the fisheye camera coordinate system in which the cable reel images are located and the binocular camera coordinate system in which the scene depth maps are located are converted to the same image coordinate system; The pixel distances between the two scene depth maps and the pixel distances between the two cable reel images collected by the fisheye cameras with the same preset position as the binocular cameras are weightedly fused to obtain a weightedly fused pixel distance, and the formula is: ; In the formula, D final is the weightedly fused pixel distance, D stereo is the pixel distance of the two scene depth maps, D momo is the pixel distance between the two cable reel images collected by the fisheye cameras with the same preset position as the binocular cameras, w 1 is the pixel distance weight between the two scene depth maps, w 2 is the pixel distance weight between the two cable reel images collected by the fisheye cameras with the same preset position as the binocular cameras; Based on the weightedly fused pixel distance, the cable reel images collected by the fisheye cameras with the same preset position as the binocular cameras and the scene depth maps are pixel by pixel fused to obtain a cable reel fusion image.

[0024] Optionally, feature points of the remaining multiple cable reel images and the cable reel fusion image are extracted and matched to obtain a coordinate position conversion relationship between the cable reel images and the cable reel fusion image, specifically including: A two-dimensional Gaussian kernel function is convolved with the cable reel images to construct a cable reel scale space, and a candidate extreme point is detected through a difference of Gaussian (DOG) space; the candidate extreme point is located at a sub-pixel level based on a Taylor expansion of the DOG function, and an edge response is removed through a Hessian matrix to obtain feature points of the multiple cable reel images; directions and gradients of pixel points in a designated field of the feature points of the cable reel images are counted; and a feature point descriptor of the cable reel images is generated based on the directions and gradients of the pixel points; The feature point descriptors in each adjacent two cable reel images are matched to obtain a matching point pair; According to the matching point pairs, a rotation matrix and a translation vector between each adjacent two cable reel images are calculated; the rotation matrix and the translation vector are coordinate position conversion relationships of the plurality of cable reel images.

[0025] Optionally, based on the direction and gradient of the pixel point, a feature point descriptor of the cable reel image is generated, specifically including: A neighborhood window of a specified size is selected as the center of the feature point, and the neighborhood window is divided into a plurality of sub-regions; A gradient direction histogram in each sub-region is calculated; each gradient direction histogram contains a plurality of direction intervals, forming a multi-dimensional feature vector; After splicing the multi-dimensional feature vectors of each sub-region, normalization is performed to obtain the feature point descriptor of each cable reel image.

[0026] Specifically, referring to Figure 2 The image coordinate system is a schematic diagram of a panoramic surround view image, with the upper left corner of the image as the origin and pixels as the unit. C11 and C44 use Bosch's binocular camera, with a distance of 12 cm between the two cameras, a pixel number of 1080960, a horizontal viewing angle of 45°, a direct viewing angle of 25°, and a maximum detection distance of 50 m. It can not only be used for automatic braking systems, but also for lane departure warning systems and traffic sign recognition systems, etc. C1-C6 are six fisheye cameras, and the images collected by them are rectangular after fisheye correction and top-down transformation. The pixel size is set as MxN, and the effective area of the corresponding image is actually a trapezoid. The trapezoid in the schematic diagram is the effective area of the figure. The actual length of the cable reel is L, and the width is W. C1 and C4 (i.e. front and rear cameras) are installed at W / 2, C2, C3, C5, and C6 are installed at L / 4 on both sides of the vehicle. C11 and C1 are installed at the same position, C11 is 50 cm above C1, C44 and C4 are installed at the same position, and C44 is 50 cm above C4.

[0027] Based on GPU acceleration technology, the to-be-spliced images are spliced respectively to generate a two-dimensional projection overhead image and a three-dimensional surround spliced image based on a preset model, and the surround splicing of the cable reel surrounding image is completed.

[0028] Image matching obtains the coordinate transformation relationship between the images to be spliced, and moves the images to the same image coordinate system. The next step is to fuse the images in the overlapping area of the two images to achieve a smooth transition visual effect. In the panoramic surround view image system, the images around are captured by different cameras, and the differences in camera technology will cause differences in brightness, color, and other aspects of the images. Therefore, a suitable fusion algorithm should be selected to make the final display image more natural. The panoramic surround view image system has high requirements for real-time performance. Therefore, on the basis of ensuring the fusion effect, the complexity of the algorithm should also be considered, and a fusion algorithm with too high complexity should not be used.

[0029] Scale-invariant feature transform (SIFT) is a visual algorithm that describes the local features of an image by finding extreme points in a scale space and extracting their position, scale, and rotation invariants. The algorithm first extracts feature points from the image to be matched, then describes the feature points, and finally compares the two feature points to find matching pairs and establish a correspondence. The specific steps are shown in Figure 3 . The algorithm specifically includes:

[0030] Generating a scale space and detecting extreme points: In order to detect feature points on images of different sizes, the SIFT algorithm uses a Gaussian kernel function to construct a scale space, thereby retaining more information and features of the original image. The two-dimensional Gaussian kernel function is convolved with the input image I(x, y) to obtain the scale space of the image, where (x, y) represents the pixel coordinates of the image, and σ represents the smoothness of the image. The DOG scale space can be obtained by differentiating the Gaussian function: .

[0031] Precise positioning of feature points: After the extreme point retrieval is completed, further screening of candidate feature points and elimination of edge responses are needed. To increase the stability of the feature points, the DOG function is fitted. The Taylor expansion of the DOG function is: . Let , the precise position of the extreme point candidate point is obtained: . The precise positioning position is brought into the DOG Taylor expansion, and the first two terms are taken: . The obtained D(X) is compared with the threshold value, and if the requirement is met, it indicates that the point is a relatively stable extreme point. The edge effect of the DOG operator is removed by obtaining the Hessian matrix at the feature point.

[0032] Set the direction parameter of the feature point: take the local feature description of the pixel point as the reference direction, assign it to the corresponding point, and count the direction and gradient of these pixels in the histogram area. Figure 4 As shown in the figure, the peak direction is the main direction of the key point. The pixel gradient modulus and direction in the key point area are represented as:

[0033] ; .

[0034] Generate the descriptor of the feature point. For all the extracted feature points, we generate a descriptor as the identification of the image feature. Rotate the coordinate axis to be consistent with the direction of the feature point and select a window, i.e. calculate the gradient direction histogram in a 4x4 area (take 8 directions). Generate a seed point by calculating the cumulative value of these data, and finally generate a 4x4x8=128-dimensional feature vector.

[0035] Because the relative height and angle of the camera installation and the ground are basically unchanged, the proportion relationship between the adjacent two corner points can be obtained by using the image coordinate difference measured during the top-down transformation and the actual distance of the two corner points on the calibration board. Let the image coordinate difference value be d, and the distance of the corner points on the calibration board be D, then the proportion k is defined as follows: k=d / D. The overall algorithm flow chart is shown in Figure 5 .

[0036] The top front and back sides of the cable winding and unwinding vehicle are installed with binocular cameras C11 and C44 at fixed points, which are fused at the data level with C1 and C4. The binocular cameras C11 / C44 measure the distance by calculating the parallax of the two images, without the need to judge the type of obstacles. The binocular cameras are not limited by recognition rate, have higher accuracy and do not need to maintain a sample database. In order to fuse the detection results of the binocular cameras C11 / C44 and the fisheye cameras C1 / C4, the data in their respective coordinate systems needs to be unified under a common coordinate system. This involves determining the spatial position transformation relationship between the binocular cameras and the fisheye cameras, i.e. the external parameters. The external parameters include a rotation matrix and a translation vector, which describe the pose and position of one coordinate system relative to another. The conversion process is shown in Figure 6 . After determining the external parameters, the detection results of the binocular cameras C11 / C44 and the fisheye cameras C1 / C4 can be mapped to the same coordinate system, so as to perform subsequent fusion and analysis, and improve the performance and robustness of the perception system.

[0037] Wherein the binocular camera C11 / C44 directly obtains scene depth information, and the positioning accuracy of the close-range obstacle is high. The fisheye camera C1 / C4 provides high-resolution texture information, and assists the binocular in matching in a low-texture area. The binocular provides an "absolute depth" prior, and the monocular (fisheye) supplements "semantic and texture constraints", thereby improving the depth reliability in a complex scene; the monocular can be used as a redundancy when the binocular fails (such as strong light, occlusion leading to binocular matching failure). The depth data fusion thereof adopts a weighted fusion strategy: the binocular depth is mainly used in a close-range (high Dstereo weight), and the monocular depth is mainly used in a long-range (high Dmono weight), and the fusion formula is: wherein w 1 and w 2 are dynamically adjusted according to the distance and the confidence (such as ). The monocular depth is used to fill in the binocular depth hole area (such as a low-texture open-pit mine retaining wall); and the binocular depth is used to constrain the monocular depth in an uncertain area (such as a long-distance blurred object).

[0038] The front obstacle is ranged, and the three-dimensional positioning of the obstacle uses a semantic mask Mobj to extract obstacle pixels in a monocular image; the pixel coordinates are converted to a binocular coordinate system through camera external parameters, combined with the fused depth value Dfinal, and the three-dimensional coordinates (X, Y, Z) of the obstacle are calculated; the positioning results of multiple frames are subjected to Kalman filtering or particle filtering, the trajectory is smoothed, and the noise influence is reduced.

[0039] Specifically, in order to obtain a panoramic overhead view around, it is necessary to collect images around. The collection device can generally be divided into ordinary wide-angle cameras and wide-angle cameras. The image collected by the wide-angle camera has small distortion and conforms to the visual habit of the human eye without distortion correction, but the visual angle of the wide-angle camera is 40-90, and the image range covered is limited, so the number of cameras needs to be increased to make up for the shortage of the field of view, which increases the installation difficulty and the amount of calculation. The visual angle of the wide-angle camera is usually greater than 120, and some even exceed 180, so a larger visual range can be obtained under the condition of a certain number of installations, but it also leads to image distortion. The system installs six fisheye cameras with a visual angle of 180 degrees around the cable winding and unwinding vehicle, as shown in Figure 7 .

[0040] The fisheye lens is an extreme wide-angle lens with a short focal length and a large visual angle, and it is named because its appearance is similar to a fish eye. The visual angle of the fisheye lens can reach or even exceed 180. The advantage of the fisheye lens is that it has a super large visual angle and can include a larger scene range. Due to the characteristics of the super large visual angle, the fisheye lens has been widely used in video conferencing, panoramic monitoring, intelligent transportation, virtual reality (VR) and other fields. The fisheye lens can ensure a large visual angle, but it also has serious image distortion, which does not conform to the visual habit of human beings, so distortion correction is generally required.

[0041] In the panoramic surround view image system, considering the actual installation difficulty and algorithm complexity, it is not suitable to use ordinary straight lens. The camera used in this system is a 180 fish-eye camera, and the camera parameters are shown in Table 1.

[0042] Table 1 Fish-eye camera parameters The fish-eye camera obtains a large angle image, but also brings image distortion, as shown in Figure 5 The fish-eye correction is the key of panoramic surround view image system, and the correction effect directly affects the next work. The commonly used fish-eye correction algorithm can be divided into correction method based on projection model and correction method based on 2D and 3D space. The projection model can be divided into spherical projection model and parabolic imaging model, among which the representative algorithm is spherical perspective projection method. The correction method based on 2D space realizes distortion correction according to the proportion of the projection relationship, and the representative algorithm is spherical coordinate method and double longitude method. The correction method based on 3D space is to map the points on 2D plane to the 2D plane composed of 3D scene, and the common algorithms are projection conversion method and calibration method. In order to obtain a larger image range, the camera needs to have a certain angle with the vertical line when installed, so the image obtained is not a bird's eye view, but a side view. The image obtained by panoramic surround view image system should be a bird's eye view, so the side view should be transformed into a bird's eye view on the basis of fish-eye correction. The transformation from side view to bird's eye view can be regarded as the perspective transformation of image. Perspective transformation is also called homography mapping, which can be understood as the calculation method of three-dimensional plane perceived by a specific observer, and the observer does not necessarily observe the plane vertically.

[0043] The images processed by fish-eye correction and perspective transformation are independent images, in order to obtain a complete bird's eye view, the images need to be spliced. Most of the existing panoramic surround view technology is to directly crop the image or find the splicing line by marking two points in the overlapping area. These schemes have great defects, mainly reflected in that the spliced image has a splicing seam, the image transition is not natural, and the visual effect is affected. The feature-based matching method can accurately determine the transformation relationship between images, and has strong stability to rotation and brightness change. However, because the images collected by panoramic surround view image system are severely distorted after preprocessing, the overlapping area between two images is small, and the real-time requirement is high, the feature-based matching algorithm has not been applied to this system for a long time. Therefore, the key to apply feature-based matching algorithm to panoramic surround view system is to ensure the quality of preprocessed image and reduce the operation time of algorithm.

[0044] The cable winding and unwinding vehicle body is 13.5 m long, 3.4 m wide, the fish-eye camera is installed at a height of 1.35 m, and the angle with the vertical line is 15. The virtual overhead image of the vehicle body is overlaid at the position corresponding to the vehicle body in the center of the image, and the final effect diagram is obtained, as shown in Figure 8 .

[0045] The above is a skin lesion image segmentation method provided by one or more embodiments of the present application. Based on the same idea, the present application also provides a corresponding skin lesion image segmentation device, as shown in Figure 9 .

[0046] Figure 9 A skin lesion image segmentation device provided by the present application is shown in the figure. The device comprises: An image acquisition module 901 is used to fix a plurality of fish-eye cameras and binocular cameras at a plurality of preset positions of a cable winding and unwinding vehicle. A plurality of cable winding and unwinding vehicle images at the same time are acquired through the plurality of fish-eye cameras and binocular cameras. A scene depth map of the cable winding and unwinding vehicle at the same time is acquired through the binocular camera. An image stitching module 902 is used to weight and fuse the cable winding and unwinding vehicle images and two scene depth maps acquired by the two fish-eye cameras at the same preset position of the binocular camera to obtain a cable winding and unwinding vehicle fusion image. Feature points of the remaining plurality of cable winding and unwinding vehicle images and the cable winding and unwinding vehicle fusion image are extracted and matched to obtain a coordinate position conversion relationship between the cable winding and unwinding vehicle images and the cable winding and unwinding vehicle fusion image. Based on the coordinate position conversion relationship, the plurality of cable winding and unwinding vehicle images are moved to the same coordinate system and stitched to obtain a cable winding and unwinding vehicle overhead stitching image. The brightness histogram information of each cable winding and unwinding vehicle image is extracted to adjust the color of the cable winding and unwinding vehicle overhead stitching image. The overlapping area in the cable winding and unwinding vehicle overhead stitching image after color adjustment is fused to obtain a cable winding and unwinding vehicle two-dimensional surround view stitching image. The cable winding and unwinding vehicle overhead stitching image after color adjustment is input into a preset three-dimensional spherical model to output a three-dimensional surround view stitching image of the cable winding and unwinding vehicle. A display module 903 is used to display the two-dimensional surround view stitching image and the three-dimensional surround view stitching image at each time in real time.

[0047] Optionally, it further comprises an obstacle detection module and a trajectory smoothing module. The obstacle detection module is used to identify obstacle pixels based on the three-dimensional surround view stitching image and convert the image coordinates of the obstacle pixels into three-dimensional coordinates. The trajectory smoothing module is used to perform Kalman filtering or particle filtering on the three-dimensional coordinates of the obstacle in the continuous frames to obtain a smooth obstacle motion trajectory.

[0048] Optionally, the display module is further configured to superimpose and display the obstacle movement track and the real-time distance information.

[0049] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered to be within the scope of the present application.

Claims

1. A method of splicing for a surround view adapted to a cable reel, characterized by, The method comprises the following steps: fixing multiple fisheye cameras and binocular cameras at multiple preset positions of a cable winding and unwinding vehicle respectively; collecting multiple images of the cable winding and unwinding vehicle at the same time through the multiple fisheye cameras and binocular cameras; collecting a scene depth map of the cable winding and unwinding vehicle at the same time through the binocular cameras; weighting and fusing the images of the cable winding and unwinding vehicle collected by the two fisheye cameras at the same preset positions of the binocular cameras and the two scene depth maps to obtain a fused image of the cable winding and unwinding vehicle; extracting feature points of the remaining multiple images of the cable winding and unwinding vehicle and the fused image of the cable winding and unwinding vehicle and performing feature point matching to obtain a coordinate position conversion relationship between the images of the cable winding and unwinding vehicle and the fused image of the cable winding and unwinding vehicle; based on the coordinate position conversion relationship, moving the multiple images of the cable winding and unwinding vehicle to the same coordinate system and splicing to obtain an overhead spliced image of the cable winding and unwinding vehicle; extracting brightness histogram information of each image of the cable winding and unwinding vehicle to adjust the color of the overhead spliced image of the cable winding and unwinding vehicle; fusing the overlapping area in the overhead spliced image of the cable winding and unwinding vehicle after the color is adjusted to obtain a two-dimensional surround view spliced image of the cable winding and unwinding vehicle; and inputting the overhead spliced image of the cable winding and unwinding vehicle after the color is adjusted into a preset three-dimensional spherical model to output a three-dimensional surround view spliced image of the cable winding and unwinding vehicle; real-time displaying the two-dimensional surround view spliced image and the three-dimensional surround view spliced image at each time.

2. A stitch around method adapted to a cable reel, according to claim 1, characterized in that, The optical axis direction of the binocular camera is arranged at an acute angle with the ground; and the fisheye camera has a visual angle of 180 degrees.

3. A stitch-in-the-round method for adapting a cable reel to a cable reel truck as claimed in claim 1, characterized in that, The multiple fisheye cameras and binocular cameras are fixed at multiple preset positions of the vehicle, specifically comprising: fixing six fisheye cameras at the center of the vehicle head, the center of the vehicle tail, the front 25% quantile point of the left side of the vehicle body, the rear 25% quantile point of the left side of the vehicle body, the front 25% quantile point of the right side of the vehicle body and the rear 25% quantile point of the right side of the vehicle body respectively; fixing two binocular cameras at the center of the vehicle head and the center of the vehicle tail respectively.

4. A stitch around method for adapting a cable reel to a cable reel truck as claimed in claim 1, characterized in that, The two fisheye cameras at the same preset positions of the binocular cameras collect the images of the cable winding and unwinding vehicle and the scene depth maps, and the images and the scene depth maps are weighted and fused to obtain a fused image of the cable winding and unwinding vehicle, specifically comprising: based on the position conversion relationship between the fisheye camera and the binocular camera, converting the fisheye camera coordinate system in which the image of the cable winding and unwinding vehicle is located and the binocular camera coordinate system in which the scene depth map is located to the same image coordinate system; performing weighted fusion on the pixel distance between the two scene depth maps and the pixel distance between the two images of the cable winding and unwinding vehicle collected by the two fisheye cameras at the same preset positions of the binocular cameras to obtain the weighted fused pixel distance, and the formula is: ； In the formula, D final is the pixel distance of the weighted fused image, D stereo is the pixel distance of the two scene depth maps, D momo is the pixel distance between the cable reel images collected by the fisheye camera at the same position as the preset position of the binocular camera, w 1 is the pixel distance weight between the two scene depth maps, w 2 is the pixel distance weight between the cable reel images collected by the fisheye camera at the same position as the preset position of the binocular camera. based on the weighted fused pixel distance, pixel by pixel fusing the images of the cable winding and unwinding vehicle collected by the two fisheye cameras at the same preset positions of the binocular cameras and the scene depth maps to obtain a fused image of the cable winding and unwinding vehicle.

5. A stitch-in-the-round method for adapting a cable reel to a cable reel truck as claimed in claim 1, characterized in that, The feature points of the remaining multiple images of the cable winding and unwinding vehicle and the fused image of the cable winding and unwinding vehicle are extracted and feature point matching is performed to obtain a coordinate position conversion relationship between the images of the cable winding and unwinding vehicle and the fused image of the cable winding and unwinding vehicle, specifically comprising: The two-dimensional Gaussian kernel function is convolved with the cable reel image to construct a cable reel scale space, and a candidate extreme point is detected by a Gaussian difference (DOG) space; the candidate extreme point is located at a sub-pixel level based on a Taylor expansion of the DOG function, and an edge response is removed by a Hessian matrix to obtain feature points of multiple cable reel images; directions and gradients of pixel points in a designated field of the feature points of the cable reel image are counted; and a feature point descriptor of the cable reel image is generated based on the directions and gradients of the pixel points; Matching is performed on the feature point descriptors in each pair of adjacent cable reel images to obtain matching point pairs; A rotation matrix and a translation vector between each pair of adjacent cable reel images are calculated based on the matching point pairs; the rotation matrix and the translation vector are coordinate position conversion relationships of the multiple cable reel images.

6. A stitch around method for adapting a cable reel to a cable reel truck as claimed in claim 5, characterized in that, The feature point descriptor of the cable reel image is generated based on the directions and gradients of the pixel points, and specifically includes: A neighborhood window of a designated size is selected with the feature point as the center, and the neighborhood window is divided into a plurality of sub-regions; A gradient direction histogram of each sub-region is calculated; each gradient direction histogram contains a plurality of direction intervals, forming a multi-dimensional feature vector; The multi-dimensional feature vectors of each sub-region are spliced and normalized to obtain the feature point descriptor of each cable reel image.

7. A wrap-around splicing system adapted to a cable reel, characterized in that It includes: An image acquisition module is configured to fix a plurality of fisheye cameras and binocular cameras at a plurality of preset positions of a cable reel; The plurality of fisheye cameras and binocular cameras are used to acquire a plurality of cable reel images at the same time; A scene depth map of the cable reel at the same time is acquired by the binocular camera; An image stitching module is configured to fuse two cable reel images and two scene depth maps acquired by two fisheye cameras at the same preset position of the binocular camera to obtain a cable reel fusion image; Feature points of the remaining plurality of cable reel images and the cable reel fusion image are extracted and matched to obtain a coordinate position conversion relationship between the cable reel images and the cable reel fusion image; based on the coordinate position conversion relationship, the plurality of cable reel images are moved to the same coordinate system and stitched to obtain a cable reel overhead stitching image; The brightness histogram information of each cable reel image is extracted to adjust the color of the cable reel overhead stitching image; the overlapping area in the cable reel overhead stitching image after color adjustment is fused to obtain a cable reel two-dimensional surround view stitching image; and the cable reel overhead stitching image after color adjustment is input into a preset three-dimensional spherical model to output a three-dimensional surround view stitching image of the cable reel; A display module is configured to display the two-dimensional surround view stitching image and the three-dimensional surround view stitching image at each time in real time.

8. A wrap around splicing system adapted for use with a cable reel as defined in claim 7, wherein, It also includes an obstacle detection module and a trajectory smoothing module. The obstacle detection module is configured to identify obstacle pixels based on the three-dimensional all-around-view spliced image, and convert image coordinates of the obstacle pixels into three-dimensional coordinates; and the trajectory smoothing module is configured to perform Kalman filtering or particle filtering on the three-dimensional coordinates of the obstacle in consecutive frames to obtain a smoothed obstacle motion trajectory.

9. A wrap around splicing system adapted for use with a cable reel, as defined in claim 7, wherein, The display module is further configured to superimposedly display the obstacle motion trajectory and real-time distance information.

Images