A multi-camera image stitching method for a multi-package separation system

Abstract

The invention discloses a multi-camera image stitching method for a multi-package separation system, and relates to the technical field of image stitching. By setting identification labels on a conveyor belt and performing proportional conversion on all photographed pictures according to the identification labels, it can be ensured that all pictures are processed at the same proportion, and a good pre-processing step is provided for a stitching algorithm; then, according to the similarity between the identification labels in the target picture and the originally photographed sub-item picture and the start picture, it is determined whether the stitching method adopted this time meets the requirements, and if so, the target picture is obtained, otherwise, the next stitching method is automatically switched until the target picture that meets the requirements is obtained. This method can automatically screen the existing stitching algorithms, and only needs to be preset and entered before stitching; the application is simple, effective, and easy to use.

Description

Technical Field

[0001] The invention belongs to the technical field of image stitching, and in particular to a multi-camera image stitching method for a multi-package separation system. Background Art

[0002] The multi-express separation system has high requirements for the detection efficiency of the express detection algorithm. The field of view of a single camera is limited and cannot collect all images of the conveyor belt at the same time. The panoramic camera is expensive and is not suitable for large-scale application of the multi-express separation system. Therefore, it is necessary to install multiple cameras on the multi-express separation system to completely cover the entire field of view of the sorting task.

[0003] However, mechanical errors may occur in the installation of cameras, and it cannot be guaranteed that each camera is installed horizontally and parallel to each other. Therefore, the images collected by the cameras are not aligned. Moreover, when stitching multiple pictures, the angles of the cameras facing the conveyor belt may be different due to the distance between the camera and the conveyor belt. How to achieve accurate stitching of pictures? The prior art provides many stitching algorithms, but there is a lack of a suitable way to measure whether different stitching algorithms are suitable for different scenes, and there is a lack of a good solution for the pre-processing of stitching. Based on this, a solution is now provided. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art;

[0005] To this end, the present invention proposes a multi-camera image stitching method for a multi-package separation system, which specifically includes the following steps:

[0006] Get all the photos taken by the camera and mark them as sub-item images. Several sub-item images correspond to the camera, and there are overlapping areas between the sub-item images taken by the cameras in sequence. Determine the starting image and the sub-item images according to the camera position and the appearance of the conveyor belt. Determine the conversion ratio between each sub-item image and the starting image according to the same identification label in the overlapping area between the starting image and the sub-item images.

[0007] Determine the reference image according to the conversion ratio, and reduce all other sub-item images or sub-item images and the initial image, so that all conversion ratios after reduction are 1; all the reduced images are marked as preliminary images;

[0008] Obtain the reference image and all preliminary images in order, and splice them so that the overlapping areas between adjacent images are completely overlapped to obtain the target image;

[0009] Then, based on the similarity between the recognition labels in the target image and the originally captured sub-item image and the starting image, determine whether the stitching method used this time meets the requirements. If so, the target image is obtained. Otherwise, the next stitching method is automatically switched until a target image that meets the requirements is obtained and the target image is updated.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] This application sets identification tags on the conveyor belt and performs proportional conversion on all captured images according to the identification tags, so as to ensure that all images are processed at the same scale, providing a good pre-processing step for the stitching algorithm;

[0012] Then, according to the similarity between the identification labels in the target image and the originally photographed sub-item image and the starting image, it is determined whether the stitching method used this time meets the requirements. If it does, the target image is obtained, otherwise the next stitching method is automatically switched until the target image that meets the requirements is obtained. This method can automatically screen the existing stitching algorithms, and only needs to be preset and entered before stitching; this application is simple, effective, and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention;

[0014] Figure 2 This is a splicing verification flow chart of the present invention. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] See also Figure 1 The present application provides a multi-camera image stitching method for a multi-package separation system. As an embodiment 1 of the present application, the specific method of the present application includes the following steps:

[0017] Step 1: First, set up multiple camera positions. Several camera positions should be able to cover the entire area to be photographed, and there should be some overlapping areas between them. The specific settings are as follows:

[0018] First, a conveyor belt is obtained. The conveyor belt is a conveyor belt in a multi-express separation system for parcels. The conveyor belt is set in a manner that requires the minimum number of cameras to be set while ensuring that there is an overlapping area between cameras and the overlapping area exceeds a preset ratio. Here, it is also necessary to ensure that all cameras are set to maintain a high degree of consistency and are set in parallel with each other.

[0019] After the camera is set up, an identification label is set on the conveyor belt in the overlapping area; the identification label here is a preset sticker, which can of course be other items, which have the characteristics of being very different from the conveyor belt and easy to identify;

[0020] The preset ratio here is understood as the ratio of the area of ​​the conveyor belt itself that can be photographed by the camera to its total area; of course, the area of ​​the conveyor belt photographed here can be measured offline, for example, after shooting with any camera, its coverage edge is determined offline, and then the area of ​​the enclosed part is measured and converted into a ratio;

[0021] For example, a camera is set above the conveyor belt, and the camera position is kept in the middle position above the conveyor belt. Then the first camera is set. The camera is placed so that the edge of one end of the conveyor belt is just covered and then stopped. In this way, the first camera is set. Then the second camera is set in a line along the position of the first camera. The second camera is kept at the same height and parallel to the first camera. The setting is complete.

[0022] Step 2: Get the photos taken by each camera and mark them as sub-items. Figure 1 1. Correspond to the camera at the corresponding position; perform preliminary processing on the sub-item map:

[0023] S1: restore the photos according to the camera position, automatically give them a sequence mark, and select the starting picture. Here, you can select according to the sequence mark, or you can also use other methods to get the starting picture;

[0024] S2: Perform benchmark positioning on the start image. The specific method of benchmark positioning is as follows:

[0025] First, get the identification label in the start image, and get the area of ​​the identification label. The area is obtained as follows:

[0026] Circle the edge of the identification label from the starting image, and then obtain the entire circled area. This process can be compared with the preset identification label. After determining the range, mark the edge and then automatically calculate the area. The area here is measured in pixels. Because the identification label here is a preset label, it can be initially designed as a square, which is convenient for calculating the area and can be obtained using the square area calculation formula;

[0027] The calculated identification tag is used as the reference area;

[0028] Then, a sub-item image adjacent to the start image is selected, and the area of ​​the identification label in the sub-item image that is consistent with the start image is obtained. The area is divided by the reference area to obtain the ratio between the sub-item image and the start image, which is marked as the conversion ratio;

[0029] Then, the identification tags existing in the overlapping area between the sub-item graph and the second sub-item graph at the other end are obtained, and the areas in the respective sub-item graphs are calculated respectively. The ratio between the two sub-item graphs is determined according to the areas, and the ratio is marked as the conversion ratio. The conversion method can refer to the above steps to obtain the conversion ratio between the sub-item graph and the second sub-item graph. According to the ratio between the sub-item graph and the starting picture, the ratio is automatically converted to the ratio between the second sub-item graph and the starting picture, and the ratio is marked as the conversion ratio between the second sub-item graph and the starting picture;

[0030] Then, the same process is performed on all other sub-item images to obtain the conversion ratio between all sub-item images and the starting image;

[0031] Here is a specific example. First, according to the starting picture, the area of ​​the identification label in the starting picture is automatically obtained, assuming it is A1. Then, the area of ​​the same identification label in the first sub-item picture is obtained, assuming it is A2. Then, the conversion ratio is obtained according to the formula A1 / A2. Then, for the first sub-item picture, the picture taken by the camera next to its corresponding camera is selected, that is, the second sub-item picture with an overlapping area with the first sub-item picture, and the area of ​​the identification label in the overlapping coverage area of ​​the two in the first sub-item picture is obtained, which is marked as A3. The area of ​​the identification label in the second sub-item picture is obtained and marked as A4. First, A3 / A4 is obtained to obtain the result, and then (A1 / A2) / (A3 / A4) is obtained to obtain the conversion ratio between the starting picture and the second sub-item picture.

[0032] Get the conversion ratio between all sub-item pictures and the starting picture;

[0033] S3: Get the maximum value in the conversion ratio and mark it as the upper limit conversion value. When the upper limit conversion value is less than 1, it means that the starting picture at this time is the smallest picture among all the sub-item pictures plus the starting picture. The smallest picture means that the camera is farthest from the conveyor belt at this time, that is, the most content is gathered in each pixel. At this time, the ratio of the starting picture is used as the reference picture, and all the other sub-item pictures are reduced according to the value of the conversion ratio to obtain all the processed sub-item pictures, which are all marked as preliminary pictures. Each preliminary picture has an initial position mark, which can correspond to the camera position.

[0034] When the upper conversion value is not less than 1, the sub-item picture corresponding to the upper conversion value is marked as the reference picture, and the conversion ratios between all other starting pictures, sub-item pictures and reference pictures are automatically obtained. The sub-item pictures and reference pictures are reduced in proportion to ensure that all conversion ratios are 1 in the end; all processed pictures are marked as preliminary pictures;

[0035] Step 3: Obtain the reference image and all preliminary images in order, and splice them so that the overlapping areas between adjacent images are completely overlapped to obtain a complete schematic diagram, which is marked as the target image;

[0036] The splicing here is achieved with the help of an existing splicing algorithm; of course, this application provides an example, which is roughly in the following way:

[0037] By selecting the repeated ROI area in the first image;

[0038] Based on the repeated ROI area, the template matching method is used, and the sliding window method is used to match the template in the second image. The matching matrix is ​​obtained according to the normalized cross-correlation algorithm. The value of the matching matrix is ​​0-1, and the value of the best matching position is 1.

[0039] Traverse the matching matrix, get the position of the maximum matching value, and save the pixel coordinates of this position.

[0040] The real-time RGB images of the cameras corresponding to the two images are solved using the pixel coordinates of the matching positions, and the images are spliced ​​according to the solved splicing positions.

[0041] Of course, the above methods are only for reference and are not specifically limited. Any existing technology that can achieve image stitching can be used;

[0042] As the second embodiment of the present invention, this embodiment is carried out on the basis of the first embodiment, and the difference from the first embodiment is that after the splicing step of step 3 is completed, the present embodiment needs to perform splicing verification, and the specific method of splicing verification is as follows:

[0043] SS1: First, obtain the initial start image and several sub-item images, obtain all the images with identification labels in the target image, and mark them as complete scene images;

[0044] For each image with a recognition label, there will be a corresponding image before stitching in the start image and the sub-item image, which will be marked as a reference image. There are two reference images corresponding to the same scene image, which exist in two sub-item images, or one sub-item image and one start image.

[0045] SS2: Select any scene image, and determine the similarity between the scene image and the corresponding two reference images to obtain two similarity values. After taking the average, mark the value as the splicing value.

[0046] SS3: Select the next scene image and repeat steps SS2-SS3 until the splicing values ​​of all scene images are obtained;

[0047] SS4: Get the number of scene images whose stitching value exceeds X1, divide it by the total number, and get the stitching judgment ratio; here X1 is a preset value, which is preset by the administrator and is generally 0.85;

[0048] SS5: When the splicing judgment ratio is lower than X2, it means that the splicing algorithm used here is unqualified, and the remaining splicing methods are automatically selected until a splicing method is selected to obtain a splicing judgment ratio not lower than X2. The result obtained in this way is marked as the target image;

[0049] Here X2 is a preset value, usually 0.8; of course, the administrator can modify it to other values;

[0050] SS6: Get the spliced ​​target image.

[0051] As the third embodiment of the present invention, the specific method of this embodiment is to integrate the first embodiment and the second embodiment for implementation.

[0052] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0053] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-camera image stitching method for a multi-package separation system, characterized in that: The method comprises the following steps: Get all the photos taken by the camera and mark them as sub-item images. Several sub-item images correspond to the camera, and there are overlapping areas between the sub-item images taken by the cameras in sequence. Determine the starting image and the sub-item images according to the camera position and the appearance of the conveyor belt. Determine the conversion ratio between each sub-item image and the starting image according to the same identification label in the overlapping area between the starting image and the sub-item images. Determine the reference image according to the conversion ratio, and reduce all other sub-item images or sub-item images and the initial image, so that all conversion ratios after reduction are 1; all the reduced images are marked as preliminary images; Obtain the reference image and all preliminary images in order, and splice them so that the overlapping areas between adjacent images are completely overlapped to obtain the target image; Then, according to the similarity between the recognition label in the target image and the original sub-item image and the starting image, it is determined whether the stitching method used this time meets the requirements. If it meets the requirements, the target image is obtained. Otherwise, the next stitching method is automatically switched until a target image that meets the requirements is obtained, and the target image is updated; The specific method to obtain the preliminary graph is: First, obtain the photos taken by each camera, mark them as sub-item pictures, and determine the starting picture according to the position of the picture on the conveyor belt; Get the identification label in the start image, get the area of ​​the identification label on the image, and mark it as the reference area; Then, a sub-item image adjacent to the start image is selected, and the area of ​​the identification label in the sub-item image that is consistent with the start image is obtained. The area is divided by the reference area to obtain the ratio between the sub-item image and the start image, which is marked as the conversion ratio; Then, the identification tags existing in the overlapping area between the sub-item graph and the second sub-item graph at the other end are obtained, and the areas in the respective sub-item graphs are calculated respectively. The ratio between the two sub-item graphs is determined according to the areas, and the ratio is marked as the conversion ratio. The conversion ratio between the sub-item graph and the second sub-item graph is obtained, and the ratio between the sub-item graph and the starting image is automatically converted to the ratio between the second sub-item graph and the starting image, and the ratio is marked as the conversion ratio between the second sub-item graph and the starting image; Then, the same process is performed on all other sub-item images to obtain the conversion ratio between all sub-item images and the starting image; Get the conversion ratio between all sub-item pictures and the starting picture; The maximum value in the conversion ratio is obtained and marked as the upper conversion value. When the upper conversion value is less than 1, the ratio of the start image is used as the base image, and all other sub-item images are reduced according to the value of the conversion ratio to obtain all processed sub-item images, which are all marked as preliminary images; each preliminary image has an initial position mark, which can correspond to the camera position; When the upper limit conversion value is not less than 1, the sub-item picture corresponding to the upper limit conversion value at this time is marked as the benchmark picture, and the conversion ratios between all other starting pictures, sub-item pictures and benchmark pictures are automatically obtained. The sub-item pictures and the benchmark pictures are reduced in proportion to ensure that all the final conversion ratios are 1; all processed pictures are marked as preliminary pictures.

2. The multi-camera image stitching method for a multi-package separation system according to claim 1, characterized in that: Before splicing, the following steps are required: Set up multiple camera positions, and make sure that several of them can cover the entire area that needs to be shot, and there are overlapping areas between them.

3. The multi-camera image stitching method for a multi-package separation system according to claim 2, characterized in that: The specific method of setting the camera position is as follows: First, obtain the conveyor belt and set it in a way that requires the minimum number of cameras while ensuring that there is an overlapping area between cameras and the overlapping area exceeds a preset ratio; and all cameras are set to be highly consistent and parallel to each other.

4. The multi-camera image stitching method for a multi-package separation system according to claim 3, characterized in that: The conveyor belt is a conveyor belt in a multi-express separation system for parcels; After the camera is set up, an identification tag is set on the conveyor belt in the overlapping area; the identification tag here is an object that is very different from the conveyor belt and is easy to identify.

5. The multi-camera image stitching method for a multi-package separation system according to claim 3, characterized in that: The preset ratio refers to the ratio of the area of ​​the conveyor belt captured by the camera to its total area; The area of ​​the conveyor belt captured was calculated by determining the edge of the capture, then measuring the area of ​​the enclosed portion and dividing it by the total area.

6. The multi-camera image stitching method for a multi-package separation system according to claim 1, characterized in that: The area on the graph is obtained as follows: Circle the edge of the identification tag from the starting image, and then obtain the entire circled area. This process can be compared with the preset identification tag. After determining the range, mark the edge and then automatically calculate the area. Here, the area is measured in pixels. The calculated identification label is used as the reference area.

7. The multi-camera image stitching method for a multi-package separation system according to claim 1, characterized in that: The specific method of updating the target graph is: SS1: First, obtain the initial start image and several sub-item images, obtain all the images with identification labels in the target image, and mark them as complete scene images; For each image with a recognition label, there will be a corresponding image before stitching in the start image and the sub-item image, which will be marked as a reference image. There are two reference images corresponding to the same scene image, which exist in two sub-item images, or one sub-item image and one start image. SS2: Select any scene image, determine the similarity between the scene image and the corresponding two reference images, obtain two similarity values, take the average and mark the average as the splicing value; SS3: Select the next scene image and repeat steps SS2-SS3 until the splicing values ​​of all scene images are obtained; SS4: Get the number of scene images whose stitching value exceeds X1, divide it by the total number, and get the stitching judgment ratio; here X1 is a preset value, which is preset by the administrator and is generally 0.85; SS5: When the splicing judgment ratio is lower than X2, it means that the splicing algorithm used here is unqualified, and the remaining splicing methods are automatically selected until a splicing method is selected to obtain a splicing judgment ratio not lower than X2. The result obtained in this way is marked as the target image; Here X2 is the preset value; SS6: Get the spliced ​​target image.

Images