Multi-image matching and fusion method, device, storage medium, and logistics collection system

Abstract

Disclosed in an embodiment of the present invention is a multi-image matching and fusion method, which is applied to a control device. The control device is located in a logistics collection system. The logistics collection system also includes multiple cameras and conveying devices. The cameras are arranged above the conveying device. The multiple cameras are at different heights from the devices. The conveying device has a transport surface for conveying multiple packages. The method includes: obtaining multiple initial images of the transport surface within the field of view of each of the multiple cameras, the initial images being multiple; normalizing the multiple initial images to obtain multiple first images; splicing and fusing the multiple first images to obtain an output image; and judging whether there are any unrecognized packages on the transport surface based on the output image. This method facilitates staff to quickly locate unrecognized packages, and at the same time solves the problem in the existing image stitching technology that complex image features are required to accurately identify images.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and specifically to a multi-image matching and fusion method, device, storage medium, and logistics collection system. Background Art

[0002] In the logistics and sorting sector, transit stations bear a significant burden of express parcel sorting. Collection is a crucial step in the entire sorting process, determining the overall speed of sorting. For valuable express parcels, manual labor is often required to scan and collect each parcel individually, impacting delivery time.

[0003] To speed up package collection, a new system has emerged in the logistics field. This system uses multiple cameras to scan barcodes to complete the collection process. To ensure reliable collection, manual verification of all scans is required. The system annotates the multiple scan results on an image and provides it to staff to help locate unrecognized packages. This process involves stitching and displaying images from multiple cameras. Traditional image stitching technology suffers from poor image fusion and difficulty guaranteeing image recognition accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a multi-image matching and fusion method, device, storage medium and logistics collection system, which aim to solve the problems of poor image fusion effect and low image recognition accuracy in the image stitching method under the existing technology.

[0005] In a first aspect, the present application provides a multi-image matching fusion algorithm applied to a control device, wherein the control device is located in a logistics collection system, wherein the logistics collection system further includes multiple cameras and a conveyor, wherein the cameras are located above the conveyor, and the multiple cameras are at different heights from the conveyor, and the conveyor has a transport surface for conveying multiple packages;

[0006] The method comprises:

[0007] Acquire a plurality of initial images of the transport surface within the fields of view of each of the plurality of camera devices, wherein the number of the initial images is multiple;

[0008] performing normalization processing on the multiple initial images to obtain multiple first images;

[0009] performing splicing and fusing of the plurality of first images to obtain an output image;

[0010] It is determined based on the output image whether there is a package on the transport surface that has not been identified.

[0011] Furthermore, the normalizing the multiple initial images to obtain the multiple first images includes:

[0012] Selecting any one of the multiple camera devices as a target camera device;

[0013] Determining a standard image height according to the target camera device;

[0014] Acquiring a standard image pre-captured by the target camera device;

[0015] The plurality of initial images are normalized according to the standard height and the standard image to obtain a plurality of first images.

[0016] Furthermore, determining the standard image height according to the target camera device includes:

[0017] The height of the highest package among the multiple packages on the transport surface within the field of view of the target camera device at the first moment is obtained, and the height of the highest package is used as the standard height.

[0018] Furthermore, the normalization process is performed on the plurality of initial images according to the standard height and the standard image to obtain a plurality of first images, including:

[0019] Obtaining initial heights corresponding to the other camera devices except the target camera device among the plurality of camera devices, wherein the number of the initial heights is multiple;

[0020] Calculating a height conversion relationship according to the initial height and the standard height;

[0021] According to the height transformation relationship, the standard image and the multiple initial images, image transformation is performed on the multiple initial images to obtain the multiple first images.

[0022] Furthermore, the obtaining of the initial heights corresponding to the other camera devices except the target camera device among the plurality of camera devices includes:

[0023] Obtain the highest package height among the multiple packages on the transport surface within the respective corresponding fields of view of the multiple camera devices except the target camera device at the first moment, and use the height of the highest package as the initial height corresponding to the other camera devices.

[0024] Furthermore, the step of stitching and fusing the plurality of first images to obtain an output image includes:

[0025] determining an order of the plurality of first images;

[0026] determining an ideal area corresponding to a plurality of the camera devices, and cropping the plurality of the first images according to the ideal area to obtain a plurality of the second images;

[0027] The plurality of second images are stitched and fused to obtain the output image.

[0028] Furthermore, the step of stitching and fusing the plurality of second images to obtain the output image includes:

[0029] performing smooth gradient processing on edges of the plurality of second images to obtain the plurality of third images;

[0030] A plurality of the third images are stitched together to obtain the output image.

[0031] Furthermore, the control device includes a main control device and multiple sub-control devices, and the multiple sub-control devices are provided in a one-to-one correspondence with the multiple camera devices; after the multiple camera devices respectively obtain initial images of the transport surface within their respective fields of view, and the initial images are multiple, the method further includes:

[0032] The sub-control device decodes the multiple initial images respectively to obtain multiple first decoding results, where the multiple initial images include the first decoding results of the camera device identifying the packages within the respective fields of view.

[0033] Furthermore, the output image includes a plurality of first decoding results; and determining whether there is an unidentified package on the transport surface based on the output image includes:

[0034] The main control device identifies a plurality of first decoding results in the output image to determine whether there is a package that has not been identified.

[0035] Furthermore, the determining, based on the output image, whether there is an unrecognized package on the transport surface includes:

[0036] Decoding the output image to obtain a second decoding result;

[0037] According to the second decoding result, it is determined whether there is a package that has not been identified.

[0038] Furthermore, the determining, based on the output image, whether there is an unrecognized package on the transport surface includes:

[0039] In the output image, a package showing an identification mark is determined as a recognized package, and a package not showing an identification mark is determined as an unrecognized package.

[0040] In a second aspect, the present application provides a multi-image matching and fusion device, the device comprising:

[0041] a camera module, the camera module being used to acquire an initial image of the transport surface within a field of view of a camera device;

[0042] A normalization module, configured to perform normalization processing on the initial image to obtain a first image;

[0043] A processing module, configured to perform splicing and fusion of the first images to obtain an output image;

[0044] A judgment module is configured to judge whether there is an unidentified package on the transport surface based on the output image.

[0045] In a third aspect, the present application further provides a storage medium storing a computer program, which includes steps for implementing the multi-image matching and fusion algorithm when operated by a multi-image matching and fusion device.

[0046] In a fourth aspect, the present application further provides a logistics collection system, comprising a control device, multiple camera devices, a conveying device, and a storage medium, wherein the camera devices are disposed above the conveying device, the multiple camera devices are at different heights from the conveying device, and the conveying device has a transport surface for conveying multiple packages; the control device is configured to:

[0047] Acquire a plurality of initial images of the transport surface within the fields of view of each of the plurality of camera devices, wherein the number of the initial images is multiple;

[0048] performing normalization processing on the multiple initial images to obtain multiple first images;

[0049] performing splicing and fusing of the plurality of first images to obtain an output image;

[0050] It is determined based on the output image whether there is a package on the transport surface that has not been identified.

[0051] In an embodiment of the present application, a multi-image stitching and fusion method is provided, which obtains multiple initial images at different heights by setting multiple camera devices at different heights, selects any one of the initial images as a standard image, and obtains the height of the camera device corresponding to the standard image. The multiple initial images are normalized based on the standard image to obtain multiple first images, and the multiple first images are stitched and fused to obtain a final output image. The final output image includes the results of identifying multiple packages, which facilitates staff to quickly locate unidentified packages. At the same time, it solves the problem of the image stitching method under the existing technology that complex image features are required to accurately identify images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A schematic diagram of a flow chart of an embodiment of a multi-image matching and fusion method provided by the present invention;

[0054] Figure 2 A schematic flow chart of an embodiment of step 11 provided by the present invention;

[0055] Figure 3 A schematic diagram of a flow chart of an embodiment of obtaining an output image provided by the present invention;

[0056] Figure 4 A side view of an embodiment of a collection system provided by the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of a server involved in an embodiment of the present invention. Specific embodiments

[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0059] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present application, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0060] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that one of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0061] An embodiment of the present invention provides a multi-image matching and fusion method, which is applied to a control device located in a logistics collection system for collecting express parcels. The logistics collection system also includes multiple cameras and a conveyor. The cameras are positioned above the conveyor for capturing images. In this embodiment of the present invention, the cameras are positioned at different heights from the conveyor, and the cameras are configured to capture images within their respective fields of view. The conveyor has a transport surface for conveying multiple parcels.

[0062] Specifically, the imaging device may be a camera. Multiple cameras are positioned above the conveyor at varying distances from the conveyor. The cameras are configured to capture images of packages moving on the conveyor and generate package image information. In embodiments of the present invention, the multiple cameras need to be identical so that the initial images captured have the same resolution.

[0063] Furthermore, in an embodiment of the present invention, the conveying device may be one of a belt conveyor, a roller conveyor, a transfer conveyor, and a wheel conveyor. The conveying device has a transport surface for transporting packages, and the camera devices capture the transport surface within their respective fields of view to obtain initial images within their respective fields of view.

[0064] like Figure 1 FIG. 1 is a flow chart of an embodiment of a multi-image matching and fusion method provided by the present invention, and the method includes:

[0065] 10. Acquire multiple initial images of the transport surface within the respective fields of view of multiple camera devices, wherein the number of initial images is multiple.

[0066] 11. Perform normalization processing on the multiple initial images to obtain multiple first images.

[0067] 12. Perform stitching and fusion on the multiple first images to obtain an output image.

[0068] 13. Based on the output image, determine whether there are any unrecognized packages on the transport surface.

[0069] The multi-image matching and fusion method provided by an embodiment of the present invention sets multiple cameras at different heights, obtains multiple initial images at different heights, selects any one of the initial images as a standard image, and obtains the height of the camera device corresponding to the standard image. The multiple initial images are normalized based on the standard image to obtain multiple first images, and the multiple first images are spliced ​​and fused to obtain a final output image. The final output image includes the results of identifying multiple packages, which facilitates staff to quickly locate unidentified packages. At the same time, it solves the problem in the image stitching method under the existing technology that complex image features are required to accurately identify images.

[0070] In an embodiment of the present invention, the cameras capture their respective fields of view, acquiring an initial image of the transport surface within their respective fields of view. Each camera captures one initial image. Since the conveyor continuously transports packages, once a package is placed on the conveyor, the conveyor begins operating to transport the package. The conveyor operates in a fixed cycle. After a cycle completes and all packages are within the camera's field of view, the conveyor stops, at which point the cameras begin capturing images. Once the capture is complete and all cameras have captured an initial image within their respective fields of view, the conveyor resumes transport, removes the identified and collected packages from the conveyor, and replaces new, unidentified packages on the conveyor's transport surface for subsequent capture, or for manual identification of unidentified packages. Therefore, during the conveyor's transport cycle, the cameras capture only one initial image within their respective fields of view. It should be noted that the conveyor's transport cycle is manually set and can vary depending on actual transport conditions.

[0071] After the camera captures the initial image of the transport surface within the field of view, the camera sends the initial image to the control device for subsequent normalization processing, thereby obtaining a plurality of first images.

[0072] In the embodiments provided by the present invention, Figure 2 FIG. 1 is a flow chart of an embodiment of step 11 provided by the present invention. Step 11 may include:

[0073] 20. Select any one of the multiple camera devices as a target camera device.

[0074] 21. Determine the standard image height based on the target camera device.

[0075] 22. Obtain a standard image pre-taken by the target camera device.

[0076] 23. According to the standard height and the standard image, the multiple initial images are normalized to obtain multiple first images.

[0077] In an embodiment of the present invention, since the height distances between multiple camera devices and the conveying device are different, although the initial images captured by different camera devices have the same resolution, the scales of the packages in the initial images are not uniform. Therefore, it is necessary to determine a standard height and a standard image, and use the standard height and the standard image as the standard to normalize the other initial images except the standard image to unify the scale of the initial images.

[0078] Specifically, in some embodiments of the present invention, the camera device farthest from the transport surface can be used as the target camera device; of course, the camera device closest to the transport surface can also be used as the target camera device; or a camera device at any distance from the transport surface can be selected as the target camera device. In subsequent processing, the target camera device is used as a standard to normalize the initial images captured by other cameras. The determination of the target camera device can be set according to actual circumstances and is not limited here.

[0079] In a specific embodiment of the present invention, taking the camera device farthest from the transport surface as the target camera device, after determining the target camera device, the standard height of the image can be determined. In this embodiment of the present invention, the standard height of the image can be determined based on the target camera device:

[0080] The height of the highest package among multiple packages on the transport surface within the field of view of the target camera device at the first moment is obtained, and the height of the highest package is used as the standard height.

[0081] At the first moment, the camera device will capture the transport surface to obtain an initial image, and after determining the camera device farthest from the transport surface as the target camera device, the standard height can be determined. Specifically, at the first moment, there are multiple packages in the field of view corresponding to the target camera device, and the package height corresponding to the package with the highest package height among the multiple packages is determined as the standard height H. The package height corresponding to the package with the highest package height, that is, the standard height H, can be confirmed by a grating set on the conveyor. The specific process of using the grating to confirm the height of different packages can refer to the existing technology and is not limited here. After determining the target camera device, the initial image pre-captured in the target camera device can also be obtained as the standard image.

[0082] In an embodiment of the present invention, after the target camera device, the standard height, and the standard image are confirmed, it is necessary to normalize the initial images other than the standard image based on the standard height and the standard image to unify the scale of the initial images.

[0083] In an embodiment of the present invention, performing normalization processing on the multiple initial images according to the standard height and the standard image to obtain the multiple first images may include:

[0084] Obtain the initial heights corresponding to the other camera devices except the target camera device among the multiple camera devices, where the initial heights are multiple; calculate the height transformation relationship based on the initial heights and the standard heights; and perform image transformation on the multiple initial images based on the height transformation relationship, the standard image and the multiple initial images to obtain multiple first images.

[0085] In the above embodiment, obtaining the initial heights corresponding to the other camera devices except the target camera device among the multiple camera devices may include: obtaining the highest package height among the multiple packages on the transport surface within the respective corresponding fields of view of the other camera devices except the target camera device among the multiple camera devices at the first moment, and using the height of the highest package as the initial heights corresponding to the other camera devices.

[0086] Specifically, in the above embodiment, the standard height is the height of the highest package among the multiple packages within the field of view of the target camera at the first moment. Therefore, the initial heights corresponding to the other cameras are also the heights of the highest package among the multiple packages within the field of view of the other cameras at the first moment, namely h1, h2, h3, etc. These initial heights can also be obtained using a grating.

[0087] To transform initial images with different depths of field into a first image with a unified standard, it is necessary to determine the positions of pixels in the initial image within the first image. Once the position of each pixel in the initial image is determined within the first image, the first image corresponding to the normalized initial image can be determined. Therefore, each initial image can be substituted into a preset calculation formula to obtain the coordinates of each pixel in the initial image corresponding to the pixel in the first image.

[0088] First, the real physical coordinates of the package under the camera's field of view need to be converted into image coordinates. The image coordinates are The preset formula can be: in, is the coordinate of the package in the initial image in the real world, and the calculated are the coordinates of the wrapping in the original image. M is the intrinsic parameter matrix of the camera device. The intrinsic parameters of the camera device include parameters such as the focal length of the camera device and the unit pixel size of the sensor in the camera device. These intrinsic parameters vary from camera to camera. However, in the embodiment of the present invention, to achieve the best transformation effect, the intrinsic parameters are set to the same across different cameras.

[0089] R is the rotation matrix of the camera device, and t is the translation matrix of the camera device. The rotation matrix and the translation matrix can be obtained through calculation. The specific calculation process can refer to the existing technology and is not limited at this time.

[0090] In order to obtain the best decoding effect and to obtain the first image with uniform depth of field, the S in the formula needs to be corrected in real time. In the embodiment of the present invention, the grating provided on the transmission device is used to obtain the initial heights of the multiple camera devices except the target camera device. The ratios of the initial heights obtained in real time to the standard heights h1, h2, h3... and the standard height H (h1 / H, h2 / H, h3 / H...) are used to obtain new proportional coefficients S1, S2, S3... The proportional coefficients are then substituted into the preset calculation formula to obtain a new image transformation formula, that is, the transformation formula between the initial image and the first image can be obtained, which is:

[0091]

[0092] Among them, since different camera devices correspond to different fields of view, that is, different camera devices correspond to different collection areas, the coordinates of the corner points corresponding to each collection area can be known. Therefore, using this formula, the first image corresponding to each initial image can be obtained.

[0093] In the embodiment of the present invention, the specific process of normalizing images with different depths of field to obtain a first image with a uniform depth of field may refer to the prior art and is not limited here.

[0094] In an embodiment of the present invention, after normalizing all initial images to obtain a plurality of first images at a uniform scale, it is necessary to splice and fuse the plurality of first images to obtain an output image.

[0095] Specifically, such as Figure 3 FIG. 1 is a flow chart of an embodiment of obtaining an output image provided by the present invention, wherein stitching and fusing multiple first images to obtain an output image may include:

[0096] 30. Determine the order of the plurality of first images.

[0097] 31. Determine ideal areas corresponding to the multiple camera devices, and crop the multiple first images according to the ideal areas to obtain multiple second images.

[0098] 32. Splice and fuse the multiple second images to obtain an output image.

[0099] Specifically, because the control device acquires first images transmitted by multiple cameras, the control device needs to sort the multiple first images to facilitate subsequent stitching and fusion of the first images. Preferably, the multiple cameras can be sorted based on the transmission direction of the transmission device to facilitate sorting of different initial images captured by different cameras.

[0100] After determining the order of the first images, the first images need to be cropped. Figure 4 The figure shows a side view of an embodiment of the collection system provided by the present invention. Since the actual shooting field of view of the camera device is larger than the shooting field of view of the camera device under the ideal state, that is, there will be overlap between the initial images captured by different camera devices, it is necessary to determine the ideal shooting areas corresponding to the different camera devices under the ideal state, that is, the ideal shooting areas in which there is no overlap between the fields of view of different camera devices; in this way, there will be no overlap between the initial images captured by different camera devices. The first image is then cropped according to the ideal area to obtain second images corresponding to the different camera devices under the ideal state, and there are multiple second images.

[0101] In the above embodiment, the sizes of the ideal imaging areas corresponding to different imaging devices under ideal conditions can be set according to actual conditions. Moreover, since different imaging devices are at different heights from the conveying device, the sizes of the ideal imaging areas corresponding to different imaging devices are also different.

[0102] In the above embodiment, after obtaining multiple second images corresponding to multiple camera devices under ideal conditions, the multiple second images need to be spliced ​​and fused according to the confirmed image order to obtain a final output image, and the staff can locate the unrecognized package based on the final output image.

[0103] In an embodiment of the present invention, stitching and fusing multiple second images to obtain an output image may include: performing smooth gradient processing on edges of the multiple second images to obtain multiple third images; and stitching the multiple third images to obtain the output image.

[0104] In an embodiment of the present invention, due to factors such as exposure compensation and different light intensities, the second images will have varying degrees of brightness differences, and directly stitching together multiple second images will result in noticeable gaps. Therefore, it is necessary to perform smooth gradient processing on the edges of the multiple second images so that when stitching together multiple third images, the image transitions at the stitching locations of different third images are smoother, ensuring that the transitions at the stitching locations of the stitched output image are smoother and that the clarity of the output image at the transition locations is maintained. The specific smooth gradient processing and the process of stitching together multiple third images can be referred to in the prior art and are not limited here.

[0105] In the embodiment of the present invention, the staff needs to check the output image to determine which packages are not recognized. However, the staff only needs to check the decoding results on the output image to directly determine which packages are not recognized.

[0106] When the output image is generated, it includes both identified and unidentified packages. Therefore, the collection system can directly read the output image and locate the unidentified package based on the output image, issuing a warning. Alternatively, after locating the unidentified package, it can directly select the unidentified package using a mechanical device. The output image can also be output to the collection system's display interface for easy viewing by staff, allowing them to quickly locate the unidentified package and manually identify it.

[0107] Specifically, in some embodiments of the present invention, the control device includes a main control device and multiple sub-control devices, and the multiple sub-control devices are provided in a one-to-one correspondence with the multiple camera devices. The sub-control devices are used to decode the multiple initial images, obtain multiple first decoding results, and display the decoding results on the initial image captured by the camera device. Because the package barcode used for identification on each package is different, after the camera device captures the initial image, the sub-control device decodes and identifies the different package barcodes. If the decoding and identification is successful, the sub-control device marks the image area corresponding to the identified package. If the corresponding image area of ​​the package is marked, it indicates that the package has been successfully identified. If the mark is not present, it indicates that the package has not been identified. The staff directly determines which packages have not been identified based on the presence of the mark in the corresponding image area of ​​the package, and then locates the unidentified packages.

[0108] In the above embodiment, the multiple sub-control devices each decode and identify the initial image captured by their corresponding camera device to obtain multiple first decoding results. The multiple first decoding results are then marked on the initial image. The sub-control devices then transmit the initial image including the first decoding results to the main control device. The first decoding results are obtained by decoding and identifying the initial image, and the multiple first decoding results correspond one-to-one to the multiple initial images.

[0109] Specifically, different packages can each correspond to a barcode or QR code. The sub-control device can decode and identify these barcodes or QR codes to obtain a first decoding result, and then determine whether any packages are unrecognized. The process of decoding and identifying barcodes and QR codes can refer to existing technologies and is not limited here.

[0110] After the main control device acquires multiple initial images including the first decoding results, the main control device normalizes the multiple initial images to obtain multiple first images, and then splices and fuses the multiple first images to obtain an output image, wherein the output image includes the multiple first decoding results. Determining whether a package is unrecognized based on the output image may include: identifying the multiple first decoding results in the output image by the main control device to determine whether a package is unrecognized.

[0111] In other embodiments of the present invention, the control device includes only a main control device, and determining whether there is an unrecognized package on the transport surface based on the output image may include: decoding the output image to obtain a second decoding result; and determining whether there is an unrecognized package based on the second decoding result.

[0112] Specifically, the main control device obtains multiple initial images and obtains output images based on the multiple initial images. In this embodiment, the multiple initial images do not include their respective decoding results; instead, after obtaining the output image, the main control device decodes the output image to obtain a second decoding result, and displays the second decoding result in the form of a mark on the output image. The staff determines whether there is a package that has not been identified based on the output image, wherein the second decoding result is obtained by decoding and identifying the output image.

[0113] In an embodiment of the present invention, determining whether there are any unrecognized packages on the transport surface based on the output image may include: in the output image, determining that packages displaying identification marks are recognized packages, and determining that packages not displaying identification marks are unrecognized packages.

[0114] Specifically, the decoding results are displayed in the output image as a marker. If a package is recognized, a marker will be placed in the image area corresponding to the recognized package. In other words, if a marker is placed around the package, it means that the package has been recognized and successfully collected. If a package is not recognized, no marker will be placed around it, indicating that the package has not been successfully collected.

[0115] In an embodiment of the present invention, the staff can directly determine whether there is a package that has not been recognized by checking the output image. Since the output image is obtained by splicing and fusion of initial images taken by multiple cameras in a certain order, after determining that a package has not been recognized, the staff does not need to check all cameras. They can directly locate the camera device corresponding to the unrecognized package based on the output image, and then determine the unrecognized package, and manually collect the unrecognized package separately, saving manpower and material resources.

[0116] It should be noted that, in the embodiment of the present invention, the processes of normalizing the initial image to obtain the first image and stitching and fusing the multiple first images to obtain the output image are all performed in the main control device.

[0117] The present invention also provides a multi-image matching and fusion device, which includes:

[0118] a camera module, the camera module being used to acquire an initial image of the transport surface within a field of view of a camera device;

[0119] A normalization module, configured to perform normalization processing on the initial image to obtain a first image;

[0120] A processing module, configured to perform splicing and fusion of the first images to obtain an output image;

[0121] A judgment module is configured to judge whether there is an unidentified package on the transport surface based on the output image.

[0122] The multi-image matching and fusion device provided by the embodiment of the present invention obtains multiple initial images at different heights by setting multiple cameras at different heights, selects any one of the initial images as a standard image, and obtains the height of the camera corresponding to the standard image. The multiple initial images are normalized based on the standard image to obtain multiple first images, and the multiple first images are spliced ​​and fused to obtain a final output image. The final output image includes the results of identifying multiple packages, which facilitates staff to quickly locate unidentified packages. At the same time, it solves the problem of the image splicing method under the existing technology that complex image features are required to accurately identify images.

[0123] Specifically, in an embodiment of the present invention, the normalization module can be used to: select any one of multiple camera devices as a target camera device; determine the standard height of the image based on the target camera device; obtain a standard image pre-taken in the target camera device; and normalize multiple initial images based on the standard height and the standard image to obtain multiple first images.

[0124] Among them, the normalization module determines the standard height of the image based on the target camera device, which may include: obtaining the height of the highest package among multiple packages on the transport surface within the field of view of the target camera device at the first moment, and taking the height of the highest package as the standard height.

[0125] The normalization module performs normalization processing on the multiple initial images according to the standard height and the standard image to obtain multiple first images, which may include:

[0126] Obtain the initial heights corresponding to the multiple camera devices except the target camera device among the multiple camera devices, where the initial heights are multiple; calculate the height transformation relationship based on the initial heights and the standard heights; perform image transformation on the multiple initial images based on the height transformation relationship, the standard image and the multiple initial images to obtain multiple first images.

[0127] In the above embodiment, the normalization module obtains the initial heights corresponding to the other camera devices except the target camera device among the multiple camera devices, which may include:

[0128] Obtain the highest package height among multiple packages on the transport surface within the respective corresponding fields of view of multiple camera devices except the target camera device at the first moment, and use the height of the highest package as the initial height corresponding to the other camera devices.

[0129] In some other embodiments of the present invention, the processing module is configured to stitch and fuse the first images to obtain an output image. Specifically, the processing module stitches and fuses the multiple first images to obtain the output image, which may include:

[0130] Determine the order of the multiple first images; determine the ideal areas corresponding to the multiple camera devices, and crop the multiple first images according to the ideal areas to obtain multiple second images; and splice and fuse the multiple second images to obtain an output image.

[0131] In the above embodiment, the processing module stitches and fuses the multiple second images to obtain the output image, which may include: performing smooth gradient processing on the edges of the multiple second images to obtain multiple third images; and stitching the multiple third images to obtain the output image.

[0132] In an embodiment of the present invention, the output image includes multiple first decoding results, and based on the decoding results, it can be determined whether any package is unrecognized. Specifically, the determination module determines whether any package is unrecognized on the transport surface based on the output image, which may include: using a main control device to identify the multiple first decoding results in the output image and determine whether any package is unrecognized.

[0133] Specifically, the judgment module judges whether there is an unrecognized package on the transport surface based on the output image, which may include: decoding the output image to obtain a second decoding result; and judging whether there is an unrecognized package based on the second decoding result.

[0134] In an embodiment of the present invention, unrecognized packages will be marked with identification marks. Based on the presence or absence of the identification marks, operators can more intuitively distinguish between recognized and unrecognized packages.

[0135] The present application also provides a logistics collection system, which includes a control device, multiple camera devices, a conveying device, and a storage medium. The multiple camera devices are arranged above the conveying device, and the multiple camera devices are at different heights from the conveying device. The conveying device has a transport surface for conveying multiple packages, and the control device is used to:

[0136] Acquiring multiple initial images of the transport surface within the fields of view of each of the plurality of camera devices, wherein the number of initial images is multiple;

[0137] performing normalization processing on the multiple initial images to obtain multiple first images;

[0138] performing splicing and fusing of the multiple first images to obtain an output image;

[0139] Based on the output image, determine whether there are any unrecognized packages on the transport surface.

[0140] The present application also provides a server, which is located in a host monitoring system, and the host monitoring system also includes multiple hosts connected to the server network. The server may include:

[0141] one or more processors;

[0142] Memory; and

[0143] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the aforementioned host monitoring method.

[0144] The present application provides a server that integrates any system monitoring device provided in the embodiments of the present application, such as Figure 5 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present invention, specifically:

[0145] The server may include one or more processing core processors 501, one or more computer-readable storage media memories 502, a power supply 503, an input unit 504 and other components. Those skilled in the art will appreciate that Figure 5 The server structure shown in the figure does not constitute a limitation to the server, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0146] in:

[0147] The processor 501 is the control center of the server, connecting the various parts of the entire server using various interfaces and lines, and executing the various functions of the server and processing data by running or executing the software programs and / or modules stored in the memory 502, and calling the data stored in the memory 502, thereby monitoring the server as a whole. Optionally, the processor 501 may include one or more processing cores; the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the volume measurement device for logistics parts, connecting the various parts of the entire device using various interfaces and lines. Preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0148] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0149] The server also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0150] The server may further include an input unit 504, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0151] Although not shown, the server may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the server will load the executable files corresponding to one or more application processes into the memory 502 according to the following instructions, and the processor 501 will run the application stored in the memory 502 to implement various functions as follows:

[0152] Acquire multiple initial images of the transport surface within the fields of view of the multiple camera devices, wherein the number of initial images is multiple.

[0153] Normalization is performed on the multiple initial images to obtain multiple first images.

[0154] The multiple first images are stitched and fused to obtain an output image.

[0155] Based on the output image, determine whether there are any unrecognized packages on the transport surface.

[0156] The present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk. The storage medium stores a computer program, which is loaded by a processor to execute the steps of any host monitoring method provided in the embodiments of the present application. For example, the computer program loaded by the processor may execute the following steps:

[0157] Acquire multiple initial images of the transport surface within the fields of view of the multiple camera devices, wherein the number of initial images is multiple.

[0158] Normalization is performed on the multiple initial images to obtain multiple first images.

[0159] The multiple first images are stitched and fused to obtain an output image.

[0160] Based on the output image, determine whether there are any unrecognized packages on the transport surface.

[0161] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above and will not be repeated here.

[0162] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to implement as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments and will not be repeated here.

[0163] The above is a detailed introduction to a host monitoring method, device, server and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and embodiments of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific embodiments and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-image matching and fusion method, characterized in that: Applied to a control device, the control device is located in a logistics collection system, the logistics collection system further includes a plurality of cameras and a conveying device, the plurality of cameras are arranged along the conveying direction of the conveying device and at different intervals, the cameras are arranged above the conveying device, the plurality of cameras are at different heights from the conveying device, the plurality of cameras are identical so that the resolution of images captured by the plurality of cameras is the same, the conveying device has a transport surface for conveying a plurality of packages, and when a package is placed on the conveying device, the conveying device conveys the package at a fixed period; The method comprises: After one cycle is completed and all packages are exposed to the field of view of the camera device, a plurality of initial images of the transport surface within the field of view of each of the camera devices are acquired, wherein the initial images are multiple; Selecting any one of the multiple camera devices as a target camera device; Obtaining the height of a highest package among a plurality of packages on the transport surface within the field of view of the target camera device at a first moment, and using the height of the highest package as a standard height; Acquiring a standard image pre-captured by the target camera device; Obtaining the highest height of the multiple packages on the transport surface within the respective corresponding fields of view of the multiple camera devices, excluding the target camera device, at a first moment, and using the height of the highest package as the initial height corresponding to the other camera devices; Calculating a height conversion relationship according to the initial height and the standard height; performing image transformation on the plurality of initial images according to the height transformation relationship, the standard image, and the plurality of initial images to obtain a plurality of first images, so as to transform the initial images at different depths of field into the first images at a uniform depth of field, and to perform normalization processing on the plurality of initial images; sorting the plurality of camera devices based on a transmission direction of the transmission device to determine an order of the plurality of first images captured by the plurality of camera devices; determining an ideal area corresponding to the plurality of camera devices, and cropping the plurality of first images according to the ideal area to obtain a plurality of second images; performing splicing and fusing of the plurality of second images to obtain an output image; Based on the output image, it is determined whether there are any unrecognized packages on the transport surface, and the unrecognized packages are then located.

2. The multi-image matching and fusion method according to claim 1, characterized in that: The step of stitching and fusing the plurality of second images to obtain an output image includes: performing smooth gradient processing on edges of the plurality of second images to obtain a plurality of third images; A plurality of the third images are stitched together to obtain the output image.

3. The multi-image matching and fusion method according to claim 1, characterized in that: The control device includes a main control device and multiple sub-control devices, and the multiple sub-control devices are provided in a one-to-one correspondence with the multiple camera devices. After obtaining initial images of the transport surface within the fields of view of each of the multiple camera devices, where the initial images are multiple, the method further includes: The sub-control device decodes the multiple initial images respectively to obtain multiple first decoding results, where the multiple initial images include the first decoding results of the camera device identifying the packages within the respective fields of view.

4. The multi-image matching and fusion method according to claim 3, characterized in that: The output image includes a plurality of first decoding results; and determining, based on the output image, whether there is an unidentified package on the transport surface includes: The main control device identifies a plurality of first decoding results in the output image to determine whether there is a package that has not been identified.

5. The multi-image matching and fusion method according to claim 1, characterized in that: The determining, based on the output image, whether there is an unrecognized package on the transport surface includes: Decoding the output image to obtain a second decoding result; According to the second decoding result, it is determined whether there is a package that has not been identified.

6. The multi-image matching and fusion method according to claim 1, characterized in that: The determining, based on the output image, whether there is an unrecognized package on the transport surface includes: In the output image, a package showing an identification mark is determined to be a recognized package, and a package not showing an identification mark is determined to be an unrecognized package.

7. A multi-image matching and fusion device, characterized in that: Applied to a control device, the control device is located in a logistics collection system, the logistics collection system also includes multiple cameras and a conveying device, the multiple cameras are arranged along the conveying direction of the transmission device and have different spacings, the cameras are arranged above the conveying device, the multiple cameras are at different heights from the conveying device, the multiple cameras are identical so that the resolution of the images captured by the multiple cameras is the same, the conveying device has a transport surface for conveying multiple packages, when the package is placed on the conveying device, the conveying device conveys it at a fixed period, the device includes: a camera module configured to capture, after a cycle has ended and all packages are exposed to the camera's field of view, a plurality of initial images of the transport surface within the camera's field of view; A normalization module, the normalization module is used to select any one of the multiple camera devices as a target camera device; obtain the height of the highest package among the multiple packages on the transport surface within the field of view of the target camera device at a first moment, and use the height of the highest package as a standard height; obtain a standard image pre-captured by the target camera device; obtain the height of the highest package among the multiple packages on the transport surface within the respective corresponding fields of view of the multiple camera devices other than the target camera device at the first moment, and use the height of the highest package as the initial height corresponding to the other camera devices; calculate a height transformation relationship based on the initial height and the standard height; perform image transformation on the multiple initial images based on the height transformation relationship, the standard image and the multiple initial images, to obtain multiple first images, so as to transform the initial images with different depths of field into the first image with a uniform depth of field, and to perform normalization processing on the multiple initial images; a processing module, the processing module being configured to sort the plurality of camera devices based on a transmission direction of the transmission device to determine an order of the plurality of first images captured by the plurality of camera devices; determine an ideal area corresponding to the plurality of camera devices, and crop the plurality of first images according to the ideal area to obtain a plurality of second images; and splice and fuse the plurality of second images to obtain an output image; A judgment module is used to judge whether there is an unrecognized package on the transport surface based on the output image, and then locate the unrecognized package.

8. A storage medium, characterized in that: The storage medium stores a computer program, which includes steps for implementing the multi-image matching and fusion method according to any one of claims 1 to 6 when operated by the multi-image matching and fusion device according to claim 7.

9. A logistics collection system, characterized in that: The logistics collection system includes a control device, multiple camera devices, a conveying device, and a storage medium. The multiple camera devices are arranged along the conveying direction of the transmission device and at different intervals. The camera devices are arranged above the conveying device and at different heights from the conveying device. The multiple camera devices are identical so that the resolution of images captured by the multiple camera devices is the same. The conveying device has a transport surface for conveying multiple packages. When a package is placed on the conveying device, the conveying device conveys it at a fixed period. The control device is used to: After one cycle is completed and all packages are exposed to the field of view of the camera device, a plurality of initial images of the transport surface within the field of view of each of the camera devices are acquired, wherein the initial images are multiple; Selecting any one of the multiple camera devices as a target camera device; Obtaining the height of a highest package among a plurality of packages on the transport surface within the field of view of the target camera device at a first moment, and using the height of the highest package as a standard height; Acquiring a standard image pre-captured by the target camera device; Obtaining the highest height of the multiple packages on the transport surface within the respective corresponding fields of view of the multiple camera devices, excluding the target camera device, at a first moment, and using the height of the highest package as the initial height corresponding to the other camera devices; Calculating a height conversion relationship according to the initial height and the standard height; performing image transformation on the plurality of initial images according to the height transformation relationship, the standard image, and the plurality of initial images to obtain a plurality of first images, so as to transform the initial images at different depths of field into the first images at a uniform depth of field, and to perform normalization processing on the plurality of initial images; sorting the plurality of camera devices based on a transmission direction of the transmission device to determine an order of the plurality of first images captured by the plurality of camera devices; determining an ideal area corresponding to the plurality of camera devices, and cropping the plurality of first images according to the ideal area to obtain a plurality of second images; performing splicing and fusing of the plurality of second images to obtain an output image; Based on the output image, it is determined whether there are any unrecognized packages on the transport surface, and the unrecognized packages are then located.

Images