A gantry crane adaptive image stitching method and system

Abstract

The application relates to a gantry crane adaptive image splicing method and system. The method comprises the following steps: in response to acquired images in a coverage range, judging the state of a container object in the images, the state comprising a moving state and a placed state; when the state of the container object is the placed state, acquiring height information of the container object; calculating splicing parameters according to the height information of the container object, the splicing parameters comprising first edge splicing parameters and second edge splicing parameters; and splicing multiple images by using the splicing parameters, the generation time of each image being the same, and each image acquisition unit generating only one image at a time point. The gantry crane adaptive image splicing method and system disclosed by the application realize complete image splicing of containers with different heights by adaptively adjusting splicing parameters, avoid image loss and distortion, and provide reliable image support for safe operation of the gantry crane.

Description

Technical Field

[0001] This application relates to the field of image stitching processing technology, and in particular to an adaptive image stitching method and system for gantry cranes. Background Technology

[0002] With the growth of global trade, container terminals play a crucial role in modern logistics. Gantry cranes, as one of the terminal's loading and unloading equipment, require precise operation and efficient monitoring systems to ensure the safety and efficiency of operations when handling and stacking containers. However, with the continuous stacking of containers, traditional monitoring systems face numerous challenges.

[0003] In existing technologies, gantry crane monitoring systems typically achieve real-time monitoring of the work area by installing cameras on the spreading equipment or the gantry crane structure. However, the images from these cameras need to be stitched together to form a complete panoramic view so that operators can view the entire work area in real time.

[0004] Traditional image stitching techniques typically operate on a fixed plane, stitching together camera images from a ground-based perspective. However, this method has significant drawbacks when dealing with scenarios involving varying container stacking heights. The main reason is its inability to adapt to over-cropping and information loss caused by height changes, which directly impacts monitoring effectiveness and the safety of the hoisting process. Summary of the Invention

[0005] This application provides an adaptive image stitching method and system for gantry cranes. By adaptively adjusting the stitching parameters, it achieves complete image stitching of containers of different heights, avoiding image loss and distortion, and providing reliable image support for the safe operation of gantry cranes.

[0006] The above-mentioned objective of this application is achieved through the following technical solution:

[0007] Firstly, this application provides an adaptive image stitching method for gantry cranes, including:

[0008] In response to the acquired images within the coverage area, determine the state of the container objects in the images, including their movement state and their placement state;

[0009] When the container object is in the placement state, obtain the height information of the container object;

[0010] The splicing parameters are calculated based on the height information of the container object. The splicing parameters include the first edge splicing parameters and the second edge splicing parameters.

[0011] Use stitching parameters to stitch together multiple images;

[0012] Each image participating in the splicing is from a different image acquisition unit and has the same generation time.

[0013] Each image acquisition unit generates only one image at a time point.

[0014] In a possible implementation of the first aspect, the method further includes fusing the overlapping area of the two images, and the fusing includes brightness fusion, chromatic aberration fusion, color balance, and brightness adjustment.

[0015] In a possible implementation of the first aspect, the splicing of the multiple images using the splicing parameters includes:

[0016] Geometric correction of the same container object in the two adjacent images using the splicing parameters;

[0017] Clipping of the two adjacent images according to the geometric correction result;

[0018] Splicing of the two adjacent images after the clipping to obtain a spliced image;

[0019] Adjustment of the spliced image according to the fusion degree at the splicing position to make the splicing baseline of the spliced image smoothly transition.

[0020] In a possible implementation of the first aspect, the adjustment of the spliced image according to the fusion degree at the splicing position to make the splicing baseline of the spliced image smoothly transition includes:

[0021] Determination of the splicing position on the two adjacent images and creation of a splicing curve at the splicing position;

[0022] Establishment of a reference baseline at the splicing reference position, the reference baseline being perpendicular to the splicing baseline;

[0023] Establishment of an adjustment curve using the pixel points on the reference baseline;

[0024] Clipping of the adjustment curve using the splicing baseline to obtain a clipped adjustment curve;

[0025] Adjustment of the splicing reference position according to the relative positions of the two clipped adjustment curves.

[0026] In a possible implementation of the first aspect, the adjustment of the splicing reference position according to the relative positions of the two clipped adjustment curves includes:

[0027] The two adjustment curves are denoted as a first adjustment curve and a second adjustment curve, respectively;

[0028] The first clipped adjustment curve is obtained on the first adjustment curve, and the second clipped adjustment curve is obtained on the second adjustment curve;

[0029] transferring the first cutting adjustment curve to the second adjustment curve and moving to the coinciding position and calculating a first relative distance between the coinciding position and the stitching baseline;

[0030] transferring the second cutting adjustment curve to the first adjustment curve and moving to the coinciding position and calculating a second relative distance between the coinciding position and the stitching baseline;

[0031] calculating a relative distance average of the first relative distance and the second relative distance;

[0032] calculating a distribution of the relative distance average on the stitching baseline and adjusting the stitching reference position according to the distribution of the relative distance average on the stitching baseline.

[0033] In a possible implementation of the first aspect, when the definition of the two adjacent images is inconsistent, the definition of the image with higher definition is adjusted to be the same as the definition of the image with lower definition.

[0034] In a possible implementation of the first aspect, when transferring the first cutting adjustment curve to the second adjustment curve and moving to the coinciding position, the moving includes horizontal moving and vertical moving.

[0035] Further comprising determining the feature points using a wavelet decomposition method and coinciding the feature points on the first cutting adjustment curve and the feature points on the second adjustment curve.

[0036] In the second aspect, the application provides a gantry crane adaptive image stitching device, comprising:

[0037] a state determining unit configured to determine the state of the container object in the image in response to the acquired image in the coverage range, the state including a moving state and a placed state;

[0038] a first data processing unit configured to acquire height information of the container object when the state of the container object is the placed state;

[0039] a second data processing unit configured to calculate stitching parameters according to the height information of the container object, the stitching parameters including a first edge stitching parameter and a second edge stitching parameter;

[0040] a stitching processing unit configured to stitch the multiple images using the stitching parameters;

[0041] wherein each image participating in the stitching is from a different image acquisition unit and the generation time of each image is the same;

[0042] Each image acquisition unit generates only one image at a time point.

[0043] In the third aspect, the application provides a gantry crane adaptive image stitching system, the system comprising:

[0044] one or more memories storing instructions; and

[0045] one or more processors for invoking and running the instructions from the memories, implementing the methods as described in the first aspect and any possible implementation of the first aspect.

[0046] In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium includes:

[0047] a program which, when run by a processor, causes the method as described in the first aspect and any possible implementation of the first aspect to be performed.

[0048] In a fifth aspect, a computer program product is provided, including program instructions, when the program instructions are run by a computing device, the method as described in the first aspect and any possible implementation of the first aspect is performed.

[0049] In a sixth aspect, a chip system is provided, the chip system includes a processor for implementing the functions involved in the above aspects, for example, generating, receiving, sending, or processing the data and / or information involved in the above methods.

[0050] The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0051] In a possible design, the chip system further includes a memory, the memory is configured to store necessary program instructions and data. The processor and the memory can be decoupled, and arranged on different devices, connected through a wired or wireless manner, or the processor and the memory can be coupled on the same device. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a schematic block diagram of a step flow of a gantry crane adaptive image stitching method provided by the present application.

[0053] Figure 2 is a schematic diagram of the position deployment of a camera on a gantry crane provided by the present application.

[0054] Figure 3 is a schematic diagram of an overlapping area between cameras provided by the present application.

[0055] Figure 4 is a corresponding relationship diagram of a stitching parameter P and a real stitching parameter P calculated by the present application.

[0056] Figure 5 is a schematic diagram of a stitching baseline provided by the present application.

[0057] Figure 6 is a schematic diagram of a reference baseline provided by the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the present application are further described in detail below with reference to the drawings.

[0059] The present application discloses a gantry crane adaptive image stitching method, please refer to Figure 1 In some examples, the gantry crane adaptive image stitching method disclosed by the present application includes the following steps:

[0060] S101, in response to the acquired image within the coverage range, judging the state of the container object in the image, the state including moving state and placing state;

[0061] S102, when the state of the container object is the placing state, acquiring the height information of the container object;

[0062] S103, calculating the stitching parameters according to the height information of the container object, the stitching parameters including the first edge stitching parameter and the second edge stitching parameter;

[0063] S104, using the stitching parameters to stitch multiple images;

[0064] Among them, each image participating in stitching comes from different image acquisition units and the generation time of each image is the same;

[0065] Each image acquisition unit generates only one image at a time point.

[0066] Specifically, in the technical solutions disclosed by the present application, please refer to Figure 2 , it is necessary to install multiple cameras on the gantry crane, and each camera is responsible for a region, and there is overlap between these regions, as Figure 3 indicated, the images collected by the cameras are generated into panoramic images after stitching processing.

[0067] Please refer to Figure 3 , in the prior art,

[0068] In step S101, first, the state of the container object in the image is judged, and the state has two kinds, which are moving state and placing state. The moving state means that the container object is in the moving state, including horizontal direction moving and vertical direction moving. The placing state means that the container object is placed on the ground or other container objects.

[0069] The determination manner of the state can be determined according to the gantry crane, for example, there is only one gantry crane in a region, at this time, two images are obtained at two different time points respectively, and it can be determined through image comparison which container objects are in a moving state and which container objects are in a placing state.

[0070] When the state of the container object is the placing state, height information of the container object is obtained, and the height information is obtained by the gantry crane. The gantry crane generates a work record in the working process, and the movement in the work record can be used as a coordinate position. The work record of the lifting tool can be used as the height information of the container object.

[0071] In some possible implementation manners, the height information of the container object is obtained by means of a distance sensor.

[0072] In step S103, splicing parameters are calculated according to the height information of the container object, and the splicing parameters include first edge splicing parameters and second edge splicing parameters, and specific contents are as follows:

[0073] Through analysis of a plurality of groups of height H and splicing parameter P data, it is found that there is a nonlinear relationship between them, and therefore a quadratic equation is used for fitting. The splicing parameter P is set as follows:

[0074]

[0075] The splicing parameter formula of the first angle and the second angle is obtained through data fitting:

[0076] The first edge splicing parameter is: ;

[0077] The second edge splicing parameter is: ;

[0078] Wherein, H represents the current lifting tool height (unit: meter), and P represents the corresponding splicing parameter value.

[0079] When the lifting tool is at a certain specific height , the is substituted into the above splicing parameter formula, and the splicing parameter at the current height is calculated:

[0080] ;

[0081] .

[0082] In step S104, the splicing parameters are used to splice a plurality of images. It is required that each image participating in splicing is from a different image acquisition unit, the generation time of each image is the same, and each image acquisition unit generates only one image at a time point.

[0083] In the stitching process, the overlapping area of the two images is also fused, including brightness fusion, color difference fusion, color balance and brightness adjustment.

[0084] The related content of geometric transformation and correction is as follows:

[0085] Suppose the image coordinates captured by each camera are , the image is geometrically transformed, and the new coordinates after transformation are denoted as .

[0086] The transformation process needs to be translated, rotated and scaled. The specific geometric transformation formula is as follows:

[0087] ;

[0088]

[0089] wherein , , , are the rotation and scaling coefficients, , are the translation coefficients. These coefficients are determined by the stitching parameters and , and are automatically calculated by the stitching module.

[0090] The related content of the image overlapping area is as follows:

[0091] There is an overlapping area between the fields of view of each camera to ensure the continuity of the stitching. Suppose the image pixel value of the overlapping area is , the system uses a weighted average method to fuse the overlapping area to eliminate the brightness and color difference of different camera images. The formula for weighted average is:

[0092]

[0093] wherein and are the pixel values of adjacent camera images in the overlapping area, is the weight coefficient, which is adjusted according to the degree of image overlap.

[0094] The related content of the image stitching sequence is as follows:

[0095] The image stitching is performed in the order of the spatial position of the camera, for example, from left to right, from top to bottom. Suppose the pixel coordinates of the th camera image after transformation are The panoramic image coordinates after splicing are:

[0096]

[0097] wherein, and are the offset amounts of each camera image in the panoramic image, and the offset amounts are dynamically calculated according to the installation positions of each camera and the splicing parameters.

[0098] The related content of color balance and brightness adjustment is as follows:

[0099] In order to ensure that there is no color difference after splicing the images of different cameras, the system performs color balance and brightness adjustment on each image block. Assuming that the original pixel brightness is , the adjusted brightness is , and the brightness adjustment formula is:

[0100] wherein, is the brightness adjustment coefficient, which is dynamically adjusted according to the light conditions of different cameras to ensure the consistency of the spliced image.

[0101] For example:

[0102] Eight cameras are installed on the girder of the gantry crane , which covers all angles of the work area and sets a certain overlap area to achieve seamless splicing. The system is configured with a height sensor to monitor the lifting height of the spreader in real time (denoted as ), and sends the height information to the splicing module.

[0103] 1. Data acquisition and splicing parameter fitting

[0104] In the preparation stage, the spreader collects images from a height of 2 meters to 14 meters (at intervals of 2 meters). At each height, the splicing parameters of each camera field of view are recorded to establish the corresponding relationship between the height and the splicing parameters.

[0105] Through data analysis, the fitting formula of the first angle and the second edge splicing parameters is obtained:

[0106] The first edge splicing parameter formula is: ;

[0107] The second edge splicing parameter formula is: .

[0108] The formula ( ) is obtained through the content shown in Figure 4 .

[0109] The first edge stitching parameters correspond to the upper left corner and the lower left corner of the container, and the second edge stitching parameters correspond to the upper right corner and the lower right corner of the container. Through the stitching parameters, the parts of a container in two images can be corresponded, and then the picture cutting and splicing are performed.

[0110] 2. Real-time splicing process

[0111] Suppose the current spreader is handling a stack of containers with a height of 10 meters, and the system receives the current height information meters, and the following steps are performed:

[0112] (1) Splicing parameter calculation

[0113] Substitute the into the splicing parameter formula to obtain:

[0114] Upper left corner splicing parameter: ;

[0115] Upper right corner splicing parameter: ;

[0116] (2) Geometric transformation and image correction

[0117] The calculated splicing parameters and will serve as the basis for image geometric correction. Assuming that the original image coordinates are , the transformed coordinates are:

[0118] ;

[0119] ;

[0120] Among them, the transformation coefficients , , , , , are calculated from the splicing parameters and . The specific calculation is as follows:

[0121] Because and are height-related parameters, they represent the size change of the image or the width of the field of view, so we can derive the scaling coefficients by calculating and under the current height.

[0122] Suppose we have a reference height and the corresponding splicing parameters and and the current height and the corresponding stitching parameters and .

[0123] Step 1: Calculate the width change ratio (i.e., scaling factor)

[0124] Reference width:

[0125] Current width:

[0126] Scaling factor:

[0127] Step 2: Calculate the rotation angle

[0128] The rotation factor reflects the image rotation caused by the height change. and The change in height not only affects the width but also the viewing angle of the image (e.g., rotation angle). We need to calculate the rotation angle from the height change.

[0129] The height change causes the image to rotate, and the rotation angle is related to the change in stitching parameters. Specifically, the rotation angle can be calculated by the following relationship:

[0130] Relationship between rotation angle and stitching parameters:

[0131] where and are empirical constants reflecting the relationship between the change in stitching parameters and the rotation angle.

[0132] Calculate the rotation factor: Once the rotation angle is obtained, the rotation factor can be calculated by the rotation matrix:

[0133] , , ,

[0134] Step 3: Calculate the translation factor

[0135] The translation factor controls the offset of the image in the x and y directions. The translation factor is related to the viewing angle and height change of the image and is usually dynamically adjusted with the height change.

[0136] The translation factor reflects the position adjustment of the image. The translation of the image is determined by the stitching parameters or other factors and can be determined by calculating the relative offset:

[0137] ,

[0138] where:

[0139] , is the relative translation scale factor of the image in x and y direction. is the current stitching parameter.

[0140] Example:

[0141] Reference height , the corresponding stitching parameter is:

[0142] ;

[0143] .

[0144] Current height , the corresponding stitching parameter is:

[0145] ;

[0146] .

[0147] Step 1: Calculate the scale factor

[0148] = 397.98 - 325.42 = 62.56

[0149] = 266.53 - 192.44 = 74.09

[0150]

[0151] Step 2: Calculate the rotation angle

[0152] The relationship between the rotation angle and the change in the stitching parameter is:

[0153]

[0154] Let and :

[0155]

[0156] Once the rotation angle is obtained, the rotation coefficient can be calculated through the rotation matrix:

[0157] ,

[0158] ,

[0159] ,

[0160]

[0161] Step 3: Calculate the translation coefficient

[0162] Assume the image translation scale factor and , then: ,

[0163] By calculation, we get:

[0164] Scaling coefficient Rotation coefficient Translation coefficient ,

[0165] Therefore, the transformation formula is:

[0166] ;

[0167] ;

[0168] 3. Processing of image overlap area

[0169] Assume the pixel value of the overlap area between camera and is , the fusion processing of the overlap area is carried out using the weighted average method:

[0170]

[0171] wherein, and are the image pixel values of camera and respectively, is the weight factor, which is automatically adjusted according to the size of the overlap area.

[0172] 4. Generation of the spliced panoramic image

[0173] The images after geometric correction and overlap area processing are spliced in the order of camera positions to generate a complete panoramic image. The image positions of each camera are adjusted according to the set offset, so that the final spliced image seamlessly connects and forms a complete image of the gantry crane operation area.

[0174] Assume the offset of camera 1 is and the offset of camera 2 is , then:

[0175]

[0176]

[0177]

[0178]

[0179] The adjusted images are spliced in sequence to obtain a complete panoramic image.

[0180] In some examples, the specific way of splicing multiple images using the splicing parameters is as follows:

[0181] S201, geometrically correcting a same container object in the two adjacent images using the splicing parameters;

[0182] S202, cutting the two adjacent images according to the geometric correction result;

[0183] S203, splicing the two adjacent images after cutting to obtain a spliced image;

[0184] S204, adjusting the spliced image according to the fusion degree at the splicing position, so that the splicing baseline of the spliced image is smoothly transitioned.

[0185] The contents in steps S201 to S203 have been introduced in the foregoing, and the general content is that because the positions of the cameras are different, the sizes of the images obtained by each camera are different, so the two images cannot be directly fused, and the images need to be corrected by using the first edge splicing parameter and the second edge splicing parameter.

[0186] However, the first edge splicing parameter and the second edge splicing parameter have certain errors in calculation, which will cause the image at the splicing position to be not naturally transitioned, so the spliced image needs to be adjusted according to the fusion degree at the splicing position, so that the splicing baseline of the spliced image is smoothly transitioned.

[0187] In some examples, the specific way of adjusting the spliced image according to the fusion degree at the splicing position, so that the splicing baseline of the spliced image is smoothly transitioned, is as follows:

[0188] S301, determining a splicing position on the two adjacent images and creating a splicing curve at the splicing position;

[0189] S302, establishing a reference baseline on the splicing reference position, the reference baseline being perpendicular to the splicing baseline;

[0190] S303, establishing an adjustment curve using the pixel points on the reference baseline;

[0191] S304, cutting the adjustment curve using the stitching baseline to obtain a cut adjustment curve;

[0192] S305, adjusting the stitching reference position according to the relative positions of the two cut adjustment curves.

[0193] In steps S301 to S305, first, a reference baseline (as shown in the figure) is established on the stitching baseline (as shown in the figure) at the stitching reference position, the reference baseline is perpendicular to the stitching baseline, then the adjustment curve is established using the pixel points on the reference baseline, and then the adjustment curve is cut using the stitching baseline to obtain the cut adjustment curve. Figure 5 Figure 6 The reference baseline is established on the stitching baseline at the stitching reference position.

[0194] The way of establishing the adjustment curve using the pixel points on the reference baseline is to calculate the difference between adjacent pixel points, then the order corresponding to the difference is taken as the abscissa, and the difference is taken as the ordinate to obtain discrete points, and these discrete points are sequentially connected to obtain the cut adjustment curve.

[0195] Of course, the second difference can also be used to obtain the cut adjustment curve.

[0196] The way of cutting the adjustment curve using the stitching baseline is described by means of Figure 6 The stitching baseline divides the adjustment curve into two segments, and then the used segment is retained as the cut adjustment curve, Figure 6 The adjustment curve on the left side of the middle retains the right half, and the adjustment curve on the right side retains the left half.

[0197] Finally, the stitching reference position is adjusted according to the relative positions of the two cut adjustment curves.

[0198] This way uses multiple points to correct the stitching baseline, and these points are arranged at intervals on the stitching baseline. By adjusting the relative positions of the two cut adjustment curves at these points, the smooth transition of the stitched image at the stitching baseline is realized.

[0199] The specific way of adjusting the stitching reference position according to the relative positions of the two cut adjustment curves is:

[0200] S401, two adjustment curves are recorded as a first adjustment curve and a second adjustment curve;

[0201] S402, a first cut adjustment curve is obtained on the first adjustment curve, and a second cut adjustment curve is obtained on the second adjustment curve;

[0202] S403, the first cut adjustment curve is transferred to the second adjustment curve and moved to the overlapping position, and the first relative distance between the overlapping position and the stitching baseline is calculated;

[0203] ​S404, transfer the second cutting adjustment curve to the first adjustment curve and move to the coincident position and calculate the second relative distance between the coincident position and the stitching baseline;

[0204] S405, calculate the relative distance average of the first relative distance and the second relative distance;

[0205] S406, calculate the distribution of the relative distance average on the stitching baseline and adjust the stitching reference position according to the distribution of the relative distance average on the stitching baseline.

[0206] In steps S401 to S406, first, the first cutting adjustment curve is obtained on the first adjustment curve, and the second cutting adjustment curve is obtained on the second adjustment curve, then the first cutting adjustment curve is transferred to the second adjustment curve and moved to the coincident position and the first relative distance between the coincident position and the stitching baseline is calculated, and at the same time, the second cutting adjustment curve is transferred to the first adjustment curve and moved to the coincident position and the second relative distance between the coincident position and the stitching baseline is calculated.

[0207] The first relative distance and the second relative distance respectively represent the moving distance of the corresponding two images during stitching, and the relative distance average of the first relative distance and the second relative distance is taken as the moving reference value of the two images.

[0208] Finally, the distribution of the relative distance average on the stitching baseline is calculated and the stitching reference position is adjusted according to the distribution of the relative distance average on the stitching baseline, and the stitching baseline at this time is selected from Figure 6 the left stitching baseline or the right stitching baseline, that is, only one image is adjusted during stitching, and the adjustment here is rotation.

[0209] In some possible implementation manners, when the clarity of the two adjacent images is inconsistent, the clarity of the image with higher clarity is adjusted to be the same as the clarity of the image with lower clarity.

[0210] In some possible implementation manners, when the first cutting adjustment curve is transferred to the second adjustment curve and moved to the coincident position, the movement includes horizontal movement and vertical movement.

[0211] When neither the horizontal movement nor the vertical movement can meet the condition of moving to the coincident position, the following method is used for processing:

[0212] The wavelet decomposition method is used to determine the feature points and the feature points on the first cutting adjustment curve and the feature points on the second adjustment curve are coincident.

[0213] The application also provides a gantry crane adaptive image stitching device, comprising:

[0214] A state determining unit is configured to determine a state of the container object in the image, the state including a moving state and a placing state, in response to the acquired image within the coverage range.

[0215] A first data processing unit is configured to acquire height information of the container object when the state of the container object is the placing state.

[0216] A second data processing unit is configured to calculate stitching parameters including first edge stitching parameters and second edge stitching parameters according to the height information of the container object.

[0217] A stitching processing unit is configured to stitch the multiple images using the stitching parameters.

[0218] Each of the images participating in the stitching is acquired by a different image acquisition unit and has a same generation time.

[0219] Each of the image acquisition units generates only one image at a time point.

[0220] Further, the method further includes fusing the overlapping regions of the two images, the fusing including brightness fusion, color difference fusion, color balance, and brightness adjustment.

[0221] Further, the method further includes:

[0222] A geometric correction unit is configured to perform geometric correction on a same container object in the two adjacent images using the stitching parameters.

[0223] A cropping unit is configured to crop the two adjacent images according to the geometric correction result.

[0224] A stitching unit is configured to stitch the two adjacent images after the cropping to obtain a stitched image.

[0225] A stitching adjustment unit is configured to adjust the stitched image according to a fusion degree at a stitching position, so that a stitching baseline of the stitched image is smoothly transitioned.

[0226] Further, the method further includes:

[0227] A stitching curve creating unit is configured to determine a stitching position on the two adjacent images and create a stitching curve at the stitching position.

[0228] A reference baseline creating unit is configured to establish a reference baseline at a stitching reference position, the reference baseline being perpendicular to the stitching baseline.

[0229] An adjustment curve creating unit is configured to establish an adjustment curve using a pixel point on the reference baseline.

[0230] The first cutting adjustment curve creating unit is configured to cut the adjustment curve using the splicing baseline to obtain a cutting adjustment curve.

[0231] The splicing reference position adjusting unit is configured to adjust the splicing reference position according to the relative position of the two cutting adjustment curves.

[0232] Further, the method further comprises:

[0233] The marking unit is configured to mark the two adjustment curves as a first adjustment curve and a second adjustment curve, respectively.

[0234] The second cutting adjustment curve creating unit is configured to obtain a first cutting adjustment curve on the first adjustment curve and a second cutting adjustment curve on the second adjustment curve.

[0235] The coincidence processing unit is configured to transfer the first cutting adjustment curve to the second adjustment curve, move the first cutting adjustment curve to a coincidence position, and calculate a first relative distance between the coincidence position and the splicing baseline.

[0236] The distance calculating unit is configured to transfer the second cutting adjustment curve to the first adjustment curve, move the second cutting adjustment curve to a coincidence position, and calculate a second relative distance between the coincidence position and the splicing baseline.

[0237] The first calculating unit is configured to calculate a relative distance average of the first relative distance and the second relative distance.

[0238] The second calculating unit is configured to calculate a distribution of the relative distance average on the splicing baseline and adjust the splicing reference position according to the distribution of the relative distance average on the splicing baseline.

[0239] Further, when the definition of the two adjacent images is inconsistent, the definition of the image with higher definition is adjusted to be the same as the definition of the image with lower definition.

[0240] Further, when the first cutting adjustment curve is transferred to the second adjustment curve and moved to the coincidence position, the movement includes horizontal movement and vertical movement.

[0241] Further, the method further comprises determining the feature points using a wavelet decomposition method and coinciding the feature points on the first cutting adjustment curve with the feature points on the second adjustment curve.

[0242] In one example, the units in any of the above apparatuses can be one or more integrated circuits, configured to implement one or more of the above methods, e.g., one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0243] For another example, when the units in the apparatuses can be implemented in the form of a processing element scheduler, the processing element can be a general purpose processor, such as a central processing unit (CPU) or other processor that can invoke a program. For yet another example, these units can be integrated together, implemented in the form of a system-on-a-chip (SOC).

[0244] In the present application, various objects such as messages / information / devices / network elements / systems / apparatuses / actions / operations / processes / concepts, etc. that can occur in the present application are named. It can be understood that these specific names do not constitute a limitation on the related objects, and the names can be changed according to the scene, context or usage habits, etc. The technical meaning of the technical terms in the present application should be mainly determined from the function and technical effect embodied / implemented in the technical scheme.

[0245] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, apparatus and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0246] In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0247] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0248] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0249] It should also be understood that in various embodiments of the present application, first, second, etc. are only to represent that a plurality of objects are different. For example, the first time window and the second time window are only to represent different time windows. The above first, second, etc. should not have any impact on the time window itself, and should not limit the embodiments of the present application.

[0250] It should also be understood that in various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0251] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product stored in a computer readable storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned computer readable storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and various program code storage media.

[0252] The present application also provides a gantry crane adaptive image stitching system, the system comprises:

[0253] One or more memories for storing instructions; and

[0254] One or more processors are configured to retrieve and execute the instructions from the memory, performing the methods described above.

[0255] This application also provides a computer program product including instructions that, when executed, cause the terminal device and the network device to perform operations corresponding to the methods described above.

[0256] This application also provides a chip system including a processor for implementing the functions involved in the above description, such as generating, receiving, transmitting, or processing the data and / or information involved in the above methods.

[0257] This chip system can consist of chips or include chips and other discrete components.

[0258] The processor mentioned above can be a CPU, a microprocessor, an ASIC, or one or more integrated circuits that execute a program to control the method of transmitting the feedback information described above.

[0259] In one possible design, the chip system also includes a memory for storing necessary program instructions and data. The processor and the memory can be decoupled and located on different devices, connected via wired or wireless means to support the chip system in implementing the various functions described in the above embodiments. Alternatively, the processor and the memory can also be coupled to the same device.

[0260] Optionally, the computer instructions are stored in memory.

[0261] Optionally, the memory can be a storage unit within the chip, such as a register or cache. Alternatively, the memory can be a storage unit located outside the chip within the terminal, such as a ROM or other types of static storage devices that can store static information and instructions, such as RAM.

[0262] It is understood that the memory in this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0263] Non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0264] Volatile memory can be RAM, which is used as an external cache. There are many different types of RAM, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory.

[0265] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. An adaptive image stitching method for gantry cranes, characterized in that, include: In response to the acquired images within the coverage area, determine the state of the container objects in the images, including their movement state and their placement state; When the container object is in the placement state, obtain the height information of the container object; The splicing parameters are calculated based on the height information of the container object. The splicing parameters include the first edge splicing parameters and the second edge splicing parameters. Use stitching parameters to stitch together multiple images; In this process, each image involved in the stitching comes from a different image acquisition unit and each image is generated at the same time. Each image acquisition unit generates only one image at a time. Using stitching parameters to stitch multiple images includes: Geometric correction of the same container object in two adjacent images is performed using stitching parameters; Cropping is performed on two adjacent images based on the geometric correction results; Two adjacent cropped images are stitched together to obtain a stitched image; Adjust the stitched image according to the degree of blending at the stitching position to make the stitching baseline of the stitched image transition smoothly; Adjusting the stitched image based on the degree of blending at the stitching location to ensure a smooth transition of the stitching baseline includes: Determine the stitching position on two adjacent images and create a stitching curve at the stitching position; Establish a reference baseline at the splicing reference position, with the reference baseline perpendicular to the splicing baseline; Use the pixels on the reference baseline to create the adjustment curve; The adjustment curve is trimmed using the splicing baseline to obtain the trimmed adjustment curve; Adjust the splicing reference position according to the relative positions of the two trimming adjustment curves; Adjusting the splicing reference position based on the relative positions of the two trimming adjustment curves includes: The two adjustment curves are designated as the first adjustment curve and the second adjustment curve, respectively. A first trimming adjustment curve is obtained on the first adjustment curve, and a second trimming adjustment curve is obtained on the second adjustment curve; Transfer the first trimming adjustment curve to the second adjustment curve and move it to the overlapping position, and calculate the first relative distance between the overlapping position and the splicing baseline; Transfer the second trimming adjustment curve to the first adjustment curve and move it to the overlapping position, and calculate the second relative distance between the overlapping position and the splicing baseline; Calculate the average relative distance between the first relative distance and the second relative distance; Calculate the distribution of the average relative distance on the stitching baseline and adjust the stitching reference position based on the distribution of the average relative distance on the stitching baseline.

2. The adaptive image stitching method for gantry cranes according to claim 1, characterized in that, It also includes merging the overlapping areas of two images, including brightness merging, color difference merging, color balance, and brightness adjustment.

3. The adaptive image stitching method for gantry cranes according to claim 1, characterized in that, When two adjacent images have different sharpness, adjust the sharpness of the image with higher sharpness to match that of the image with lower sharpness.

4. The adaptive image stitching method for gantry cranes according to claim 1, characterized in that, When the first trimming adjustment curve is transferred to the second adjustment curve and moved to the overlapping position, the movement includes horizontal movement and vertical movement. It also includes using wavelet decomposition to determine feature points and aligning feature points on the first trimming adjustment curve with feature points on the second adjustment curve.

5. A gantry crane adaptive image stitching device, characterized in that, include: The state determination unit is used to determine the state of the container objects in the image in response to the acquired image within the coverage area. The state includes the movement state and the placement state. The first data processing unit is used to obtain the height information of the container object when the container object is in the placement state. The second data processing unit is used to calculate splicing parameters based on the height information of the container object. The splicing parameters include first edge splicing parameters and second edge splicing parameters. The stitching processing unit is used to stitch multiple images together using stitching parameters; In this process, each image involved in the stitching comes from a different image acquisition unit and each image is generated at the same time. Each image acquisition unit generates only one image at a time. Using stitching parameters to stitch multiple images includes: Geometric correction of the same container object in two adjacent images is performed using stitching parameters; Cropping is performed on two adjacent images based on the geometric correction results; Two adjacent cropped images are stitched together to obtain a stitched image; Adjust the stitched image according to the degree of blending at the stitching position to make the stitching baseline of the stitched image transition smoothly; Adjusting the stitched image based on the degree of blending at the stitching location to ensure a smooth transition of the stitching baseline includes: Determine the stitching position on two adjacent images and create a stitching curve at the stitching position; Establish a reference baseline at the splicing reference position, with the reference baseline perpendicular to the splicing baseline; Use the pixels on the reference baseline to create the adjustment curve; The adjustment curve is trimmed using the splicing baseline to obtain the trimmed adjustment curve; Adjust the splicing reference position according to the relative positions of the two trimming adjustment curves; Adjusting the splicing reference position based on the relative positions of the two trimming adjustment curves includes: The two adjustment curves are designated as the first adjustment curve and the second adjustment curve, respectively. A first trimming adjustment curve is obtained on the first adjustment curve, and a second trimming adjustment curve is obtained on the second adjustment curve; Transfer the first trimming adjustment curve to the second adjustment curve and move it to the overlapping position, and calculate the first relative distance between the overlapping position and the splicing baseline; Transfer the second trimming adjustment curve to the first adjustment curve and move it to the overlapping position, and calculate the second relative distance between the overlapping position and the splicing baseline; Calculate the average relative distance between the first relative distance and the second relative distance; Calculate the distribution of the average relative distance on the stitching baseline and adjust the stitching reference position based on the distribution of the average relative distance on the stitching baseline.

6. A gantry crane adaptive image stitching system, characterized in that, The system includes: One or more memories for storing instructions; and One or more processors are configured to retrieve and execute the instructions from the memory to perform the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: The program, when run by the processor, executes the method as described in any one of claims 1 to 4.

Images