Self-adaptive image splicing method and system for gantry crane

Abstract

The invention relates to a gantry crane adaptive image splicing method and system, and the method comprises the steps: responding to an obtained image in a coverage range, and judging the state of a container object in the image, the state including a moving state and a placing state; when the state of the container object is a placement state, acquiring height information of the container object; calculating splicing parameters according to the height information of the container object, wherein the splicing parameters comprise a first edge splicing parameter and a second edge splicing parameter; the splicing parameters are used for splicing the multiple images, the generation time of each image is the same, and each image obtaining unit only generates one image at one time point. According to the gantry crane adaptive image splicing method and system disclosed by the invention, complete image splicing of containers with different heights is realized in a manner of adaptively adjusting splicing parameters, image loss and distortion are avoided, and reliable image support is provided for safe operation of a gantry crane.

Description

Technical Field

[0001] The present application relates to the technical field of image stitching processing, and in particular to a method and system for adaptive image stitching of a gantry crane. Background Art

[0002] With the growth of global trade, container terminals play a key role in modern logistics. Gantry cranes, as one of the terminal loading and unloading equipment, require precise operation and efficient monitoring systems when handling and stacking containers to ensure the safety and efficiency of operations. However, with the continuous stacking of containers, traditional monitoring systems face many challenges.

[0003] In the prior art, the monitoring system of the gantry crane usually realizes real-time monitoring of the working area by installing cameras on the spreader or the gantry crane structure. However, the images of these cameras need to be spliced ​​to form a complete panoramic view for the operator to view the entire working area in real time.

[0004] Traditional image stitching technology is usually performed on a fixed plane, that is, stitching camera images based on the ground perspective. However, this method has significant defects when dealing with scenes where the stacking height of containers changes. The main reason is that it cannot adapt to the height change, resulting in excessive image cropping and information loss, which will directly affect the monitoring effect and the safety of the lifting process. Summary of the invention

[0005] The present application provides an adaptive image stitching method and system for a gantry crane, which realizes complete image stitching of containers of different heights by adaptively adjusting stitching parameters, avoids image loss and distortion, and provides reliable image support for the safe operation of the gantry crane.

[0006] The above-mentioned purpose of the present application is achieved through the following technical solutions: In a first aspect, the present application provides a gantry crane adaptive image stitching method, comprising: In response to the acquired image within the coverage area, determining the state of the container object in the image, the state including the moving state and the placement state; When the state of the container object is the placement state, obtain the height information of the container object; Calculate splicing parameters according to the height information of the container object, where the splicing parameters include a first edge splicing parameter and a second edge splicing parameter; Use stitching parameters to stitch multiple images; Among them, each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same; Each image acquisition unit generates only one image at a time point.

[0007] In a possible implementation manner of the first aspect, the overlapping area of ​​the two images is further fused, and the fusion includes brightness fusion, color difference fusion, color balance and brightness adjustment.

[0008] In a possible implementation manner of the first aspect, stitching the multiple images using the stitching parameters includes: Use stitching parameters to perform geometric correction on the same container object in two adjacent images; Crop two adjacent images according to the geometric correction results; The two adjacent cropped images are stitched together to obtain a stitched image; The stitched image is adjusted according to the fusion degree at the stitching position so that the stitching baseline of the stitched image has a smooth transition.

[0009] In a possible implementation manner of the first aspect, adjusting the stitched image according to the fusion degree at the stitching position so that the stitching baseline of the stitched image transitions smoothly includes: Determine a stitching position on two adjacent images and create a stitching curve at the stitching position; Establish a reference baseline at the splicing reference position, and the reference baseline is perpendicular to the splicing baseline; Use the pixels on the reference baseline to create an adjustment curve; The adjustment curve is trimmed using the stitching baseline to obtain a trimmed adjustment curve; Adjust the stitching reference position according to the relative positions of the two cropping adjustment curves.

[0010] In a possible implementation manner of the first aspect, adjusting the splicing reference position according to the relative positions of the two cutting adjustment curves includes: The two adjustment curves are respectively recorded as a first adjustment curve and a second adjustment curve; A first cutting adjustment curve is obtained on the first adjustment curve, and a second cutting adjustment curve is obtained on the second adjustment curve; Transferring the first cutting adjustment curve to the second adjustment curve and moving it to the overlapping position and calculating a first relative distance between the overlapping position and the stitching baseline; Transferring the second cutting adjustment curve to the first adjustment curve and moving it to the overlapping position and calculating a second relative distance between the overlapping position and the stitching baseline; Calculate the relative distance average of the first relative distance and the second relative distance; The distribution of the relative distance average on the stitching baseline is calculated and the stitching reference position is adjusted according to the distribution of the relative distance average on the stitching baseline.

[0011] In a possible implementation manner of the first aspect, when the sharpness of two adjacent images is inconsistent, the sharpness of the image with higher sharpness is adjusted to be the same as the sharpness of the image with lower sharpness.

[0012] In a possible implementation manner of the first aspect, when the first cutting adjustment curve is transferred to the second adjustment curve and moved to an overlapping position, the movement includes a lateral movement and a longitudinal movement; The method also includes determining feature points by wavelet decomposition and overlapping the feature points on the first cutting adjustment curve with the feature points on the second adjustment curve.

[0013] In a second aspect, the present application provides a gantry crane adaptive image stitching device, comprising: A state judgment unit, configured to judge the state of the container object in the image in response to the acquired image within the coverage range, wherein the state includes a moving state and a placement state; A first data processing unit, configured to obtain height information of the container object when the state of the container object is a placement state; A second data processing unit is used to calculate splicing parameters according to the height information of the container object, where the splicing parameters include a first edge splicing parameter and a second edge splicing parameter; A stitching processing unit, used for stitching a plurality of images using stitching parameters; Among them, each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same; Each image acquisition unit generates only one image at a time point.

[0014] In a third aspect, the present application provides a gantry crane adaptive image stitching system, the system comprising: one or more memories for storing instructions; and One or more processors, used to call and run the instructions from the memory to execute the method as described in the first aspect and any possible implementation of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising: Program, when the program is executed by a processor, the method described in the first aspect and any possible implementation of the first aspect is executed.

[0016] In a fifth aspect, the present application provides a computer program product, comprising program instructions. When the program instructions are executed by a computing device, the method described in the first aspect and any possible implementation of the first aspect is executed.

[0017] In a sixth aspect, the present application provides a chip system, which includes a processor for implementing the functions involved in the above aspects, for example, generating, receiving, sending, or processing the data and / or information involved in the above methods.

[0018] The chip system may be composed of chips, or may include chips and other discrete devices.

[0019] In a possible design, the chip system also includes a memory, which is used to store necessary program instructions and data. The processor and the memory can be decoupled and respectively set on different devices, connected by wired or wireless means, or the processor and the memory can also be coupled on the same device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic flowchart of the steps of a gantry crane adaptive image stitching method provided in this application.

[0021] Figure 2 This is a schematic diagram of the position deployment of cameras on a gantry crane provided in this application.

[0022] Figure 3 This is a schematic diagram of an overlapping area between cameras provided in this application.

[0023] Figure 4 A correspondence diagram between a calculated splicing parameter P and a real splicing parameter P is provided in this application.

[0024] Figure 5 It is a schematic diagram of a splicing baseline provided in this application.

[0025] Figure 6 It is a schematic diagram of a reference baseline provided in this application. DETAILED DESCRIPTION

[0026] The technical solution in this application is further described in detail below in conjunction with the accompanying drawings.

[0027] This application discloses a gantry crane adaptive image stitching method, please refer to Figure 1 In some examples, the gantry crane adaptive image stitching method disclosed in the present application includes the following steps: S101, in response to the acquired image within the coverage area, determining the state of the container object in the image, where the state includes a moving state and a placement state; S102, when the state of the container object is a placed state, obtaining height information of the container object; S103, calculating splicing parameters according to the height information of the container object, where the splicing parameters include a first edge splicing parameter and a second edge splicing parameter; S104, stitching the multiple images using stitching parameters; Among them, each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same; Each image acquisition unit generates only one image at a time point.

[0028] Specifically, in the technical solution disclosed in this application, please refer to Figure 2 , it is necessary to install multiple cameras on the gantry crane, each camera is responsible for one area, and there are overlaps between these areas, such as Figure 3 As shown, the images captured by the camera are stitched together to generate a panoramic image.

[0029] See also Figure 3 In the prior art, In step S101, the state of the container object in the image must first be determined. There are two states, namely, a moving state and a placed state. The moving state means that the container object is in a moving state, including horizontal movement and vertical movement. The placed state means that the container object is placed on the ground or on other container objects.

[0030] The state can be determined based on the gantry crane. For example, if there is only one gantry crane in an area, two images are obtained at two different time points. By comparing the images, it can be determined which container objects are in a moving state and which container objects are in a placed state.

[0031] When the state of the container object is the placement state, the height information of the container object is obtained. The height information here is obtained through the gantry crane. The gantry crane will generate a work record during the work process. The movement in the work record can be used as the coordinate position, and the work record of the lifting device can be used as the height information of the container object.

[0032] In some possible implementations, the height information of the container object is obtained by means of a distance sensor.

[0033] In step S103, the splicing parameters are calculated according to the height information of the container object. The splicing parameters include the first edge splicing parameters and the second edge splicing parameters. The specific contents are as follows: By analyzing multiple sets of data on height H and splicing parameter P, it is found that there is a nonlinear relationship between the two, so a quadratic equation is used for fitting. The splicing parameter P is set to:

[0034] The splicing parameter formula of the first angle and the second angle is obtained by data fitting: First edge stitching parameters: ; Second edge stitching parameters: ; Where H represents the current spreader height (unit: meter), and P represents the corresponding splicing parameter value.

[0035] When the spreader is at a certain height When Substitute the above splicing parameter formula to calculate the splicing parameters at the current height: ; .

[0036] In step S104, multiple images are stitched using stitching parameters, where it is required that each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same, and each image acquisition unit only generates one image at a time point.

[0037] The stitching process also includes fusion of the overlapping areas of the two images, including brightness fusion, color difference fusion, color balance and brightness adjustment.

[0038] The relevant contents of geometric transformation and correction are as follows: Assume that each camera (where i=1,2,...,8) The captured image coordinates are , the image is geometrically transformed, and the new coordinates after the transformation are marked as .

[0039] The transformation process requires translation, rotation and scaling. The specific geometric transformation formula is as follows: ;

[0040] in, , , , are the rotation and scaling factors, , are the translation coefficients. These coefficients are determined by the splicing parameters and Determined and automatically calculated by the stitching module.

[0041] The relevant content of the image overlap area is as follows: There are overlapping areas between the camera fields of view to ensure the continuity of stitching. Assume that the image pixel values ​​in the overlapping area are , the system uses weighted average method to fuse the overlapping areas to eliminate the brightness and color difference of different camera images. The formula of weighted average is: in, and is the pixel value of the adjacent camera images in the overlapping area, is the weight coefficient, which is adjusted according to the degree of image overlap.

[0042] The relevant contents of the image stitching order are as follows: Image stitching is performed in the order of the spatial positions of the cameras, for example, from left to right and from top to bottom. The pixel coordinates of the camera image after transformation are , then the coordinates of the stitched panoramic image are:

[0043] in, and It is the offset of each camera image in the panoramic image. The offset is dynamically calculated according to the installation position of each camera and the stitching parameters.

[0044] The relevant contents of color balance and brightness adjustment are as follows: To ensure that there is no color difference after stitching images from different cameras, the system performs color balance and brightness adjustment on each image block. Assume that the original pixel brightness is , the adjusted brightness is , then the brightness adjustment formula is: in, It is the brightness adjustment coefficient, which is dynamically adjusted according to the lighting conditions of different cameras to ensure the consistency of the stitched image.

[0045] For example: 8 cameras installed on the gantry crane beam The system is equipped with a height sensor to monitor the lifting height of the spreader in real time (denoted as ), and send the height information to the splicing module.

[0046] 1. Data Collection and Splicing Parameter Fitting In the preparation stage, the spreader collects images from a height of 2 meters to 14 meters (at intervals of 2 meters). At each height, the stitching parameters of each camera's field of view are recorded to establish a corresponding relationship between height and stitching parameters.

[0047] The fitting formula for the first corner and second edge splicing parameters is obtained through data analysis: First edge splicing parameter formula: ; Second edge splicing parameter formula: .

[0048] formula( )pass Figure 4 The content displayed in is obtained.

[0049] The first edge stitching parameter here corresponds to the upper left corner and the lower left corner of the container, and the second edge stitching parameter corresponds to the upper right corner and the lower right corner of the container. Through the stitching parameters, the parts of a container in the two images can be matched, and then the pictures can be cropped and stitched.

[0050] 2. Real-time stitching process Assume that the current spreader is handling a container with a stack height of 10 meters, and the system receives the current height information m, and perform the following steps: (1) Calculation of splicing parameters Will Substituting into the splicing parameter formula, we get: Upper left corner stitching parameters: ; Upper right corner stitching parameters: ; (2) Geometric transformation and image correction Calculated splicing parameters and It will be used as the basis for image geometric correction. Assume that the original image coordinates are , then the transformed coordinates are: ; ; Among them, the transformation coefficient , , , , , By splicing parameters and The specific calculation is as follows: because and are parameters related to height, which represent the size change of the image or the width of the field of view, so we can calculate the current height by and to derive the scaling factor.

[0051] Assume we have a base height And the corresponding splicing parameters and , and the current altitude And the corresponding splicing parameters and .

[0052] Step 1: Calculate the width change ratio (i.e. scaling factor) Base width: Current Width: Scaling factor: Step 2: Calculate the rotation angle The rotation factor reflects the image rotation due to height changes. and The change of affects not only the width, but also the viewing angle of the image (for example, the rotation angle). We need to infer the rotation angle through the height change.

[0053] The height change causes the image to rotate, the rotation angle It is related to the change of the splicing parameters. Specifically, the rotation angle can be calculated by the following relationship: The relationship between rotation angle and stitching parameters: in, and is an empirical constant that reflects the relationship between the change of splicing parameters and the rotation angle.

[0054] Calculate the rotation coefficient: Once you have the rotation angle , rotation coefficient It can be calculated by the rotation matrix: , , , Step 3: Calculate the translation coefficient The translation factor controls the offset of the image in the x and y directions. The translation factor is related to the perspective and height changes of the image, and is usually adjusted dynamically as the height changes.

[0055] The translation factor reflects the position adjustment of the image. The translation of the image is determined by the stitching parameters or other factors, and can be determined by calculating the relative offset: , in: , is the relative translation scale factor of the image in the x and y directions. are the current splicing parameters.

[0056] example: Base height , the corresponding splicing parameters are: ; .

[0057] Current altitude , the corresponding splicing parameters are: ; .

[0058] Step 1: Calculate the scaling factor =397.98-325.42=62.56 =266.53-192.44=74.09

[0059] Step 2: Calculate the rotation angle The relationship between the rotation angle and the change of splicing parameters is:

[0060] set up and :

[0061] Once the rotation angle is obtained , rotation coefficient It can be calculated by the rotation matrix: , , ,

[0062] Step 3: Calculate the translation coefficient Assume the image is shifted by a factor of and ,but: , Through calculation, we get: Scaling factor Rotation coefficient Translation coefficient , Therefore, the transformation formula is: ; ; 3. Processing of overlapping image areas Assume the camera and The pixel value of the overlapping area is , use the weighted average method to fuse the overlapping areas: in, and Camera and The image pixel value, is a weight factor that is automatically adjusted based on the size of the overlap area.

[0063] 4. Generation of stitched panoramas The images after geometric correction and overlap area processing are stitched together in the order of camera positions to generate a complete panoramic image. The image position of each camera is adjusted according to the set offset so that the final stitched image is seamlessly connected to form a complete image of the gantry crane operation area.

[0064] Assume that the offset of camera 1 is ( , the offset of camera 2 is ,but:

[0065]

[0066]

[0067] These adjusted images are stitched together in sequence to obtain a complete panoramic image.

[0068] In some examples, the specific method of stitching multiple images using stitching parameters is: S201, performing geometric correction on the same container object in two adjacent images using stitching parameters; S202, cropping two adjacent images according to the geometric correction result; S203, stitching the two adjacent cropped images to obtain a stitched image; S204, adjusting the stitched image according to the fusion degree at the stitching position, so that the stitching baseline of the stitched image has a smooth transition.

[0069] The contents of step S201 to step S203 have been introduced in the above content. The general content is that due to the different positions of the cameras, the images obtained by each camera are different in size and distance. Therefore, the two images cannot be directly fused, and the images need to be corrected with the help of the first edge stitching parameters and the second edge stitching parameters.

[0070] However, there are certain errors in the calculation of the first edge stitching parameter and the second edge stitching parameter, which will cause unnatural image transition at the stitching position. Therefore, it is necessary to adjust the stitching image according to the fusion degree at the stitching position to make the stitching baseline of the stitching image transition smoothly.

[0071] In some examples, the stitched image is adjusted according to the fusion degree at the stitching position so that the stitching baseline of the stitched image has a smooth transition. Specifically, the stitching baseline of the stitched image is smoothly transitioned as follows: S301, determining a stitching position on two adjacent images and creating a stitching curve at the stitching position; S302, establishing a reference baseline at the splicing reference position, wherein the reference baseline is perpendicular to the splicing baseline; S303, establishing an adjustment curve using the pixel points on the reference baseline; S304, using the splicing baseline to trim the adjustment curve to obtain a trimmed adjustment curve; S305: Adjust the splicing reference position according to the relative positions of the two cutting adjustment curves.

[0072] In step S301 to step S305, first, a stitching baseline ( Figure 5 ) to establish a reference baseline ( Figure 6 As shown in FIG. 1 ), the reference baseline is perpendicular to the stitching baseline, and then the pixel points on the reference baseline are used to establish an adjustment curve, and then the stitching baseline is used to cut the adjustment curve to obtain a cut adjustment curve.

[0073] The way to establish an adjustment curve using the pixel points on the reference baseline is to calculate the difference between adjacent pixel points, and then use the order corresponding to the difference as the horizontal coordinate and the difference as the vertical coordinate to obtain discrete points. These discrete points are connected in sequence to obtain a cropping adjustment curve.

[0074] Of course, the quadratic difference can also be used here to obtain the cutting adjustment curve.

[0075] The method of trimming the adjustment curve using the stitching baseline is to Figure 6 To describe, the stitching baseline divides the adjustment curve into two segments, and then retains the one segment used as the trimming adjustment curve. Figure 6 The adjustment curve on the left side of the center retains the right half, and the adjustment curve on the right side retains the left half.

[0076] Finally, the stitching reference position is adjusted according to the relative positions of the two cutting adjustment curves.

[0077] This method uses multiple points to correct the stitching baseline. These points are set at intervals on the stitching baseline. By adjusting the relative positions of the two cropping adjustment curves at these points, a smooth transition of the stitching image at the stitching baseline is achieved.

[0078] The specific method of adjusting the stitching reference position according to the relative positions of the two cutting adjustment curves is: S401, recording two adjustment curves as a first adjustment curve and a second adjustment curve respectively; S402, obtaining a first cutting adjustment curve on the first adjustment curve, and obtaining a second cutting adjustment curve on the second adjustment curve; S403, transferring the first cutting adjustment curve to the second adjustment curve and moving it to the overlapping position and calculating a first relative distance between the overlapping position and the stitching baseline; S404, transferring the second cutting adjustment curve to the first adjustment curve and moving it to the overlapping position and calculating a second relative distance between the overlapping position and the stitching baseline; S405, calculating a relative distance average of the first relative distance and the second relative distance; S406, calculating the distribution of the average relative distance on the stitching baseline and adjusting the stitching reference position according to the distribution of the average relative distance on the stitching baseline.

[0079] In step S401 to step S406, firstly, a first cutting adjustment curve is obtained on the first adjustment curve, and at the same time, a second cutting adjustment curve is obtained on the second adjustment curve. Then, the first cutting adjustment curve is transferred to the second adjustment curve and moved to the overlapping position, and a first relative distance between the overlapping position and the stitching baseline is calculated. At the same time, the second cutting adjustment curve is transferred to the first adjustment curve and moved to the overlapping position, and a second relative distance between the overlapping position and the stitching baseline is calculated.

[0080] The first relative distance and the second relative distance respectively represent the movement distances of the two corresponding images during stitching. At this time, the relative distance average of the first relative distance and the second relative distance is used as the movement reference value of the two images.

[0081] Finally, calculate the distribution of the relative distance average on the stitching baseline and adjust the stitching reference position based on the distribution of the relative distance average on the stitching baseline. Figure 6 The stitching baseline on the left or the stitching baseline on the right, that is, only one image is adjusted during stitching, and the adjustment here is rotation.

[0082] In some possible implementations, when the sharpness of two adjacent images is inconsistent, the sharpness of the image with higher sharpness is adjusted to be the same as the sharpness of the image with lower sharpness.

[0083] In some possible implementations, when the first cutting adjustment curve is transferred to the second adjustment curve and moved to an overlapping position, the movement includes a horizontal movement and a vertical movement.

[0084] When both the horizontal movement and the vertical movement cannot meet the condition of moving to the overlapping position, the following method is used for processing: The feature points are determined by wavelet decomposition and the feature points on the first cutting adjustment curve are overlapped with the feature points on the second adjustment curve.

[0085] The present application also provides a gantry crane adaptive image stitching device, comprising: A state judgment unit, configured to judge the state of the container object in the image in response to the acquired image within the coverage range, wherein the state includes a moving state and a placement state; A first data processing unit, configured to obtain height information of the container object when the state of the container object is a placement state; A second data processing unit is used to calculate splicing parameters according to the height information of the container object, where the splicing parameters include a first edge splicing parameter and a second edge splicing parameter; A stitching processing unit, used for stitching a plurality of images using stitching parameters; Among them, each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same; Each image acquisition unit generates only one image at a time point.

[0086] Furthermore, it also includes fusing the overlapping areas of the two images, and the fusion includes brightness fusion, color difference fusion, color balance and brightness adjustment.

[0087] Furthermore, it also includes: A geometric correction unit, used for performing geometric correction on the same container object in two adjacent images using stitching parameters; A cropping unit, used for cropping two adjacent images according to the geometric correction result; A stitching unit, used for stitching two adjacent cropped images to obtain a stitched image; The splicing adjustment unit is used to adjust the spliced ​​image according to the fusion degree at the splicing position so that the splicing baseline of the spliced ​​image has a smooth transition.

[0088] Furthermore, it also includes: A stitching curve creating unit, used for determining a stitching position on two adjacent images and creating a stitching curve at the stitching position; A reference baseline creation unit is used to establish a reference baseline at a splicing reference position, wherein the reference baseline is perpendicular to the splicing baseline; An adjustment curve creation unit, used for creating an adjustment curve using pixel points on a reference baseline; A first cutting adjustment curve creating unit, used for cutting the adjustment curve using the splicing baseline to obtain a cutting adjustment curve; The splicing reference position adjustment unit is used to adjust the splicing reference position according to the relative positions of the two cutting adjustment curves.

[0089] Furthermore, it also includes: a marking unit, used for marking the two adjustment curves as a first adjustment curve and a second adjustment curve respectively; A second cutting adjustment curve creating unit, used for obtaining a first cutting adjustment curve on the first adjustment curve, and obtaining a second cutting adjustment curve on the second adjustment curve; An overlap processing unit, used for transferring the first cutting adjustment curve to the second adjustment curve and moving it to an overlap position and calculating a first relative distance between the overlap position and a stitching baseline; A distance calculation unit is used to transfer the second cutting adjustment curve to the first adjustment curve and move it to the overlapping position and calculate the second relative distance between the overlapping position and the splicing baseline; A first calculation unit, used for calculating a relative distance average of the first relative distance and the second relative distance; The second calculation unit is used to calculate the distribution of the relative distance average value on the stitching baseline and adjust the stitching reference position according to the distribution of the relative distance average value on the stitching baseline.

[0090] Furthermore, when the sharpness of two adjacent images is inconsistent, the sharpness of the image with higher sharpness is adjusted to be the same as the sharpness of the image with lower sharpness.

[0091] Further, when the first cutting adjustment curve is transferred to the second adjustment curve and moved to the overlapping position, the movement includes horizontal movement and vertical movement; The method also includes determining feature points by wavelet decomposition and overlapping the feature points on the first cutting adjustment curve with the feature points on the second adjustment curve.

[0092] In one example, the unit in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0093] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0094] Various objects such as various messages / information / equipment / network elements / systems / devices / actions / operations / processes / concepts that may appear in this application are named. It can be understood that these specific names do not constitute a limitation on the relevant objects. The names assigned may change with factors such as scenarios, contexts or usage habits. The understanding of the technical meaning of the technical terms in this application should be mainly determined from the functions and technical effects embodied / executed in the technical scheme.

[0095] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0097] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] It should also be understood that in various embodiments of the present application, the first, second, etc. are only used to indicate that multiple objects are different. For example, the first time window and the second time window are only used to indicate different time windows. They should not have any impact on the time window itself, and the first, second, etc. mentioned above should not impose any limitations on the embodiments of the present application.

[0100] It should also be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0101] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0102] The present application also provides a gantry crane adaptive image stitching system, the system comprising: one or more memories for storing instructions; and One or more processors are used to call and run the instructions from the memory to execute the method as described above.

[0103] The present application also provides a computer program product, which includes instructions. When the instructions are executed, the terminal device and the network device perform operations of the terminal device and the network device corresponding to the above method.

[0104] The present application also provides a chip system, which includes a processor for implementing the functions involved in the above content, such as generating, receiving, sending, or processing the data and / or information involved in the above method.

[0105] The chip system may be composed of chips, or may include chips and other discrete devices.

[0106] The processor mentioned in any of the above places can be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for executing programs for controlling the above-mentioned feedback information transmission method.

[0107] In a possible design, the chip system also includes a memory, which is used to store necessary program instructions and data. The processor and the memory can be decoupled and respectively set on different devices, connected by wire or wireless means to support the chip system to implement various functions in the above embodiments. Alternatively, the processor and the memory can also be coupled on the same device.

[0108] Optionally, the computer instructions are stored in a memory.

[0109] Optionally, the memory is a storage unit within the chip, such as a register, a cache, etc. The memory can also be a storage unit within the terminal located outside the chip, such as a ROM or other types of static storage devices that can store static information and instructions, RAM, etc.

[0110] It can be understood that the memory in the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0111] The non-volatile memory may be a ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory.

[0112] The volatile memory may be a RAM, which is used as an external cache. There are many different types of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus RAM.

[0113] The embodiments of this specific implementation method are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Therefore, all equivalent changes made based on the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A gantry crane adaptive image stitching method, characterized in that: include: In response to the acquired image within the coverage area, determining the state of the container object in the image, the state including the moving state and the placement state; When the state of the container object is the placement state, obtain the height information of the container object; Calculate splicing parameters according to the height information of the container object, where the splicing parameters include a first edge splicing parameter and a second edge splicing parameter; Use stitching parameters to stitch multiple images; Among them, each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same; Each image acquisition unit generates only one image at a time point.

2. The gantry crane adaptive image stitching method according to claim 1 is characterized in that: It also includes the fusion of the overlapping areas of the two images, including brightness fusion, color difference fusion, color balance and brightness adjustment.

3. The gantry crane adaptive image stitching method according to claim 1, characterized in that: Using stitching parameters to stitch multiple images includes: Use stitching parameters to perform geometric correction on the same container object in two adjacent images; Crop two adjacent images according to the geometric correction results; The two adjacent cropped images are stitched together to obtain a stitched image; The stitched image is adjusted according to the fusion degree at the stitching position so that the stitching baseline of the stitched image has a smooth transition.

4. The gantry crane adaptive image stitching method according to claim 3 is characterized in that: Adjusting the stitched image according to the fusion degree at the stitching position so that the stitching baseline of the stitched image has a smooth transition includes: Determine a stitching position on two adjacent images and create a stitching curve at the stitching position; Establish a reference baseline at the splicing reference position, and the reference baseline is perpendicular to the splicing baseline; Use the pixels on the reference baseline to create an adjustment curve; The adjustment curve is trimmed using the stitching baseline to obtain a trimmed adjustment curve; Adjust the stitching reference position according to the relative positions of the two cropping adjustment curves.

5. The gantry crane adaptive image stitching method according to claim 4 is characterized in that: Adjusting the stitching reference position according to the relative position of two cutting adjustment curves includes: The two adjustment curves are respectively recorded as a first adjustment curve and a second adjustment curve; A first cutting adjustment curve is obtained on the first adjustment curve, and a second cutting adjustment curve is obtained on the second adjustment curve; Transferring the first cutting adjustment curve to the second adjustment curve and moving it to the overlapping position and calculating a first relative distance between the overlapping position and the stitching baseline; Transferring the second cutting adjustment curve to the first adjustment curve and moving it to the overlapping position and calculating a second relative distance between the overlapping position and the stitching baseline; Calculate the relative distance average of the first relative distance and the second relative distance; The distribution of the relative distance average on the stitching baseline is calculated and the stitching reference position is adjusted according to the distribution of the relative distance average on the stitching baseline.

6. The gantry crane adaptive image stitching method according to claim 4, characterized in that: When the sharpness of two adjacent images is inconsistent, the sharpness of the image with higher sharpness is adjusted to be the same as the sharpness of the image with lower sharpness.

7. The gantry crane adaptive image stitching method according to claim 5, characterized in that: When the first cutting adjustment curve is transferred to the second adjustment curve and moved to the overlapping position, the movement includes horizontal movement and vertical movement; The method also includes determining feature points by wavelet decomposition and overlapping the feature points on the first cutting adjustment curve with the feature points on the second adjustment curve.

8. A gantry crane adaptive image stitching device, characterized in that: include: A state judgment unit, configured to judge the state of the container object in the image in response to the acquired image within the coverage range, wherein the state includes a moving state and a placement state; A first data processing unit, configured to obtain height information of the container object when the state of the container object is a placed state; A second data processing unit, used to calculate splicing parameters according to the height information of the container object, the splicing parameters including a first edge splicing parameter and a second edge splicing parameter; A stitching processing unit, used for stitching a plurality of images using stitching parameters; Among them, each image involved in the stitching comes from a different image acquisition unit and the generation time of each image is the same; Each image acquisition unit generates only one image at a time point.

9. A gantry crane adaptive image stitching system, characterized in that: The system comprises: one or more memories for storing instructions; and One or more processors, configured to call and execute the instructions from the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer readable storage medium comprises: The program, when the program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images