Image splicing method, system and device for gate and storage medium

Abstract

The invention provides an image splicing method, system and device for a gate, and a storage medium. The method comprises the following steps: collecting an image in real time through a camera arranged at the position of the gate; inputting the previous frame image and the current frame image into an optical flow estimation model, and determining transverse motion data and longitudinal motion data according to output data of the optical flow estimation model; determining a starting point and an ending point of an image extracted from the current frame image based on the transverse motion data; extracting a splicing segment image from the current frame image based on a preset image extraction starting point and the width of the extracted image; processing the spliced segment image according to the longitudinal motion data to obtain a to-be-spliced image; and splicing the to-be-spliced image to one side of the output image corresponding to the previous frame of image to serve as the output image corresponding to the current frame of image. According to the invention, the image splicing effect for the gate is improved.

Description

Background Art

[0002] In the gate project, it is necessary to extract screenshots of all sides of the vehicle for subsequent manual inspection. Due to the long length of the truck itself and the insufficient lane width, the side camera cannot capture a complete picture of the vehicle. It is necessary to use image stitching for the gate and other means to obtain a complete side picture of the vehicle. The existing stitching method is based on motion estimation. The previous frame is used as a reference to calculate the amount of motion of the object between the current frame and the previous frame. Based on this amount of motion, it is calculated how wide a picture needs to be taken from the current frame and arranged behind the previous frame. This process is repeated until the vehicle leaves and the stitching process is completed.

[0003] The motion estimation currently used is based on the disparity estimation algorithm. The previous frame and the current frame are used as the left and right images respectively, and the binocular camera is simulated to calculate the difference between the two frames, thereby calculating the amount of motion. This method has the following problems: First, under special conditions such as bad weather, the disparity calculation will not get the expected results, resulting in poor quality of the spliced ​​image; in addition, the method of disparity calculation is to extract features manually, and it cannot be adapted to various scenes through training methods; thirdly, since the binocular camera is generally arranged horizontally, the disparity can only calculate the amount of horizontal motion, but not the amount of vertical motion. When the vehicle moves, there will be ups and downs, resulting in a small amount of vertical motion of the vehicle. If the vertical motion is not considered, the spliced ​​image will have jagged edges, which cannot be solved by the disparity method.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0005] In view of the problems in the prior art, the purpose of the present application is to provide an image stitching method, system, device and storage medium for a gate, so as to improve the image stitching effect for the gate.

[0006] The present application provides an image stitching method for a gate, comprising the following steps:

[0007] The camera set at the gate position collects images in real time;

[0008] Inputting the previous frame image and the current frame image into an optical flow estimation model, and determining the lateral motion data and the longitudinal motion data according to the output data of the optical flow estimation model;

[0009] Determine a starting point and an end point for extracting an image from the current frame image based on the lateral motion data;

[0010] Extracting a spliced ​​segment image from the current frame image based on a preset image extraction starting point and a width of the extracted image;

[0011] Processing the stitching segment images according to the longitudinal motion data to obtain images to be stitched;

[0012] The image to be spliced ​​is spliced ​​to one side of the output image corresponding to the previous frame image as the output image corresponding to the current frame image.

[0013] In some embodiments, determining the lateral motion data and the longitudinal motion data according to the output data of the optical flow estimation model includes:

[0014] Obtaining an output matrix of the optical flow estimation model;

[0015] The output matrix based on the optical flow estimation model is split into a lateral motion matrix and a longitudinal motion matrix, which are used as lateral motion data and longitudinal motion data respectively.

[0016] In some embodiments, determining the starting point and the end point of extracting an image from the current frame image based on the lateral motion data comprises the following steps:

[0017] Calculate the mean and standard deviation of the lateral movement amount based on the lateral movement data;

[0018] Determine the range of lateral processing amount according to the mean and standard deviation of the lateral movement amount;

[0019] Selecting the movement amount with the highest frequency within the range of the lateral processing amount as the lateral processing amount;

[0020] The starting point and the end point of the extracted image are determined according to the horizontal processing amount and the width of the current frame image.

[0021] In some embodiments, before calculating the mean and standard deviation of the lateral motion amount based on the lateral motion data, the following steps are also included:

[0022] The lateral motion data is processed based on a two-dimensional Gaussian distribution matrix.

[0023] In some embodiments, processing the stitching segment image according to the longitudinal motion data comprises the following steps:

[0024] Calculate the mean and standard deviation of the longitudinal motion amount based on the longitudinal motion data;

[0025] Determine the range of longitudinal offset values ​​according to the mean and standard deviation of the longitudinal motion;

[0026] Selecting the movement with the highest frequency in the longitudinal offset value range as the longitudinal offset;

[0027] A longitudinal offset process is performed on the spliced ​​segment image according to the longitudinal offset amount.

[0028] In some embodiments, before calculating the mean and standard deviation of the longitudinal motion amount based on the longitudinal motion data, the following steps are also included:

[0029] The longitudinal motion data is processed based on a two-dimensional Gaussian distribution matrix.

[0030] In some embodiments, the shooting direction of the camera is perpendicular to the vehicle passing direction at the gate position, and the images to be stitched are stitched into the output image, including stitching the images to be stitched to a side of the output image corresponding to the rear side of the vehicle driving direction.

[0031] In some embodiments, the following steps are also included:

[0032] Collecting a sample image set, taking two adjacent frames of images in the sample image set as a set of input data, and adding a horizontal motion data label and a vertical motion data label;

[0033] An optical flow estimation model is trained based on the sample image set and the corresponding lateral motion data labels and longitudinal motion data labels.

[0034] In some embodiments, for the first frame image, based on a preset image extraction starting point and a preset initial image extraction width, a spliced ​​segment image is extracted from the first frame image as an output image corresponding to the first frame image.

[0035] The embodiment of the present application further provides an image stitching system for a gate, which is used to implement the image stitching method for a gate, and the system includes:

[0036] An image acquisition module is used to acquire images in real time through a camera set at the gate position;

[0037] A model prediction module, used for inputting a previous frame image and a current frame image into an optical flow estimation model, and determining lateral motion data and longitudinal motion data according to output data of the optical flow estimation model;

[0038] An image extraction module, configured to determine a starting point and an end point for extracting an image from the current frame image based on the lateral motion data; extract a splicing segment image from the current frame image based on a preset image extraction starting point and a width of the extracted image; and process the splicing segment image according to the longitudinal motion data to obtain an image to be spliced;

[0039] The image stitching module is used to stitch the images to be stitched into an output image.

[0040] The embodiment of the present application also provides an image stitching device for a gate, including:

[0041] processor;

[0042] a memory storing executable instructions of the processor;

[0043] The processor is configured to execute the steps of the image stitching method for a gate by executing the executable instructions.

[0044] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the image stitching method for a gate are implemented.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0046] The image stitching method, system, device and storage medium for the gate of the present application have the following features:

[0047] Beneficial effects:

[0048] By adopting the image stitching method for the gate of the present application, the image at the gate is collected in real time, and the image extraction and stitching steps are repeated. The image to be stitched is extracted from each frame of the image and stitched in sequence. The side of the vehicle in the spliced ​​output image is complete, that is, after the vehicle leaves, a complete side image of the vehicle can be obtained. The present application can simultaneously obtain lateral and longitudinal motion data through the optical flow estimation method. It not only determines the position of extracting the stitching segment image based on the lateral motion data, but also processes the stitching segment image based on the longitudinal motion data to obtain the image to be stitched. It also considers the lateral and longitudinal motion factors, which can solve the jagged edge problem caused by longitudinal motion, improve the image stitching effect, and obtain more accurate vehicle side data, which is more conducive to the subsequent analysis and processing of the vehicle side image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0050] Figure 1 is a flow chart of an image stitching method for a gate according to an embodiment of the present application;

[0051] Figure 2 is a top view of a gate position of an embodiment of the present application;

[0052] Figures 3 to 6 is a schematic diagram of an image captured when a vehicle passes through a gate according to an embodiment of the present application;

[0053] Figure 7 and Figure 8 is a schematic optical flow diagram of two frames of images of an embodiment of the present application;

[0054] Fig. 9 is a structural schematic diagram of an image stitching system for a gate according to an embodiment of the present application;

[0055] Fig.10 is a structural schematic diagram of an image stitching device for a gate according to an embodiment of the present application;

[0056] Fig.11 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0058] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.

[0059] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined. Therefore, the actual execution order may be changed according to the actual situation.

[0060] like Figure 1 As shown, the embodiment of the present application provides an image stitching method for a gate, comprising the following steps:

[0061] S100: collecting images in real time through a camera set at the gate position;

[0062] Specifically, at least one camera is pre-installed on at least one side of the gate, and the shooting direction of the camera is perpendicular to the direction of the vehicle passing through the gate, and the image captured is a side image of the vehicle; since the vehicle at the gate is generally long, multiple images need to be captured and stitched together to obtain a complete vehicle image for subsequent further vehicle image processing procedures;

[0063] Figure 2 is a top view of the gate position of an embodiment of the present application, wherein the arrow indicates the direction of vehicle movement, and the front side of the vehicle is Figure 2 The right side of the vehicle, the rear side of the vehicle is Figure 2 On the left side of the vehicle, vehicle P3 is traveling in lane P2. Figure 2 The part of the roof of the vehicle P3 is shown in FIG. 1 . A camera P4 is arranged on at least one side of the gate position P1 . The camera P4 can take a picture of the body of the vehicle P3 .

[0064] The changes in the real-time images collected by the camera can be referred to Figures 3 to 6 As shown, P10 represents the image captured by the camera, and the arrow represents the direction of vehicle movement. Since the vehicle P3 is relatively long, the image captured by the camera is difficult to cover the entire side of the vehicle. In different frames of images, the side areas of the vehicle P3 captured are different. By extracting a section of multiple frames of images and then stitching them together, a complete side image of the vehicle P3 can be stitched together.

[0065] S200: inputting a previous frame image and a current frame image into an optical flow estimation model, and determining lateral motion data and longitudinal motion data according to output data of the optical flow estimation model;

[0066] Optical flow estimation is a method that uses the changes in pixels in an image sequence in the time domain and the correlation between adjacent frames to calculate the motion information of an object; this application uses the optical flow estimation method to simultaneously obtain the lateral motion data and longitudinal motion data of the vehicle;

[0067] S300: Determine the starting point and the end point of extracting an image from the current frame image based on the lateral motion data; the starting point and the end point here refer to the pixel position corresponding to the extraction starting point in the image and the pixel position corresponding to the extraction end point in the image; for example, when the lateral direction of the two-dimensional image is taken as the x-axis and the longitudinal direction is taken as the y-axis, determining the starting point and the end point is to determine the x-axis direction coordinate values ​​of the starting point and the end point;

[0068] S400: extracting a splicing segment image from the current frame image based on a preset image extraction starting point and a width of the extracted image;

[0069] by Figure 3For example, the P11 region level corresponding to the spliced ​​segment image extracted from the captured image P10 has a starting point corresponding to the x1 position and an end point corresponding to the x2 position;

[0070] S500: Processing the splicing segment images according to the longitudinal motion data to obtain an image to be spliced;

[0071] S600: Splicing the image to be stitched to one side of the output image corresponding to the previous frame image as the output image corresponding to the current frame image;

[0072] Specifically, the image to be spliced ​​is spliced ​​to a side of the previous output image corresponding to the rear side of the vehicle's driving direction, to obtain an output image corresponding to the current frame image.

[0073] By adopting the image stitching method for the gate of the present application, the image at the gate is collected in real time through step S100, and steps S200 to S600 are executed in a loop, and the image to be stitched is extracted from each frame of the image and stitched in sequence. The side of the vehicle in the spliced ​​output image is complete, that is, after the vehicle leaves, a complete side image of the vehicle can be obtained. The present application can simultaneously obtain lateral and longitudinal motion data through the optical flow estimation method, not only determine the position of extracting the stitching segment image based on the lateral motion data, but also process the stitching segment image based on the longitudinal motion data to obtain the image to be stitched, and consider the lateral and longitudinal motion factors at the same time, which can solve the jagged edge problem caused by longitudinal motion, improve the image stitching effect, and obtain more accurate vehicle side data, which is more conducive to the subsequent analysis and processing of the vehicle side image. By adopting the image stitching method of the present application, the box number in the obtained vehicle side image is completely visible, and the vehicle box number can be more accurately identified.

[0074] In the step S200, determining the lateral motion data and the longitudinal motion data according to the output data of the optical flow estimation model includes:

[0075] Obtaining an output matrix of the optical flow estimation model;

[0076] The output matrix based on the optical flow estimation model is split into a lateral motion matrix and a longitudinal motion matrix, which are used as lateral motion data and longitudinal motion data respectively.

[0077] In this embodiment, the input of the optical flow estimation model is two M*N*3 images, where M*N is the resolution of the image taken by the camera and 3 represents the RGB three channels. Based on the output of the optical flow estimation model, the following can be obtained: Figure 7 and Figure 8 Visualized optical flow map shown. Figure 7 and Figure 8Schematic optical flow diagrams of two frames of images are shown respectively, and the two frames of images are not adjacent frames. In the optical flow diagram, the white part indicates no motion, while the colored part indicates motion, and the darker the color, the greater the motion. Specifically, the output of the optical flow estimation model is an M*N*2 matrix, which can be divided into an M*N*1 matrix, which respectively represents the lateral motion and longitudinal motion of each pixel point of the current frame image relative to each pixel point of the previous frame image, that is, as lateral motion data and longitudinal motion data. The lateral motion data of a frame image includes the lateral motion of each pixel of the frame image compared to the previous frame image, and the longitudinal motion data of a frame image includes the longitudinal motion of each pixel of the frame image compared to the previous frame image.

[0078] When obtaining the lateral motion data of the current frame, considering that the amount of motion at each position of the vehicle is the same, but the amount of motion at different pixel positions estimated by the optical flow estimation model may be different. Therefore, it is necessary to calculate a value based on the lateral motion data to represent the lateral motion of the entire vehicle. In this embodiment, the step S300: determining the starting point and end point of extracting an image from the current frame image based on the lateral motion data includes the following steps:

[0079] Processing the lateral motion data based on a two-dimensional Gaussian distribution matrix;

[0080] That is, the captured image is weighted by a two-dimensional Gaussian distribution. Since the vehicle is always in the middle of the captured image and surrounded by the background, a pre-generated two-dimensional Gaussian distribution matrix is ​​multiplied by the lateral motion matrix to strengthen the central part and suppress the surrounding part. The values ​​in the two-dimensional Gaussian distribution matrix conform to the characteristics of the two-dimensional Gaussian distribution. Here, the two-dimensional Gaussian distribution matrix is ​​a matrix of the same size as the lateral motion matrix, in which the values ​​conform to the Gaussian distribution, the values ​​in the central area are the largest, and the data decrease toward the surrounding areas. The value range of the values ​​in the two-dimensional Gaussian distribution matrix is ​​pre-set, for example, [0,1] or other value ranges, and then a matrix is ​​constructed according to the value range so that each value in the two-dimensional Gaussian distribution matrix is ​​within the value range.

[0081] Calculate the mean and standard deviation of the lateral movement amount based on the lateral movement data;

[0082] Determine the range of lateral processing amount according to the mean and standard deviation of the lateral movement amount;

[0083] For example, let mean be the mean of the lateral motion and sigma be the standard deviation of the lateral motion. The range of the lateral processing amount d is: (mean-2*sigma, mean+2*sigma);

[0084] Select the movement with the highest frequency in the range of the lateral processing amount as the lateral processing amount d;

[0085] Determine the starting point and the end point of the extracted image according to the horizontal processing amount d and the width of the current frame image;

[0086] For example, assuming that the width of the current frame image is w, the starting point for extracting the image in the current frame image is (wd) / 2, and the end point is (w+d) / 2.

[0087] According to the central limit theorem, the captured image can be regarded as a large number of independent and identically distributed random variables, and the image can be considered to approximately conform to the Gaussian distribution; the mean (mean) and standard (sigma) of the optical flow map are calculated. According to the characteristics of the Gaussian distribution, when the interval (mean-2*sigma, mean+2*sigma) is selected, 95.44% of the points will fall in this interval, that is, the movement of most pixels will fall in this interval. Therefore, the starting point and end point of the value are set as mean-2*sigma and mean+2*sigma. If mean-2*sigma is negative, it is set to 0. Mean+2*sigma cannot exceed the preset upper limit to prevent problems in the calculation of a certain frame, resulting in image abnormalities.

[0088] In this embodiment, the processing method of the longitudinal motion data is similar to the processing method of the lateral motion data. The step S500: processing the splicing segment image according to the longitudinal motion data includes the following steps:

[0089] Processing the longitudinal motion data based on a two-dimensional Gaussian distribution matrix;

[0090] A pre-generated two-dimensional Gaussian distribution matrix is ​​multiplied by the longitudinal motion matrix to strengthen the central part and suppress the surrounding parts; the two-dimensional Gaussian distribution matrix is ​​a matrix of the same size as the longitudinal motion matrix, in which the values ​​conform to the Gaussian distribution, the values ​​in the central area are the largest, and the data decreases toward the surrounding areas. The value range of the values ​​in the two-dimensional Gaussian distribution matrix is ​​pre-set, for example, [0,1] or other value ranges, and then a matrix is ​​constructed according to the value range, so that each value in the two-dimensional Gaussian distribution matrix is ​​within the value range;

[0091] Calculate the mean and standard deviation of the longitudinal motion amount based on the longitudinal motion data;

[0092] Determining a range of longitudinal offset d' according to the mean and standard deviation of the longitudinal motion;

[0093] For example, let mean' be the mean of the longitudinal motion, sigma' be the standard deviation of the longitudinal motion, and the range of the longitudinal processing amount d' is: (mean'-2*sigma', mean'+2*sigma');

[0094] The splicing segment image is subjected to longitudinal offset processing according to the longitudinal offset d', thereby obtaining the image to be spliced. The longitudinal offset processing here is to perform an upward and downward offset on the splicing segment image by a corresponding value, and then splice it with the image to be spliced ​​of the previous frame, thereby solving the aliasing problem caused by upward and downward jitter during the splicing process.

[0095] In this embodiment, the above steps S200 to S600 are processing steps for other frame images except the first frame image. For the first frame image, based on a preset image extraction starting point and a preset initial image extraction width, a spliced ​​segment image is extracted from the first frame image as the output image corresponding to the first frame image. After the subsequent frame images are extracted to the image to be spliced, they are sequentially spliced ​​to the rear side of the output image. The step S600: splicing the image to be spliced ​​to the output image includes splicing the image to be spliced ​​to the side of the output image corresponding to the rear side of the vehicle's driving direction. After the vehicle finally leaves the shooting range of the camera, the output image obtained is the spliced ​​image, and the spliced ​​image is formed by sequentially splicing multiple images to be spliced.

[0096] In this embodiment, the optical flow estimation model is an optical flow estimation model based on deep learning. The image stitching method further includes training the optical flow estimation model by using the following steps:

[0097] Collecting a sample image set, taking two adjacent frames of images in the sample image set as a set of input data, and adding a horizontal motion data label and a vertical motion data label;

[0098] An optical flow estimation model is trained based on the sample image set and the corresponding lateral motion data labels and longitudinal motion data labels.

[0099] When collecting sample image sets, sample image sets in various different scenarios can be collected. For example, camera images taken in severe weather conditions can be collected as sample images, and lateral motion data labels and longitudinal motion data labels can be added through manual annotation, and model training can be performed to solve the problem of poor stitching effects in severe weather conditions, so that the optical flow estimation model can be applied to motion estimation in various scenarios.

[0100] like Fig. 9 As shown, the embodiment of the present application further provides an image stitching system for a gate, which is used to implement the image stitching method for a gate, and the system includes:

[0101] The image acquisition module M100 is used to acquire images in real time through a camera set at the gate position; specifically, at least one camera is pre-installed on at least one side of the gate position, and the image acquisition module M100 communicates with the camera to obtain each frame of images acquired in real time from the camera. The shooting direction of the camera is perpendicular to the passing direction of the vehicle at the gate position, and the image captured is a side image of the vehicle; since the vehicle at the gate is generally long, it is necessary to capture multiple images and stitch them together to obtain a complete vehicle image for subsequent further vehicle image processing procedures;

[0102] The model prediction module M200 is used to input the previous frame image and the current frame image into the optical flow estimation model, and determine the lateral motion data and the longitudinal motion data according to the output data of the optical flow estimation model;

[0103] The image extraction module M300 is used to determine the starting point and the end point of extracting an image from the current frame image based on the lateral motion data; extract a splicing segment image from the current frame image based on a preset image extraction starting point and the width of the extracted image; and process the splicing segment image according to the longitudinal motion data to obtain an image to be spliced; in this embodiment, the splicing segment image is processed according to the longitudinal motion data, that is, the splicing segment image is shifted up and down to solve the up and down jitter problem caused by the longitudinal motion;

[0104] The image stitching module M400 is used to stitch the image to be stitched into an output image; specifically, stitch the image to be stitched to a side of the previous output image corresponding to the rear side of the vehicle's driving direction to obtain an output image corresponding to the current frame image.

[0105] By adopting the image stitching system for the gate of the present application, the image at the gate is collected in real time through the image acquisition module M100, and the image extraction and stitching steps are cyclically executed through the model prediction module M200, the image extraction module M300, and the image stitching module M400, and the image to be stitched is extracted from each frame of the image and stitched in sequence. The side of the vehicle in the spliced ​​output image is complete, that is, after the vehicle leaves, a complete side image of the vehicle can be obtained. The present application can simultaneously obtain lateral and longitudinal motion data through the optical flow estimation method, not only determine the position of the extracted splicing segment image based on the lateral motion data, but also process the splicing segment image based on the longitudinal motion data to obtain the image to be stitched, and consider the lateral and longitudinal motion factors at the same time, which can solve the jagged edge problem caused by longitudinal motion, improve the image stitching effect, and obtain the vehicle side data More accurate, more conducive to the subsequent analysis and processing of the vehicle side image. Using the image stitching system of the present application, the box number in the obtained vehicle side image is completely visible, and the vehicle box number can be more accurately identified.

[0106] In the image stitching system for a gate of the present application, the functions of each module can be implemented by using the specific implementation of the image stitching method for a gate as described above, which will not be described in detail here.

[0107] An embodiment of the present application also provides an image stitching device for a gate, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to execute the steps of the image stitching method for the gate by executing the executable instructions.

[0108] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "platform" herein.

[0109] Refer to the following Fig.10 The electronic device 600 according to this embodiment of the present application is described. Fig.10 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0110] like Fig.10 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0111] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above-mentioned image stitching method for the gate of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0112] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0113] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0114] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0115] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0116] In the image stitching device for a gate, the program in the memory implements the steps of the image stitching method for a gate when executed by the processor. Therefore, the device can also obtain the technical effect of the image stitching method for a gate.

[0117] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the image stitching method for a gate are implemented. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present application described in the above-mentioned image stitching method for a gate of this specification.

[0118] refer to Fig.11As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be executed on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0119] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0120] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0121] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0122] When the program in the computer storage medium is executed by the processor, the steps of the image stitching method for the gate are implemented. Therefore, the computer storage medium can also obtain the technical effect of the image stitching method for the gate.

[0123] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. An image stitching method for a gate, characterized in that: The steps include: The camera set at the gate position collects images in real time; Inputting the previous frame image and the current frame image into an optical flow estimation model, and determining the lateral motion data and the longitudinal motion data according to the output data of the optical flow estimation model; Determine a starting point and an end point for extracting an image from the current frame image based on the lateral motion data; Extracting a spliced ​​segment image from the current frame image based on a preset image extraction starting point and a width of the extracted image; Processing the stitching segment images according to the longitudinal motion data to obtain images to be stitched; The image to be spliced ​​is spliced ​​to one side of the output image corresponding to the previous frame image as the output image corresponding to the current frame image.

2. The image stitching method for a gate according to claim 1, characterized in that: Determining the lateral motion data and the longitudinal motion data according to the output data of the optical flow estimation model includes: Obtaining an output matrix of the optical flow estimation model; The output matrix based on the optical flow estimation model is split into a lateral motion matrix and a longitudinal motion matrix, which are used as lateral motion data and longitudinal motion data respectively.

3. The image stitching method for a gate according to claim 1, characterized in that: Determining the starting point and the end point of extracting an image from the current frame image based on the lateral motion data comprises the following steps: Calculate the mean and standard deviation of the lateral movement amount based on the lateral movement data; Determine the range of lateral processing amount according to the mean and standard deviation of the lateral movement amount; Selecting the movement amount with the highest frequency within the range of the lateral processing amount as the lateral processing amount; The starting point and the end point of the extracted image are determined according to the horizontal processing amount and the width of the current frame image.

4. The image stitching method for a gate according to claim 3, characterized in that: Before calculating the mean and standard deviation of the lateral movement amount based on the lateral movement data, the following steps are also included: The lateral motion data is processed based on a two-dimensional Gaussian distribution matrix.

5. The image stitching method for a gate according to claim 1, characterized in that: Processing the spliced ​​segment image according to the longitudinal motion data comprises the following steps: Calculate the mean and standard deviation of the longitudinal motion amount based on the longitudinal motion data; Determine the range of longitudinal offset values ​​according to the mean and standard deviation of the longitudinal motion; Selecting the movement with the highest frequency in the longitudinal offset value range as the longitudinal offset; A longitudinal offset process is performed on the spliced ​​segment image according to the longitudinal offset amount.

6. The image stitching method for a gate according to claim 5, characterized in that: Before calculating the mean and standard deviation of the longitudinal motion amount based on the longitudinal motion data, the following steps are also included: The longitudinal motion data is processed based on a two-dimensional Gaussian distribution matrix.

7. The image stitching method for a gate according to claim 1, characterized in that: The shooting direction of the camera is perpendicular to the passing direction of the vehicle at the gate position, and the images to be stitched are stitched into the output image, including stitching the images to be stitched to a side of the output image corresponding to the rear side of the vehicle driving direction.

8. The image stitching method for a gate according to claim 1, characterized in that: The following steps are also included: Collecting a sample image set, taking two adjacent frames of images in the sample image set as a set of input data, and adding a horizontal motion data label and a vertical motion data label; An optical flow estimation model is trained based on the sample image set and the corresponding lateral motion data labels and longitudinal motion data labels.

9. The image stitching method for a gate according to claim 1, characterized in that: For the first frame image, based on a preset image extraction starting point and a preset initial image extraction width, a splicing segment image is extracted from the first frame image as an output image corresponding to the first frame image.

10. An image stitching system for a gate, characterized in that: The system is used to implement the image stitching method for a gate according to any one of claims 1 to 9, comprising: An image acquisition module is used to acquire images in real time through a camera set at the gate position; A model prediction module, used for inputting a previous frame image and a current frame image into an optical flow estimation model, and determining lateral motion data and longitudinal motion data according to output data of the optical flow estimation model; An image extraction module, configured to determine a starting point and an end point for extracting an image from the current frame image based on the lateral motion data; extract a splicing segment image from the current frame image based on a preset image extraction starting point and a width of the extracted image; and process the splicing segment image according to the longitudinal motion data to obtain an image to be spliced; The image stitching module is used to stitch the images to be stitched into an output image.

11. An image stitching device for a gate, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the image stitching method for a gate according to any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image stitching method for a gate described in any one of claims 1 to 9 are implemented.

Images