Image stitching method, device, equipment, medium and computer program product

Abstract

The invention provides an image splicing method, device and equipment, a medium and a computer program product, and relates to the technical field of image processing. The method comprises the following steps: acquiring two adjacent frames of original images; performing image segmentation on the two frames of original images to obtain a first scene segmentation image and a second scene segmentation image; extracting a first scene point from the first scene segmentation image, and extracting a second scene point from the second scene segmentation image; determining a first range based on the position of the first scene point in the first original image, and extracting a first feature point in the first range; determining a second range based on the position of the second scene point in the second original image, and extracting a second feature point in the second range; calculating a change matrix according to the first feature point and the second feature point; and splicing the two frames of original images based on the change matrix. According to the method, the calculated amount can be reduced, the real-time performance of image splicing of the vehicle-mounted device is improved, and the safety of the small unmanned vehicle is improved while the image splicing effect is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image stitching method, device, equipment, medium and computer program product. Background Art

[0002] Small robots are widely used in more and more fields, including scientific research, industrial production, and home services. Typical applications include sweeping robots and power inspection robots. The vehicle's external environment and road perception module is an important technical link in realizing the path planning of small robots. The vehicle's external environment and road perception module mainly includes obstacle detection, drivable area detection, etc., and the specific key technologies include: image segmentation, image stitching, and ranging.

[0003] Image stitching, especially panoramic image stitching, can provide robots with a wider perspective, more quickly and accurately detect obstacles around the robot that are difficult to detect, reduce the occurrence of equipment collisions, and greatly improve driving safety. The current image stitching technology generally extracts a large number of feature points from two images, finds the transformation model of the two images from these feature points, and then uses the transformation model to stitch the two images together. However, this method requires a large number of feature points, a large amount of calculation, and a lot of computing time, and requires a lot of storage resources.

[0004] However, in the field of path planning for small robots, it is difficult to quickly calculate large amounts of data due to the limited resources of small robots. Therefore, how to increase the speed of image stitching and ensure the effect of stitched images is a technical problem that urgently needs to be solved in this field. Summary of the invention

[0005] The present invention provides an image stitching method, device, equipment, medium and computer program product, which are used to solve the defects of the existing technology that the image stitching process in path planning of small and medium-sized robots is complicated, the calculation amount is large, and the time is long, so as to improve the image stitching speed while ensuring the stitching image effect.

[0006] The present invention provides an image stitching method, comprising the following steps.

[0007] Acquire two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; Performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; Extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; Based on the position of the first scene point in the first original image, a first range of a preset size is determined in the first original image, and a first feature point is extracted within the first range; based on the position of the second scene point in the second original image, a second range of the preset size is determined in the second original image, and a second feature point is extracted within the second range; Calculate a change matrix according to the first feature point and the second feature point; Based on the change matrix, the two frames of original images are spliced.

[0008] According to an image stitching method provided by the present invention, extracting a first scene point from the first scene segmentation map and extracting a second scene point from the second scene segmentation map include: Determine, from the first scene segmentation map, an edge region where a scene segmentation line is located; Extracting the first scene point from the edge area in the first scene segmentation map; calculating the first coordinate of the first scene point in the camera coordinate system; wherein the camera coordinate system is the camera coordinate system of the camera device; Calculating the corresponding position change amount of the unmanned vehicle within the time interval between the two adjacent frames of the original image; wherein the unmanned vehicle is the unmanned vehicle where the camera device is located; Based on the posture change amount and the first coordinate, calculate the second coordinate corresponding to the first scene point in the second scene segmentation map after the first scene point is transformed in position; Determine a second scene point in the second scene segmentation map based on the second coordinates.

[0009] According to an image stitching method provided by the present invention, the image segmentation is performed on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image, including: The two frames of original images are segmented respectively using the trained artificial intelligence model to obtain scene segmentation maps corresponding to the original images of each frame.

[0010] According to an image stitching method provided by the present invention, the posture change comprises an unmanned vehicle displacement change and an unmanned vehicle rotation angle change; the step of calculating, based on the posture change and the first coordinate, a second coordinate corresponding to the first scene point after position transformation in the second scene segmentation map comprises: Calculate the position of the first scene point in the second scene segmentation map according to the first coordinate and the displacement change of the unmanned vehicle; Calculate the angle of the first scene point in the second scene segmentation map according to the first coordinate and the change in the rotation angle of the unmanned vehicle; The position of the first scene point in the second scene segmentation map and the angle of the first scene point in the second scene segmentation map are used as the second coordinates.

[0011] According to an image stitching method provided by the present invention, the step of calculating the corresponding position change of the unmanned vehicle within the time interval between the two adjacent frames of original images being taken includes: According to the real-time posture acquired by the six-axis sensor on the unmanned vehicle and / or the stepping motor on the unmanned vehicle, the corresponding posture change amount of the unmanned vehicle when shooting the two adjacent frames of original images is calculated.

[0012] According to an image stitching method provided by the present invention, the camera device is a monocular camera device.

[0013] The present invention also provides an image stitching device, comprising the following modules: The original image acquisition module is used to acquire two adjacent frames of original images from the video stream captured by the camera device; wherein the two frames of original images include a first original image and a second original image; An image segmentation module, used to perform image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; A scene point extraction module, configured to extract a first scene point from the first scene segmentation map, and extract a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; a feature point extraction module, configured to determine a first range of a preset size in the first original image based on the position of the first scene point in the first original image, and extract a first feature point within the first range; determine a second range of the preset size in the second original image based on the position of the second scene point in the second original image, and extract a second feature point within the second range; A change matrix calculation module, used for calculating a change matrix according to the first feature point and the second feature point; An image stitching module is used to stitch the two frames of original images based on the change matrix.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned image stitching methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image stitching method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image stitching method described above is implemented.

[0017] The image stitching method, device, equipment, medium and computer program product provided by the present invention are as follows: obtaining two adjacent frames of original images from a video stream shot by a camera device; wherein the two frames of original images include a first original image and a second original image; performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; based on the position of the first scene point in the first original image, determining a first range of a preset size in the first original image, and extracting a first feature point within the first range; based on the position of the second scene point in the second original image, determining a second range of a preset size in the second original image, and extracting a second feature point within the second range; calculating a change matrix according to the first feature point and the second feature point; and stitching the two frames of original images based on the change matrix. Compared with the traditional method of extracting feature points from the whole image, this method first determines the local area range with spatial correlation on two adjacent frames of original images, then extracts feature points from the local area range, obtains a change matrix based on the feature points, and splices the two original images through the change matrix. Compared with the traditional method of extracting feature points based on the entire original image, this method can reduce the amount of calculation by about an order of magnitude, thereby increasing the speed of image processing and the real-time performance of image stitching of vehicle-mounted camera devices. When applied to small robots, it does not require a large amount of computing resources. While ensuring the image stitching effect, it can further improve the safety of unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the image stitching method provided by the present invention.

[0020] Figure 2 This is the second flow chart of the image stitching method provided by the present invention.

[0021] Figure 3 It is a schematic diagram of two adjacent frames of original images provided by the present invention.

[0022] Figure 4 It is a schematic diagram of a scene segmentation diagram corresponding to adjacent original images provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the edge area in the adjacent scene segmentation map provided by the present invention.

[0024] Figure 6 It is a schematic diagram of the local area range in the adjacent scene segmentation map provided by the present invention.

[0025] Figure 7 It is a schematic diagram of the final splicing effect of adjacent original images provided by the present invention.

[0026] Figure 8 It is a structural schematic diagram of the image stitching device provided by the present invention.

[0027] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Combine the following Figure 1-Figure 9 Specific embodiments of the present invention are described.

[0030] Figure 1It is one of the flowcharts of the image stitching method provided by the present invention, such as Figure 1 As shown, the method includes the following.

[0031] Step 101: Acquire two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image.

[0032] Specifically, this application is mainly used for path planning of small robots, especially the image stitching task of the on-board camera device carried by the small robot during the shooting process. The overall process of the image stitching process is as follows Figure 2 First, two adjacent frames of original images are obtained from the video stream captured by the vehicle-mounted camera device, which are the first original image and the second original image that are in a sequential relationship in terms of shooting time. The first original image and the second original image are shown in FIG. Figure 3 shown.

[0033] Optionally, in order to facilitate subsequent image processing, the original image may be preprocessed, including image denoising, sharpening, smoothing, etc.

[0034] Step 102, performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image.

[0035] Among them, image segmentation is an important task in digital image processing, and its goal is to divide an image into regions or objects with semantic or structural consistency.

[0036] Specifically, the two frames of original images or the preprocessed original images are segmented to obtain a first scene segmentation map and a second scene segmentation map, such as Figure 4 As shown in Figure 2, image segmentation methods include threshold segmentation, edge detection, and other methods.

[0037] Step 103, extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively.

[0038] Among them, the scene points are representative feature points selected from the two scene segmentation images.

[0039] Specifically, since the shooting time of the original images corresponding to the two frames of scene segmentation maps is very close, about 100ms-500ms, the scenes in the two frames of scene segmentation maps are similar. For the convenience of selection, the points in the edge area corresponding to the scene segmentation line can be selected as scene points, such as Figure 5As shown; and, the feature points selected on the two-frame scene segmentation map are the same physical entity points in reality.

[0040] It should be noted that in order to ensure that the scene points extracted on the two scene segmentation maps are mappings of the same physical entity points and are spatially related, the second scene point in this application is determined based on the first scene point and combined with the posture change when the unmanned vehicle takes the two images. The specific steps are described later.

[0041] Step 104: Based on the position of the first scene point in the first original image, determine a first range of a preset size in the first original image, and extract a first feature point within the first range; based on the position of the second scene point in the second original image, determine a second range of the preset size in the second original image, and extract a second feature point within the second range.

[0042] The existing image stitching mainly includes the following steps: image preprocessing, feature detection and feature description, feature matching, and image synthesis. Among them, feature detection and feature description, and feature matching are the most critical and core steps, which directly affect the final stitching effect.

[0043] This step is mainly to extract feature points. Although the first scene point and the second scene point are at different positions in the two-frame scene segmentation map, they correspond to the same physical entity in reality. Therefore, the position coordinates of the first scene point in the camera coordinate system can be calculated first, and then the coordinates of the first scene point in the second scene point after the position transformation are calculated according to the transformation of the position of the unmanned vehicle during the shooting time. Since the first scene point and the second scene point are the same physical entity points, the points near the positions of these two points on the original image are also related areas in reality and are spatially correlated. Accordingly, a first range of a preset size can be selected according to the coordinates of the first scene point on the first original image, and an area similar to the first range, i.e., a second range, can be selected according to the coordinates of the second scene point on the second original image. For example, Figure 6 As shown; the first feature point and the second feature point are extracted in the first range and the second range respectively.

[0044] Step 105: Calculate a change matrix according to the first feature point and the second feature point.

[0045] The change matrix refers to an image mapping relationship, which refers to a set of parameters by which the coordinate positions in one image are mapped to new coordinate positions in another image.

[0046] Specifically, since the first feature point and the second feature point are points with a correlation in the actual physical space, the spatial position transformation relationship of the pixel points in the image between the two frames of images, that is, the above-mentioned change matrix, can be calculated based on the position coordinates of the first feature point and the second feature point.

[0047] Step 106: splicing the two frames of original images based on the transformation matrix.

[0048] Specifically, the above change matrix is ​​applied to the entire image, that is, to all pixels of the first original image and the second original image, to obtain a spliced ​​image of the complete data, such as Figure 7 shown.

[0049] In the above embodiment, two adjacent frames of original images are obtained from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; image segmentation is performed on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; a first scene point is extracted from the first scene segmentation map, and a second scene point is extracted from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; based on the position of the first scene point in the first original image, a first range of a preset size is determined in the first original image, and a first feature point is extracted within the first range; based on the position of the second scene point in the second original image, a second range of a preset size is determined in the second original image, and a second feature point is extracted within the second range; a change matrix is ​​calculated based on the first feature point and the second feature point; based on the change matrix, the two frames of original images are spliced. Compared with the traditional method of extracting feature points from the whole image, this method first determines the local area range with spatial correlation on two adjacent frames of original images, then extracts feature points from the local area range, obtains a change matrix based on the feature points, and splices the two original images through the change matrix. Compared with the traditional method of extracting feature points based on the entire original image, this method can reduce the amount of calculation by about an order of magnitude. When applied to small robots, it does not require a large amount of computing resources, thereby improving the speed of image processing and the real-time performance of image stitching of the on-board camera device, thereby improving the safety of unmanned vehicles.

[0050] In one embodiment, the above-mentioned step 103 includes: determining the edge area where the scene segmentation line is located from the first scene segmentation map; extracting the first scene point from the edge area in the first scene segmentation map; calculating the first coordinate of the first scene point in the camera coordinate system; wherein the camera coordinate system is the camera coordinate system of the camera device; calculating the corresponding posture change of the unmanned vehicle where the camera device is located during the time interval between the two adjacent frames of original images; based on the posture change and the first coordinate, calculating the second coordinate corresponding to the first scene point in the second scene segmentation map after the position transformation; and determining the second scene point in the second scene segmentation map based on the second coordinate.

[0051] Specifically, since the scene segmentation map contains the scene segmentation line, the edge area can be selected according to the area where the scene segmentation line is located, such as Figure 5 As shown, the first scene point is extracted from the edge area in the first scene segmentation map, and the scene point at the junction of multiple different scenes is selected as much as possible. According to the camera posture of the camera device when shooting the first original image, the first coordinate of the first scene point M in the camera coordinate system is calculated, wherein the camera coordinate system takes the camera as the coordinate origin O (0,0,0), and the first coordinate is determined to be M (x1, y1, z1). According to the data fed back by the six-axis sensor and the data fed back by the stepper motor, the corresponding posture change of the unmanned vehicle where the camera device is located when shooting two frames of original images is calculated, and the position of the first scene point M on the second scene segmentation map after the position change is calculated according to the posture change.

[0052] The above-mentioned posture change includes the unmanned vehicle change (tx, ty, tz) and the unmanned vehicle rotation angle change θ; the specific steps of calculating the position of the above-mentioned first scene point M on the second scene segmentation map after the position change include: calculating the position of the first scene point in the second scene segmentation map according to the first coordinate and the unmanned vehicle displacement change; calculating the angle of the first scene point in the second scene segmentation map according to the first coordinate and the unmanned vehicle rotation angle change; taking the position of the first scene point in the second scene segmentation map and the angle of the first scene point in the second scene segmentation map as the second coordinate N (x2, y2, z2).

[0053] That is, the second coordinate N (x2, y2, z2) is: ; And the rotation change of the second coordinate: ; Among them, (tx, ty, tz) is the displacement change of the unmanned vehicle; θ is the rotation angle change of the unmanned vehicle.

[0054] The position on the second scene segmentation map is determined according to the second coordinates to obtain the second scene point N (x2, y2, z2).

[0055] Optionally, the above-mentioned posture change is obtained by calculating the posture change corresponding to the unmanned vehicle when shooting the two adjacent frames of original images based on the real-time posture obtained by the six-axis sensor on the unmanned vehicle and / or the stepper motor on the unmanned vehicle.

[0056] In the above embodiment, scene points on the edge area near the scene segmentation line are selected, and the position of the scene points on the second scene segmentation map after position conversion is determined based on the change in posture of the unmanned vehicle between adjacent picture frames, thereby providing a reliable basis for subsequently determining spatially related local areas on the two pictures.

[0057] In one embodiment, the above step 102 includes: using the trained artificial intelligence model to perform image segmentation on the two frames of original images respectively to obtain scene segmentation maps corresponding to each frame of the original image.

[0058] Specifically, the present application uses an AI image segmentation method to achieve image segmentation of the original image. The AI ​​image segmentation models used include but are not limited to: FCN (Fully Convolutional Networks), PSPNet (Pyramid Scene Parseing Network), etc.

[0059] The above-mentioned embodiment can realize fast segmentation of the original image by utilizing AI image segmentation, and can also realize fast segmentation of different scenes by training sample images of different scenes.

[0060] In one embodiment, the camera device is a monocular camera device.

[0061] Among them, a monocular camera refers to a device with only one camera lens.

[0062] In the above embodiment, by setting the camera device installed on the small robot as a monocular camera device, the manufacturing cost of the small robot can be saved.

[0063] The image stitching device provided by the present invention is described below. The image stitching device described below and the image stitching method described above can be referred to each other.

[0064] like Figure 8 As shown, the present application also provides an image stitching device, the device comprising: The original image acquisition module 801 is used to acquire two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; An image segmentation module 802 is used to perform image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; A scene point extraction module 803 is used to extract a first scene point from the first scene segmentation map, and extract a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; The feature point extraction module 804 is used to determine a first range of a preset size in the first original image based on the position of the first scene point in the first original image, and extract a first feature point within the first range; determine a second range of the preset size in the second original image based on the position of the second scene point in the second original image, and extract a second feature point within the second range; A change matrix calculation module 805, configured to calculate a change matrix according to the first feature point and the second feature point; The image stitching module 806 is used to stitch the two frames of original images based on the change matrix.

[0065] In one embodiment, the scene point extraction module 803 is further used to: Determine the edge area where the scene segmentation line is located from the first scene segmentation map; extract the first scene point from the edge area in the first scene segmentation map; calculate the first coordinate of the first scene point in the camera coordinate system; wherein the camera coordinate system is the camera coordinate system of the camera device; calculate the corresponding posture change of the unmanned vehicle within the time interval between the two adjacent frames of the original image; calculate the second coordinate of the first scene point in the second scene segmentation map after the position transformation based on the posture change and the first coordinate; determine the second scene point in the second scene segmentation map based on the second coordinate.

[0066] In one embodiment, the image segmentation module 802 is further used to: use the trained artificial intelligence model to perform image segmentation on the two frames of original images respectively, and obtain scene segmentation maps corresponding to the respective frames of original images.

[0067] In one embodiment, the posture change includes the displacement change of the unmanned vehicle and the rotation angle change of the unmanned vehicle; the scene point extraction module 803 is further used to: The position of the first scene point in the second scene segmentation map is calculated based on the first coordinate and the change in displacement of the unmanned vehicle; the angle of the first scene point in the second scene segmentation map is calculated based on the first coordinate and the change in rotation angle of the unmanned vehicle; the position of the first scene point in the second scene segmentation map and the angle of the first scene point in the second scene segmentation map are used as the second coordinate.

[0068] In one embodiment, the scene point extraction module 803 is further used to: According to the real-time posture acquired by the six-axis sensor on the unmanned vehicle and / or the stepping motor on the unmanned vehicle, the corresponding posture change amount of the unmanned vehicle when shooting the two adjacent frames of original images is calculated.

[0069] In one embodiment, the camera device is a monocular camera device.

[0070] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the image stitching method, which includes: obtaining two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; based on the position of the first scene point in the first original image, determining a first range of a preset size in the first original image, and extracting a first feature point within the first range; based on the position of the second scene point in the second original image, determining a second range of a preset size in the second original image, and extracting a second feature point within the second range; calculating a change matrix based on the first feature point and the second feature point; and stitching the two frames of original images based on the change matrix.

[0071] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image stitching method provided by the above methods, which includes: obtaining two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image, and a second scene segmentation map corresponding to the second original image; extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; based on the position of the first scene point in the first original image, determining a first range of a preset size in the first original image, and extracting a first feature point within the first range; based on the position of the second scene point in the second original image, determining a second range of a preset size in the second original image, and extracting a second feature point within the second range; calculating a change matrix based on the first feature point and the second feature point; and stitching the two frames of original images based on the change matrix.

[0073] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the image stitching method provided by the above methods when executed, the method comprising: obtaining two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image, and a second scene segmentation map corresponding to the second original image; extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; based on the position of the first scene point in the first original image, determining a first range of a preset size in the first original image, and extracting a first feature point within the first range; based on the position of the second scene point in the second original image, determining a second range of a preset size in the second original image, and extracting a second feature point within the second range; calculating a change matrix based on the first feature point and the second feature point; and stitching the two frames of original images based on the change matrix.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image stitching method, characterized in that: include: Acquire two adjacent frames of original images from a video stream captured by a camera device; wherein the two frames of original images include a first original image and a second original image; Performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; Extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; Based on the position of the first scene point in the first original image, a first range of a preset size is determined in the first original image, and a first feature point is extracted within the first range; based on the position of the second scene point in the second original image, a second range of the preset size is determined in the second original image, and a second feature point is extracted within the second range; Calculate a change matrix according to the first feature point and the second feature point; Based on the change matrix, the two frames of original images are spliced.

2. The image stitching method according to claim 1, characterized in that: The extracting a first scene point from the first scene segmentation map, and extracting a second scene point from the second scene segmentation map, comprises: Determine, from the first scene segmentation map, an edge region where a scene segmentation line is located; Extracting the first scene point from the edge area in the first scene segmentation map; calculating the first coordinate of the first scene point in the camera coordinate system; wherein the camera coordinate system is the camera coordinate system of the camera device; Calculating the corresponding position change amount of the unmanned vehicle within the time interval between the two adjacent frames of the original image; wherein the unmanned vehicle is the unmanned vehicle where the camera device is located; Based on the posture change amount and the first coordinate, calculate the second coordinate corresponding to the first scene point in the second scene segmentation map after the first scene point is transformed in position; Determine a second scene point in the second scene segmentation map based on the second coordinates.

3. The image stitching method according to claim 2, characterized in that: The posture change includes the displacement change of the unmanned vehicle and the rotation angle change of the unmanned vehicle; the calculating, based on the posture change and the first coordinate, the second coordinate corresponding to the first scene point after the position transformation in the second scene segmentation map includes: Calculate the position of the first scene point in the second scene segmentation map according to the first coordinate and the displacement change of the unmanned vehicle; Calculate the angle of the first scene point in the second scene segmentation map according to the first coordinate and the change in the rotation angle of the unmanned vehicle; The position of the first scene point in the second scene segmentation map and the angle of the first scene point in the second scene segmentation map are used as the second coordinates.

4. The image stitching method according to claim 2, characterized in that: The calculating of the position change of the unmanned vehicle corresponding to the time interval between the two adjacent frames of original images includes: According to the real-time posture acquired by the six-axis sensor on the unmanned vehicle and / or the stepping motor on the unmanned vehicle, the corresponding posture change amount of the unmanned vehicle when shooting the two adjacent frames of original images is calculated.

5. The image stitching method according to claim 1, characterized in that: The step of performing image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image includes: The two frames of original images are segmented respectively using the trained artificial intelligence model to obtain scene segmentation maps corresponding to the original images of each frame.

6. The image stitching method according to any one of claims 1 to 5, characterized in that: The camera device is a monocular camera device.

7. An image stitching device, characterized in that: include: The original image acquisition module is used to acquire two adjacent frames of original images from the video stream captured by the camera device; wherein the two frames of original images include a first original image and a second original image; An image segmentation module, used to perform image segmentation on the two frames of original images respectively to obtain a first scene segmentation map corresponding to the first original image and a second scene segmentation map corresponding to the second original image; A scene point extraction module, configured to extract a first scene point from the first scene segmentation map, and extract a second scene point from the second scene segmentation map; wherein the first scene point and the second scene point are mapping points of the same physical entity point in the first scene segmentation map and the second scene segmentation map respectively; a feature point extraction module, configured to determine a first range of a preset size in the first original image based on the position of the first scene point in the first original image, and extract a first feature point within the first range; determine a second range of the preset size in the second original image based on the position of the second scene point in the second original image, and extract a second feature point within the second range; A change matrix calculation module, used for calculating a change matrix according to the first feature point and the second feature point; An image stitching module is used to stitch the two frames of original images based on the change matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image stitching method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image stitching method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image stitching method according to any one of claims 1 to 6 is implemented.

Images