Image stitching method, device, equipment, and storage medium

By using the lateral offset point of the slit position as a reference point in the vehicle driving situation, the target slit position is redetermined, and the problem of misalignment or ghosting of the target object under the fixed slit strategy is solved, achieving better vision Experience.

CN112184541BActive Publication Date: 2024-11-26HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910604336.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-07-05
Publication Date
2024-11-26
Estimated Expiration
2039-07-05

AI Technical Summary

Technical Problem

In vehicle driving situations, parallax changes caused by changes in relative posture of the camera cause problems of misalignment or ghosting when the fixed slit strategy splices the image.

Method used

When determining the intersection of the target object and the seam position in the preset panoramic stitching map, the point after the offset side of the seam position is distributed as a reference point, and the target is re-determined based on the reference point and seam position information. The position of the slit is spliced ​​and two sub-maps of the target object are spliced ​​at this position, so that the target object is distributed in one of the sub-maps of the splicing map.

Benefits of technology

It avoids misalignment or ghosting on both sides of the seams in the stitching map, improves the visual effect and ensures the integrity and clarity of the stitching map.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112184541B_ABST
    Figure CN112184541B_ABST
Patent Text Reader

Abstract

The present invention provides an image stitching method, device, equipment, and storage medium, the method comprising: when a target object intersects with a stitching position in a preset panoramic stitching image, taking a point of the target object distributed on the target side along a point offset away from the stitching position as a reference point, the target side being one of the two sides of the stitching position; determining the target stitching position based on the position information of the reference point and the stitching position information; stitching two sub-images of the target object at the target stitching position, so that the target object is distributed in one of the sub-images of the stitching image. The stitching can bypass the target object, avoiding the problem of dislocation or ghosting of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image technology, and in particular to an image stitching method, device and equipment, and a storage medium. Background Art

[0002] In some situations, such as when driving a vehicle, it is necessary to stitch together sub-images collected from different directions of the vehicle. The resulting image will have a larger field of view and a more complete scene than the sub-images. For example, sub-images collected from all directions around the vehicle can be stitched together to obtain a panoramic image that presents an all-round scene.

[0003] In the related stitching method, a fixed stitching strategy is used to stitch images, that is, no matter how the target objects are distributed in the global stitching map, the stitching position in the global stitching map is fixed, and the two related sub-images are always stitched at this stitching position.

[0004] However, when the relative posture of two adjacent cameras changes (for example, when two cameras are installed on a vehicle and the vehicle body deforms to a certain extent due to external forces, the relative posture of the two adjacent cameras will change), the parallax of the two cameras will change. When the target objects are distributed on both sides of the seam position in the global stitching image, the above-mentioned fixed seam strategy is used to stitch the sub-images captured by the two cameras. Misalignment will occur during stitching, and the target objects will be distributed on both sides of the seam of the stitching image, which will cause misalignment or ghosting and affect the visual experience. Summary of the invention

[0005] In view of this, the present invention provides an image stitching method, device, equipment, and storage medium, which can make the stitching seam bypass the target object and avoid the problem of misalignment or ghosting of the target object.

[0006] A first aspect of the present invention provides an image stitching method, comprising:

[0007] When the target object intersects with the stitching position in the preset panoramic stitching image, the points of the target object distributed on the target side are offset away from the stitching position as reference points, and the target side is one of the two sides of the stitching position;

[0008] Determine the target seam position based on the reference point position information and the seam position information;

[0009] The two sub-images of the target object are spliced ​​at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

[0010] According to an embodiment of the present invention, the reference point at least includes a point whose distance from the seam position is greater than or equal to the distance between the pole and the seam position;

[0011] The extreme point is the point farthest from the seam position among all points of the target object distributed on the target side.

[0012] According to one embodiment of the present invention,

[0013] The reference points also include points different from the extreme points along the edge of the target object among all points where the target object is distributed on the target side.

[0014] According to an embodiment of the present invention, determining a target seam position based on position information of a reference point and seam position information includes:

[0015] Determining the position information of the target point on the target side according to the position information of the reference point, wherein the distance between the target point and the seam position is greater than the distance between the reference point and the seam position;

[0016] The target seam position is determined according to the position information of the target point and the position information of the start and end points of the seam position.

[0017] According to an embodiment of the present invention, determining the position information of the target point on the target side according to the position information of the reference point includes:

[0018] Checking whether the reference point is between the stitching position and a specified boundary, wherein the specified boundary is a boundary of a stitchable range located on the target side;

[0019] If yes, the position information of one point between the reference point and the target boundary point on the designated boundary is determined as the position information of the target point, and the target boundary point is the point closest to the reference point and the designated boundary.

[0020] According to an embodiment of the present invention, determining the target seam position according to the position information of the target point and the position information of the start and end points of the seam position includes:

[0021] Performing curve fitting on the target point and the start and end points of the seam position according to the position information of the target point and the position information of the start and end points of the seam position to obtain a fitting curve;

[0022] The position of the fitting curve is determined as the target seam position.

[0023] According to one embodiment of the present invention, before taking the points of the target object distributed on the target side and shifted away from the seam position as reference points, the method further comprises:

[0024] Calculate the distance between each point of the target object distributed in the global splicing graph and the splicing position;

[0025] Determine a first side where the farthest point on both sides of the seam position is located, the farthest point being the point where the distance between each point of the target object and the seam position is the largest;

[0026] The other side of the first side of the two sides of the seam position is used as the target side of the seam.

[0027] According to one embodiment of the present invention, the method further comprises:

[0028] Acquire the collected location data of the target object;

[0029] Determine a position information set corresponding to the position data of the target object in the global mosaic graph, wherein the position information set includes position information of all points of the target object distributed in the global mosaic graph;

[0030] Check whether there is an intersection between the position information set and the seam position in the global splicing graph.

[0031] A second aspect of the present invention is an image stitching device, characterized by comprising:

[0032] A reference point determination module, for, when a target object intersects with a preset stitching position in a panoramic stitching image, using points of the target object distributed on a target side offset away from the stitching position as reference points, wherein the target side is one of the two sides of the stitching position;

[0033] A target seam position determination module, used to determine the target seam position based on the position information of the reference point and the seam position information;

[0034] The sub-image splicing module is used to splice two sub-images of the target object at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

[0035] According to an embodiment of the present invention, the reference point at least includes a point whose distance from the seam position is greater than or equal to the distance between the pole and the seam position;

[0036] The extreme point is the point farthest from the seam position among all points of the target object distributed on the target side.

[0037] According to one embodiment of the present invention,

[0038] The reference points also include points different from the extreme points along the edge of the target object among all points where the target object is distributed on the target side.

[0039] According to an embodiment of the present invention, when the target seam position determination module determines the target seam position based on the position information of the reference point and the seam position information, it is specifically used to:

[0040] Determining the position information of the target point on the target side according to the position information of the reference point, wherein the distance between the target point and the seam position is greater than the distance between the reference point and the seam position;

[0041] The target seam position is determined according to the position information of the target point and the position information of the start and end points of the seam position.

[0042] According to an embodiment of the present invention, when the target seam position determination module determines the position information of the target point on the target side according to the position information of the reference point, it is specifically used to:

[0043] Checking whether the reference point is between the stitching position and a specified boundary, wherein the specified boundary is a boundary of a stitchable range located on the target side;

[0044] If yes, the position information of one point between the reference point and the target boundary point on the designated boundary is determined as the position information of the target point, and the target boundary point is the point closest to the reference point and the designated boundary.

[0045] According to an embodiment of the present invention, when the target seam position determination module determines the target seam position according to the position information of the target point and the position information of the start and end points of the seam position, it is specifically used to:

[0046] Performing curve fitting on the target point and the start and end points of the seam position according to the position information of the target point and the position information of the start and end points of the seam position to obtain a fitting curve;

[0047] The position of the fitting curve is determined as the target seam position.

[0048] According to an embodiment of the present invention, before the reference point determination module uses the point of the target object distributed on the target side and offset away from the seam position as the reference point, it is further used to:

[0049] Calculate the distance between each point of the target object distributed in the global splicing graph and the splicing position;

[0050] Determine a first side where the farthest point on both sides of the seam position is located, the farthest point being the point where the distance between each point of the target object and the seam position is the largest;

[0051] The other side of the first side of the two sides of the seam position is used as the target side of the seam.

[0052] According to one embodiment of the present invention, the device further comprises:

[0053] A position data acquisition module, used to acquire the collected position data of the target object;

[0054] A position information set determination module, configured to determine a position information set corresponding to the position data of the target object in the global mosaic graph, wherein the position information set includes position information of all points of the target object distributed in the global mosaic graph;

[0055] A checking module is used to check whether there is an intersection between the position information set and the seam position in the global splicing graph.

[0056] A third aspect of the present invention provides an electronic device, comprising a processor and a memory; the memory stores a program that can be called by the processor; wherein when the processor executes the program, the image stitching method described in the above embodiment is implemented.

[0057] A fourth aspect of the present invention provides a machine-readable storage medium having a program stored thereon, which, when executed by a processor, implements the image stitching method as described in the aforementioned embodiment.

[0058] The embodiments of the present invention have the following beneficial effects:

[0059] In an embodiment of the present invention, when a target object intersects with a preset seam position in a panoramic stitching image, a point on the target side of the target object that is offset away from the seam position is used as a reference point. Based on the position information of the reference point and the seam position information, the seam position required for stitching two sub-images of the target object, that is, the target seam position, can be re-determined. The two sub-images of the target object are stitched at the target seam position, so that the target object is distributed in one of the sub-images of the stitching image, thereby avoiding the same target object being distributed on both sides of the seam in the stitching image, that is, the seam in the stitching image will bypass the target object, thereby avoiding dislocation or ghosting of the target object and improving the visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flowchart of an image stitching method according to an embodiment of the present invention;

[0061] Figure 2 is a structural block diagram of an image stitching device according to an embodiment of the present invention;

[0062] Figure 3 It is a schematic diagram of a two-dimensional image spliced ​​by using a fixed seam strategy;

[0063] Figure 4 is a schematic diagram of a spliced ​​two-dimensional image in one embodiment of the present invention;

[0064] Figure 5 It is a schematic diagram of a three-dimensional image spliced ​​by using a fixed seam strategy;

[0065] Figure 6 is a schematic diagram of a spliced ​​three-dimensional image in one embodiment of the present invention;

[0066] Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0068] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more related listed items.

[0069] It should be understood that although the terms first, second, third, etc. may be used to describe various devices in the present invention, this information should not be limited to these terms. These terms are only used to distinguish devices of the same type from each other. For example, without departing from the scope of the present invention, a first device may also be referred to as a second device, and similarly, a second device may also be referred to as a first device. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0070] The image stitching method of the embodiment of the present invention can be applied to assistive driving, such as in a vehicle-mounted surround view system, which can include a number of camera devices arranged around the vehicle, each of which captures sub-images of different local scenes, and a processor in the vehicle-mounted surround view system stitches the captured sub-images, so that people in the vehicle can see the panoramic view around the vehicle through the stitched image, so as to assist driving. Of course, the scenes in the fields of view of adjacent camera devices may intersect.

[0071] Of course, the image stitching method of the embodiment of the present invention can also be applied to other occasions where image stitching is required. The above-mentioned auxiliary driving occasion is only an example and is not a limitation on the application occasions of the image stitching method of the present invention.

[0072] The following is a more detailed description of the image stitching method of an embodiment of the present invention, but it should not be limited to this. In one embodiment, refer to Figure 1 , shows an image stitching method according to an embodiment of the present invention, comprising:

[0073] S100: when the target object intersects with a preset stitching position in the panoramic stitching image, taking points of the target object distributed on the target side that are offset away from the stitching position as reference points, the target side being one of the two sides of the stitching position;

[0074] S200: Determine a target seam position based on the position information of the reference point and the seam position information;

[0075] S300: splicing two sub-images of the target object at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

[0076] The execution subject of the image stitching method of the embodiment of the present invention is an electronic device. The electronic device can be a computer device, an embedded device, etc., which can be connected to a camera device and receive images captured by the camera device as sub-images to be stitched. The electronic device can also be a camera device, which uses images captured by itself and / or obtained from other camera devices as sub-images to be stitched. The specific type of the electronic device is not limited, as long as it has image processing capabilities.

[0077] In one method of obtaining sub-images, multiple cameras can be installed on a target object at different installation angles, and images are collected from multiple directions around the target object (such as front, back, left, right, top, bottom, etc.) to obtain sub-images of local scenes in various directions. It can be understood that the collection and splicing of sub-images can be performed continuously at a certain frequency.

[0078] For example, the target object is a vehicle, and there are four cameras installed around the vehicle. The four cameras simultaneously capture images from the left, front, right, and rear of the vehicle. The captured sub-images include the scene on the left side of the vehicle, the scene in front of the vehicle, the scene on the right side of the vehicle, and the scene behind the vehicle. By stitching together the sub-images, a panoramic image of the scenes around the vehicle can be obtained.

[0079] There is a certain overlap in the fields of view of adjacent cameras. If the target object is in the overlapping space of the fields of view of adjacent cameras, then the target object will be (partially or completely) distributed in the sub-images captured by the adjacent cameras.

[0080] In this case, if a fixed seam strategy is used for stitching, the seam in the stitched image may pass through the target object, that is, the target object will be distributed on both sides of the seam in the stitched image. When the relative posture of adjacent camera devices of the vehicle changes due to external force, the target object will be misplaced or have a double image on both sides of the seam in the stitched image. The embodiment of the present invention can solve the above problem.

[0081] In step S100, when the target object intersects with a preset stitching position in the panoramic stitching image, points of the target object distributed on the target side that are offset away from the stitching position are used as reference points, and the target side is one of the two sides of the stitching position.

[0082] Before each stitching, it can be checked whether the target object intersects with the preset stitching position in the panoramic stitching image. If not, the two sub-images to be stitched can be stitched at the stitching position; if so, it means that the target object is distributed in the two sub-images to be stitched. If stitching at the stitching position may cause the target object to be distributed in the two sub-images of the stitching image, that is, the stitching of the stitching image may pass through the target object. Therefore, in an embodiment of the present invention, when the two sub-images are stitched, a new stitching position will be determined. The target object can be a pedestrian, a vehicle, a lane line, or other object, without limitation.

[0083] In the embodiment of the present invention, the points and stitching positions of the target object may be in the same coordinate system, such as the coordinate system used for the panoramic stitching image, and correspondingly, the determined reference points and target stitching position information may also be in the coordinate system.

[0084] The reference point can be determined from the target side of the seam, and the points of the target object distributed on the target side are offset away from the seam position as the reference point. The offset point on the target object can be a point on the edge of the target object, and the number can be one or more than two. Correspondingly, the number of reference points is also one or more than two. The target side of the seam is one of the two sides of the seam position. The target side can be specified or determined according to a preset rule, and is not specifically limited.

[0085] In step S200, the target seam position is determined based on the position information of the reference point and the seam position information.

[0086] The reference point and the seam position information are known, and a line can be determined based on this, and the position of the line is used as the target seam position. For example, the line can have the starting and ending points of the seam as two endpoints and pass through the reference point. Of course, the specific determination method is not limited.

[0087] Since the reference point is the point of the target object distributed on the target side after being offset away from the seam position, the reference point farthest from the seam position must have no intersection with all points of the target object before offset. Therefore, based on the position information of the reference point and the seam position information, the target seam position that has no intersection with all points of the target object before offset can be determined, ensuring that when two sub-images are spliced ​​at the target seam position, the seam in the spliced ​​image can bypass the target object.

[0088] In step S300, two sub-images of the target object are spliced ​​at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

[0089] The target object intersects with the preset stitching position in the panoramic stitching image, indicating that the target object has entered the field of view of two adjacent cameras. The target object is distributed in the sub-images captured by the two cameras. The two sub-images of the target object are stitched at the target stitching position, so that the target object is distributed in one of the sub-images of the stitched stitching image.

[0090] A panoramic mosaic image can be formed by stitching together two or more sub-images. The above steps of the embodiment of the present invention are applicable to any two sub-images to be stitched together that have the same target object distributed therein.

[0091] Since the sub-images collected by different cameras may differ in brightness, chromaticity, etc., the two sub-images of the mosaic may differ accordingly, so obvious mosaic traces may appear. The mosaic can be processed by image fusion, color balancing, etc. to reduce the mosaic traces and make the two sub-images of the mosaic transition smoothly. In addition, the vehicle's driving trajectory and virtual radar obstacle board (indicating the obstacle ahead) can be drawn into the mosaic and displayed together.

[0092] As the sub-images are collected and stitched, the panoramic stitching image is also updated at a certain frequency, and the updated panoramic stitching image can be displayed in real time on the vehicle. The electronic device can transmit the panoramic stitching image to a display device such as a vehicle display device or a mobile phone display screen for display; or, if the electronic device has a display device, the panoramic stitching image can also be displayed on the electronic device.

[0093] In an embodiment of the present invention, when a target object intersects with a preset seam position in a panoramic stitching image, a point on the target side of the target object that is offset away from the seam position is used as a reference point. Based on the position information of the reference point and the seam position information, the seam position required for stitching two sub-images of the target object, that is, the target seam position, can be re-determined. The two sub-images of the target object are stitched at the target seam position, so that the target object is distributed in one of the sub-images of the stitching image, thereby avoiding the same target object being distributed on both sides of the seam in the stitching image, that is, the seam in the stitching image will bypass the target object, thereby avoiding dislocation or ghosting of the target object and improving the visual effect.

[0094] In one embodiment, the above method flow may be executed by the image stitching device 100, such as Figure 2 As shown, the image stitching device 100 mainly includes three modules: a reference point determination module 101, a target stitching position determination module 102 and a sub-image stitching module 103. The reference point determination module 101 is used to perform the above step S100, the target stitching position determination module 102 is used to perform the above step S200, and the sub-image stitching module 103 is used to perform the above step S300.

[0095] In one embodiment, the reference point at least includes a point whose distance from the seam position is greater than or equal to the distance between the pole point and the seam position;

[0096] The extreme point is the point farthest from the seam position among all points of the target object distributed on the target side.

[0097] In other words, in this embodiment, only the pole of the target object can be offset away from the seam position to obtain the corresponding reference point. Since the pole is the point farthest from the seam position among all the points of the target object distributed on the target side, the obtained reference point is farther away from the seam position, ensuring that the target seam position determined thereby can avoid all points before the target object is offset.

[0098] Optionally, the reference point also includes a point different from the pole among all points of the target object distributed on the target side along the edge of the target object.

[0099] In other words, the reference points may include all points along the edge of the target object among all points of the target object distributed on the target side, that is, the entire contour of the target object on the target side is offset away from the seam position, and a series of reference points are obtained to further ensure that the target seam position determined thereby can avoid all points before the target object is offset.

[0100] In this embodiment, these reference points depict the contour of the target object in the target side, so the corresponding target seam position determined according to these reference points can also surround the contour of the target object to prevent the target seam position from exceeding the splicing range.

[0101] In one embodiment, in step S200, determining the target seam position based on the position information of the reference point and the seam position information may include the following steps:

[0102] S201: determining the position information of the target point at the target side according to the position information of the reference point, wherein the distance between the target point and the seam position is greater than the distance between the reference point and the seam position;

[0103] S202: Determine a target seam position according to the position information of the target point and the position information of the start and end points of the seam position.

[0104] The position information of the target point is determined based on the position information of the reference point. Both the reference point and the target point are located on the target side. The distance between the target point and the seam position is greater than the distance between the reference point and the seam position. Of course, the target point should not exceed the splicing range for splicing two sub-images.

[0105] The target point and the reference point may correspond to each other. When there are multiple reference points, there are also multiple target points. If the target point and the start and end points of the seam position are known, the target seam position can be determined based on these points. For example, the position of the connecting line of these points can be used as the target seam position, or the position of the line obtained by fitting these points can be used as the target seam position. The specific method is not limited.

[0106] Combine the following Figure 3-6 The difference between the embodiment of the present invention and the splicing method using the fixed seam strategy will be described.

[0107] Figure 3 The two-dimensional image is spliced ​​by using the fixed seam strategy in the existing method, and the seam position is fixed and unchanged. Figure 3 The positions of lines A1, B1, C1, and D1 are shown, where A1 is a straight line connecting M and N. If the target object S intersects with the stitching position in the preset panoramic stitching image, stitching is still performed to obtain a panoramic stitching image composed of sub-images LEFT1, FRONT1, RIGHT1, and REAR1, and the target object S is distributed on both sides of the stitching A1.

[0108] Figure 4 FIG. 4 shows a two-dimensional image spliced ​​in an embodiment of the present invention. The position of the splicing seam can be adjusted according to the distribution of the target object. Figure 4The positions of lines A2, B2, C2, and D2 in the image are taken as an example. Taking A2 as an example, when the target object S intersects with the preset seam position (the position of the straight line connecting M and N) in the panoramic mosaic image, the mosaic is no longer made at the position of the straight line connecting M and N, but is made at the position of the curve A2 adjusted according to the target object S, and a panoramic mosaic image composed of sub-images such as LEFT2, FRONT2, RIGHT2, and REAR2 is obtained. The target object S is only distributed on one side of the seam A2 in the panoramic mosaic image, ensuring that the target object S is completely presented in one sub-image.

[0109] Figure 5 The 3D image is spliced ​​by using the fixed seam strategy in the existing method, and the position of the seam is fixed and unchanged. Figure 5 The positions of the lines A3, B3, C3, and D3 are shown, where A3 is the arc connecting M' and N'. If the target object S intersects with the stitching position in the preset panoramic stitching image, stitching is still performed to obtain a panoramic stitching image composed of sub-images such as LEFT3, FRONT3, RIGHT3, and REAR3, and the target object S is distributed on both sides of the stitching A3.

[0110] Figure 6 FIG. 4 shows a three-dimensional image spliced ​​in an embodiment of the present invention. The splicing position can be adjusted according to the distribution of the target object. Figure 4 The positions of lines A4, B4, C4, and D4 in the image are taken as an example. Taking A4 as an example, when the target object S intersects with the preset seam position in the panoramic mosaic image (the position of the arc connected by M' and N'), it is no longer stitched at the position of the arc connected by M' and N', but stitched at the position of an irregular curve A4 that bypasses the target object S with M' and N' as the starting and ending points, and a panoramic mosaic image composed of sub-images such as LEFT4, FRONT4, RIGHT4, and REAR4 is obtained. The target object S is only distributed on one side of the seam A4 in the panoramic mosaic image, ensuring that the target object S is completely presented in one sub-image.

[0111] Combine the following Figure 4 The image stitching method of this embodiment may include the following steps:

[0112] First determine the reference point: Figure 4 Taking the horizontal direction of the target object S as the X-axis and the vertical direction as the Y-axis as an example, S1 and S2 are two points on the contour of the target object S, S1 is represented by (x1′, y1′), S2 is represented by (x2′, y2′), the maximum coordinate value of all points on the target object S on the X-axis is x2′, and the maximum coordinate value on the Y-axis is y1′, construct P(x2′, y1′), and P can be used as a reference point required by this embodiment (that is, a point whose distance from the seam position is greater than or equal to the distance between the pole and the seam position);

[0113] Next, determine the target point: take P as a reference to obtain P1 (x′, y′) which is farther from the straight line connecting M and N. This P1 can be used as the target point required in this embodiment;

[0114] Next, determine the target seam position: the position information of the start and end points of the seam is M(x0, y0) and N(x1, y1). The target seam position, i.e. the position of line A2, can be obtained by curve fitting M(x0, y0), N(x1, y1) and P1(x′, y′).

[0115] Next, the two sub-images of the target object are spliced ​​at A2, and the seam in the spliced ​​image bypasses the target object S.

[0116] In the above method, only one reference point is used. Of course, other reference points may also be used, such as points different from the poles along the edge of the target object among all points where the target object is distributed on the target side.

[0117] like Figure 6 As shown, the position of the seam in the global stitching graph is, for example, the arc line connected by M' and N'. The reference point can be a point on the edge of the target object among all points on the target object S located on the target side of the straight line connected by M' and N'. Accordingly, the target point around the edge can be determined; (x i ,y i , z i ) represents the location information of the reference point, i ranges from 1 to N, N is the number of reference points, (x i ,y i , z i ) corresponds to the position information of the target point (x′ i , y′ i , z i ), curve fitting is performed on M', N' and each target point to obtain the target seam position of seam A4, and seam A4 bypasses the target object S.

[0118] It can be understood that the above method can be used to stitch three-dimensional images or two-dimensional images. This is just an example and not a limitation.

[0119] In one embodiment, in step S201, determining the position information of the target point on the target side according to the position information of the reference point includes:

[0120] S2011: Check whether the reference point is between the stitching position and a specified boundary, where the specified boundary is a boundary of a stitchable range on the target side;

[0121] S2012: If yes, determine the position information of one point between the reference point and the target boundary point on the designated boundary as the position information of the target point, and the target boundary point is the point that is closest to the reference point and the designated boundary.

[0122] Check whether the reference point is between the seam position and the specified boundary. If so, it means that the reference point is within the splicing range. The target point can be determined from the splicing range, and then the new seam position is determined as the target seam position.

[0123] When determining the location information of the target point, the location information of one point between the reference point and the target boundary point on the specified boundary can be determined as the location information of the target point. The one point can be a point at a specified distance from the reference point, or can be any point, without specific limitation.

[0124] In this way, it can be ensured that the target point is also within the splicing range, and the final target seam position is also within the splicing range, achieving normal splicing.

[0125] When the number of reference points is greater than or equal to two, the above steps S2011 and S2012 can be performed for each reference point. The position information of a target point can be correspondingly determined based on the position information of each reference point. When multiple target points are used for curve fitting, the fitting effect is better.

[0126] Of course, if the reference point is not between the seam position and the specified boundary, it means that the reference point is not within the splicing range. In this case, the two sub-images can still be spliced ​​at the seam position.

[0127] In one embodiment, in step S202, determining the target seam position according to the position information of the target point and the position information of the start and end points of the seam position includes the following steps:

[0128] S2021: performing curve fitting on the target point and the start and end points of the seam position according to the position information of the target point and the position information of the start and end points of the seam position to obtain a fitting curve;

[0129] S2022: Determine the position of the fitting curve as the target seam position.

[0130] The number of target points may be one or more than two. Curve fitting is performed on the target points and the start and end points of the seam. The obtained fitting curve may bypass the target object. The position of the fitting curve is determined as the target seam position.

[0131] Curve fitting can be achieved by using least square method, B-spline fitting method and other algorithms. The specific algorithm is not limited as long as it can achieve fitting of the corresponding curve according to a number of points.

[0132] In one embodiment, in step S100, before taking the point of the target object distributed on the target side and shifted away from the seam position as the reference point, the method further includes:

[0133] S110: Calculating the distance between each point of the target object distributed in the global splicing graph and the seam position;

[0134] S120: Determine a first side where the farthest point on both sides of the seam position is located, the farthest point being the point where the distance between each point of the target object and the seam position is the largest;

[0135] S130: taking the other side of the first side on both sides of the seam position as the target side of the seam.

[0136] For the convenience of explanation, the parts of the target object distributed on both sides of the seam in the global mosaic graph are respectively referred to as the first local object and the second local object.

[0137] If the point farthest from the seam position on the first partial object is farther from the seam than the point farthest from the seam position on the second partial object, that is, the farthest point is the point on the first partial object, and the side where the first partial object is located is the first side, it means that the first partial object should be closer to the boundary of the splicing range relative to the second partial object. Therefore, the side where the second partial object is located is taken as the target side, and the target seam position is determined in the target side, so as to avoid the target seam position exceeding the splicing range as much as possible.

[0138] In one embodiment, the method further comprises:

[0139] S010: Acquire the collected position data of the target object;

[0140] S020: determining a position information set corresponding to the position data of the target object in the global mosaic graph, wherein the position information set includes position information of all points of the target object distributed in the global mosaic graph;

[0141] S030: Check whether there is an intersection between the position information set and the seam position in the global splicing graph.

[0142] In step S010, the position data of the target object can be collected by sensing devices such as camera equipment, radar sensors, laser sensors, etc. The position data of the target object can represent the relative position of the target object and the sensing device. Since the target object is a three-dimensional object, the relative positions of various points on the target object and the sensing device are different, and the position data collected by the sensing device can reflect these different relative positions. The electronic device can be connected to the sensing device through a data transmission line and receive the position data transmitted by the sensing device. The number of sensing devices connected to the electronic device can be greater than or equal to 1.

[0143] In step S020, a position information set corresponding to the position data of the target object is determined in the global mosaic graph, and the position information set includes the position information of all points of the target object distributed in the global mosaic graph.

[0144] Taking the perception device as a camera device as an example, a depth image containing a target object can be collected, and the position data of the target object in the coordinate system used by the camera device can be determined based on the depth data in the image. The coordinate system is converted into the coordinate system used by the global mosaic image, and the position information of all points of the target object in the global mosaic image can be obtained.

[0145] Of course, the correspondence between location data and location information is not limited to this, and can be determined according to the actual situation of the sensing device.

[0146] In step S030, it is checked whether there is an intersection between the position information set and the seam position in the global splicing graph.

[0147] If there is an intersection between the position information set and the position information of all points on the seam in the global stitching image, it means that the seam will pass through the target object, that is, it is determined that the target object and the preset seam position in the panoramic stitching image have an intersection, and step S100 can be executed.

[0148] The following two methods can be used to check whether there is an intersection between the position information set and the position information of all points on the seam in the global splicing graph:

[0149] The first method is to traverse the position information of all points at the seam position in the global mosaic graph, and search for the traversed position information in the position information set. If it is found, it means there is an intersection, and the traversal can be ended. If it is still not found after the traversal, it means there is no intersection.

[0150] The second method is to traverse the location information in the location information set, and search for the traversed location information in the location information of all points at the seam position in the global mosaic graph. If the location information is found, it means there is an intersection, and the traversal can be terminated. If the location information is still not found after the traversal, it means there is no intersection.

[0151] The present invention also provides an image splicing device, see Figure 2 , the image stitching device 100 comprises:

[0152] A reference point determination module 101 is used for, when a target object intersects with a preset stitching position in a panoramic stitching image, using points of the target object distributed on a target side offset away from the stitching position as reference points, wherein the target side is one of the two sides of the stitching position;

[0153] A target seam position determination module 102, configured to determine a target seam position based on the position information of the reference point and the seam position information;

[0154] The sub-image splicing module 103 is used to splice two sub-images of the target object at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

[0155] In one embodiment, the reference point at least includes a point whose distance from the seam position is greater than or equal to the distance between the pole point and the seam position;

[0156] The extreme point is the point farthest from the seam position among all points of the target object distributed on the target side.

[0157] In one embodiment, the reference point further includes points different from the extreme point along the edge of the target object among all points of the target object distributed on the target side.

[0158] In one embodiment, when the target seam position determination module determines the target seam position based on the position information of the reference point and the seam position information, it is specifically used to:

[0159] Determining the position information of the target point on the target side according to the position information of the reference point, wherein the distance between the target point and the seam position is greater than the distance between the reference point and the seam position;

[0160] The target seam position is determined according to the position information of the target point and the position information of the start and end points of the seam position.

[0161] In one embodiment, when the target seam position determination module determines the position information of the target point on the target side according to the position information of the reference point, it is specifically used to:

[0162] Checking whether the reference point is between the stitching position and a specified boundary, wherein the specified boundary is a boundary of a stitchable range located on the target side;

[0163] If yes, the position information of one point between the reference point and the target boundary point on the designated boundary is determined as the position information of the target point, and the target boundary point is the point closest to the reference point and the designated boundary.

[0164] In one embodiment, when the target seam position determination module determines the target seam position according to the position information of the target point and the position information of the start and end points of the seam position, it is specifically used to:

[0165] Performing curve fitting on the target point and the start and end points of the seam position according to the position information of the target point and the position information of the start and end points of the seam position to obtain a fitting curve;

[0166] The position of the fitting curve is determined as the target seam position.

[0167] In one embodiment, before the reference point determination module uses the point of the target object distributed on the target side and offset away from the seam position as the reference point, it is further used to:

[0168] Calculate the distance between each point of the target object distributed in the global splicing graph and the splicing position;

[0169] Determine a first side where the farthest point on both sides of the seam position is located, the farthest point being the point where the distance between each point of the target object and the seam position is the largest;

[0170] The other side of the first side of the two sides of the seam position is used as the target side of the seam.

[0171] In one embodiment, the device further comprises:

[0172] A position data acquisition module, used to acquire the collected position data of the target object;

[0173] A position information set determination module, configured to determine a position information set corresponding to the position data of the target object in the global mosaic graph, wherein the position information set includes position information of all points of the target object distributed in the global mosaic graph;

[0174] A checking module is used to check whether there is an intersection between the position information set and the seam position in the global splicing graph.

[0175] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0176] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units.

[0177] The present invention also provides an electronic device, comprising a processor and a memory; the memory stores a program that can be called by the processor; wherein when the processor executes the program, the image stitching method described in the above embodiment is implemented.

[0178] The embodiment of the image stitching device of the present invention can be applied to electronic devices. Taking software implementation as an example, as a device in a logical sense, the processor of the electronic device in which it is located reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 7 As shown, Figure 7 is a hardware structure diagram of an electronic device in which an image stitching device 100 is located according to an exemplary embodiment of the present invention, except Figure 7 In addition to the processor 510, memory 530, interface 520, and non-volatile memory 540 shown, the electronic device where the device 100 is located in the embodiment may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0179] The present invention also provides a machine-readable storage medium on which a program is stored. When the program is executed by a processor, the image stitching method as described in any one of the aforementioned embodiments is implemented.

[0180] The present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Machine-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. The information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of machine-readable storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0181] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image stitching method, characterized in that: include: When the target object intersects with a preset stitching position in the panoramic stitching image, the distance between each point of the target object distributed in the global stitching image and the stitching position is calculated; Determine a first side where the farthest point on both sides of the seam position is located, the farthest point being the point where the distance between each point of the target object and the seam position is the largest; Using the other side of the first side of the two sides of the seam position as the target side of the seam; Taking the points along the edge of the target object among all the points of the target object distributed on the target side as the reference points after being offset away from the seam position; For each reference point, check whether the reference point is between the stitching position and a specified boundary, where the specified boundary is a boundary of a stitchable range on the target side; if so, determine the position information of one point between the reference point and a target boundary point on the specified boundary as the position information of the target point, where the target boundary point is the point closest to the reference point and the specified boundary; the distance between the target point and the stitching position is greater than the distance between the reference point and the stitching position; Determine the target seam position according to the position information of all target points and the position information of the start and end points of the seam position; The two sub-images of the target object are spliced ​​at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

2. The image stitching method according to claim 1, characterized in that: Determining the target seam position according to the position information of the target point and the position information of the start and end points of the seam position includes: Performing curve fitting on the target point and the start and end points of the seam position according to the position information of the target point and the position information of the start and end points of the seam position to obtain a fitting curve; The position of the fitting curve is determined as the target seam position.

3. The image stitching method according to claim 1, characterized in that: The method further comprises: Acquire the collected location data of the target object; Determine a position information set corresponding to the position data of the target object in the global mosaic graph, wherein the position information set includes position information of all points of the target object distributed in the global mosaic graph; Check whether there is an intersection between the position information set and the seam position in the global splicing graph.

4. An image stitching device, characterized in that: include: A reference point determination module is used to calculate the distance between each point of the target object distributed in the global mosaic image and the stitching position when the target object has an intersection with the stitching position in the preset panoramic mosaic image; determine the first side where the farthest point on both sides of the stitching position is located, the farthest point is the point where the distance between each point of the target object and the stitching position is the largest; use the other side of the first side on both sides of the stitching position as the target side of the stitching; use the point along the edge of the target object among all the points of the target object distributed on the target side after being offset away from the stitching position as the reference point; The target seam position determination module is used to check, for each reference point, whether the reference point is between the seam position and a specified boundary, where the specified boundary is a boundary of a splicable range on the target side; if so, determine the position information of one point between the reference point and a target boundary point on the specified boundary as the position information of the target point, where the target boundary point is the point closest to the reference point and the specified boundary; the distance between the target point and the seam position is greater than the distance between the reference point and the seam position; determine the target seam position based on the position information of all target points and the position information of the start and end points of the seam position; The sub-image splicing module is used to splice two sub-images of the target object at the target seam position, so that the target object is distributed in one of the sub-images of the spliced ​​spliced ​​image.

5. The image stitching device according to claim 4, characterized in that: When the target seam position determination module determines the target seam position according to the position information of the target point and the position information of the start and end points of the seam position, it is specifically used to: Performing curve fitting on the target point and the start and end points of the seam position according to the position information of the target point and the position information of the start and end points of the seam position to obtain a fitting curve; The position of the fitting curve is determined as the target seam position.

6. The image stitching device according to claim 4, characterized in that: The device further comprises: A position data acquisition module, used to acquire the collected position data of the target object; A position information set determining module, configured to determine a position information set corresponding to the position data of the target object in the global mosaic graph, wherein the position information set includes position information of all points of the target object distributed in the global mosaic graph; A checking module is used to check whether there is an intersection between the position information set and the seam position in the global splicing graph.

7. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores a program that can be called by the processor; wherein when the processor executes the program, the image stitching method as described in any one of claims 1 to 3 is implemented.

8. A machine-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the image stitching method as described in any one of claims 1 to 3 is implemented.