Splice line movement method, apparatus, device, medium, and vehicle

Abstract

The present disclosure relates to a stitching line moving method, device, equipment, medium and vehicle. The stitching line moving method comprises: obtaining a first image and a second image; determining point clouds corresponding to each target object in a region of interest image through a three-dimensional reconstruction technology; wherein the first image comprises a coincident image region with the second image having coincident image content, and the coincident image region comprises the region of interest image; when the point clouds corresponding to at least one target object overlap with the point clouds corresponding to a plane associated with the stitching line, moving the stitching line to make the point clouds corresponding to the plane associated with the moved stitching line not overlap with the point clouds corresponding to each target object; and the plane associated with the stitching line is perpendicular to the ground and comprises a straight line corresponding to the stitching line in a physical space. In this way, the stitching line can avoid each target object in the region of interest image, and thus when the first image is stitched with other images, the phenomenon of target object misalignment in stitching can be avoided.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, device, medium, and vehicle for moving splicing lines. Background Technology

[0002] Currently, in order to enable drivers to clearly see whether there are obstacles around the vehicle and understand the relative position of the obstacles, and to help drivers park the vehicle easily, multiple (e.g., 4) cameras that can cover the entire field of view around the vehicle are usually installed around the vehicle. The multiple images collected at the same time are stitched together to form a 360-degree top view of the vehicle's surroundings (i.e., the vehicle surround view top view), which is then displayed on the screen of the center console.

[0003] However, when an obstacle is located in an area where both cameras are capturing images repeatedly, the significant difference in the angles from which the two cameras view the obstacle can lead to misalignment of the obstacle during subsequent image stitching if the stitching line is not properly positioned. This makes it difficult for the driver to accurately determine the obstacle's location. Therefore, a method for rationally setting the stitching line is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, and medium for moving splicing lines.

[0005] In a first aspect, this disclosure provides a method for moving a splicing line, including:

[0006] Acquire the first image and the second image;

[0007] Using 3D reconstruction technology, the point cloud corresponding to each target object in the region of interest is determined; wherein, the first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes the region of interest.

[0008] When the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, the stitching line is moved so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each target object; wherein, the plane associated with the stitching line is perpendicular to the ground and includes the straight line corresponding to the stitching line in physical space.

[0009] Secondly, this disclosure provides a splicing line moving device, including:

[0010] The first acquisition module is used to acquire the first image and the second image;

[0011] The first determining module is used to determine the point cloud corresponding to each target object in the region of interest through three-dimensional reconstruction technology; wherein, the first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes the region of interest;

[0012] The first moving module is used to move the stitching line when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each target object; wherein the plane associated with the stitching line is perpendicular to the ground and includes the straight line corresponding to the stitching line in physical space.

[0013] Thirdly, this disclosure provides an electronic device, including:

[0014] processor;

[0015] Memory, used to store executable instructions;

[0016] The processor is used to read executable instructions from memory and execute the executable instructions to implement the splicing line moving method described in the first aspect.

[0017] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the splicing line movement method described in the first aspect.

[0018] Fifthly, this disclosure provides a vehicle comprising at least one of the following: the splicing line moving device described in the second aspect; the electronic device described in the third aspect; and the computer-readable storage medium described in the fourth aspect.

[0019] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0020] The splicing line moving method, apparatus, device, medium, and vehicle of this disclosure are capable of acquiring a first image and a second image; determining the point cloud corresponding to each target object in the region of interest (ROI) using 3D reconstruction technology; wherein the first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes the ROI; when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the splicing line, the splicing line is moved so that the point cloud corresponding to the plane associated with the moved splicing line does not overlap with the point clouds corresponding to each target object; wherein the plane associated with the splicing line is perpendicular to the ground and includes a straight line in physical space corresponding to the splicing line. In this way, the splicing line can avoid each target object in the ROI, thereby preventing misalignment of target objects when the first image is spliced ​​with other images. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A flowchart illustrating a splicing line moving method provided in an embodiment of this disclosure is shown;

[0023] Figure 2 This illustration shows a schematic diagram of an image acquisition device installed on a target vehicle according to an embodiment of the present disclosure;

[0024] Figure 3 This illustration shows a schematic diagram of a splicing line cutting target object according to an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a splicing line bypassing a target object, according to an embodiment of this disclosure, is shown;

[0026] Figure 5 A schematic diagram of a local coordinate system provided in an embodiment of this disclosure is shown;

[0027] Figure 6 A schematic diagram of the structure of a splicing line moving device provided in an embodiment of this disclosure is shown;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] To address the aforementioned problems, this disclosure provides a method, apparatus, device, medium, and vehicle for moving splicing lines. The splicing line moving method provided in this disclosure will be described first.

[0036] Figure 1 A flowchart illustrating a splicing line moving method provided in an embodiment of this disclosure is shown.

[0037] In some embodiments of this disclosure, Figure 1 The splicing line movement method shown can be applied to electronic devices such as vehicle controllers, and is not specifically limited thereto.

[0038] like Figure 1 As shown, the splicing line moving method may include the following steps.

[0039] S110, Obtain the first image and the second image.

[0040] In this embodiment of the disclosure, the electronic device may be connected to at least two image acquisition devices to receive images acquired by each image acquisition device.

[0041] For example, Figure 2 A schematic diagram illustrating an embodiment of this disclosure of installing an image acquisition device on a target vehicle is shown. See also... Figure 2An image acquisition device 210 is installed at the front, rear, left, and right sides of the target vehicle. The image acquisition device 210 may include, but is not limited to, devices with image acquisition capabilities such as wide-angle cameras. Electronic devices can be connected to each image acquisition device 210 to receive images acquired by each device.

[0042] In some embodiments, S110 may specifically include: acquiring a first image captured by a first image acquisition device at a first moment, and a second image captured by a second image acquisition device at the first moment. It should be noted that this specific implementation of S110 is applicable whether the target vehicle is stationary or in motion.

[0043] Specifically, the first image acquisition device and the second image acquisition device can be any two image acquisition devices connected to an electronic device, and the first image acquisition device and the second image acquisition device can repeatedly acquire images of a certain physical space area (referred to as the repeated acquisition area).

[0044] For example, the first image acquisition device and the second image acquisition device can be two adjacent image acquisition devices in the surrounding direction among a plurality of image acquisition devices arranged around the target vehicle. For example, the first image acquisition device can be an image acquisition device located on the left side of the target vehicle, and the second image acquisition device can be an image acquisition device located on the front side of the target vehicle, etc., but are not limited to this.

[0045] In some other embodiments, S110 may specifically include: acquiring a first image acquired by the first image acquisition device at a first moment, and a second image acquired by the first image acquisition device at a second moment; wherein the second moment is later than the first moment. It should be noted that the specific implementation of this S110 is applicable when the target vehicle is in motion.

[0046] Specifically, the first image acquisition device can repeatedly acquire images of a certain physical space region at a first time and a second time. The physical space includes the repeated acquisition areas corresponding to the first image acquisition device and the second image acquisition device.

[0047] Specifically, the first image acquisition device can be any image acquisition device connected to an electronic device. For example, the first image acquisition device can be... Figure 2 The image acquisition device shown is any of those shown, but is not limited to them.

[0048] S120. Using 3D reconstruction technology, determine the point cloud corresponding to each target object in the region of interest.

[0049] The first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes an image of interest region.

[0050] In this embodiment of the disclosure, the first image includes an overlapping image region, which includes an image region of interest. The electronic device can determine the point cloud corresponding to each target object in the image region of interest through three-dimensional reconstruction technology.

[0051] Specifically, the image area in the first image that has overlapping image content with the second image is the overlapping image area.

[0052] Specifically, the overlapping image area includes the image area of ​​interest. The image area containing the image content obtained by the first image acquisition device from the repeatedly acquired area is the image area of ​​interest.

[0053] In some embodiments, the first image is an image captured by a first imaging device at a first moment, and the second image is an image captured by a second imaging device at the first moment. In this case, the overlapping image region is the region of interest.

[0054] In other embodiments, the first image is an image captured by the first image device at a first moment, the second image is an image captured by the first image device at a second moment, and the electronic device can also acquire a third image captured by the second image device at the first moment. In this case, the image area in the first image that has overlapping image content with the second image is called the overlapping image area, and the overlapping image area includes the region of interest. The image area in the first image that has overlapping image content with the third image is called the region of interest.

[0055] 3D reconstruction is based on a series of images of an environment or object from different angles, and through a series of processes, obtains a 3D model (e.g., a point cloud) of the object. Specifically, in the embodiments of this disclosure, an electronic device can use 3D reconstruction technology to obtain the point cloud corresponding to the target object in the image region of interest.

[0056] Specifically, the target object can be any object in the region of interest other than the background (e.g., the sky). For example, the target object can include vehicles, pedestrians, trees, etc., but is not limited to these.

[0057] In one example, an electronic device can be reconstructed in 3D using the conversion relationship between pixel coordinates and world coordinates.

[0058] For example, assume the pixel coordinate system is (u, v), the image coordinate system is (x, y), and the camera coordinate system is (X, v). c Y c Z c The world coordinate system is (X, Y, Z). The transformation relationship between pixel coordinates and world coordinates can be as follows:

[0059]

[0060] Where, d x d represents the size of a pixel on the u-axis. y Let be the pixel size on the v-axis, u0 and v0 be the center of the image plane, f be the focal length of the first image acquisition device, R be the rotation matrix, T be the translation matrix, and Z be the focal length of the pixel on the v-axis. c The pixel's lower edge Z in camera coordinates c The coordinate values ​​of the axis.

[0061] When performing 3D reconstruction, the following parameters in the above formula can be determined first through calibration: d x d y Z can be determined based on the directional difference (i.e., parallax) caused by sampling the same object from two viewpoints at a certain distance. c By substituting the parameters into the above formula, the conversion relationship between pixel coordinates and world coordinates can be determined. Based on the conversion relationship between pixel coordinates and world coordinates, the coordinates of the pixel in the pixel coordinate system are converted to the coordinates in the world coordinate system, so as to obtain the coordinates of the object point corresponding to the pixel in the physical space. The physical space is the three-dimensional space in reality. The pixel is obtained by image acquisition of the object point corresponding to the pixel through an image acquisition device.

[0062] Specifically, by converting the coordinates of each pixel on the target object in the pixel coordinate system to the coordinates in the world coordinate system, the point cloud corresponding to the target object can be obtained.

[0063] S130. When the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, move the stitching line so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each target object.

[0064] The plane associated with the splicing line is perpendicular to the ground and includes the straight line in the physical space corresponding to the splicing line.

[0065] In this embodiment of the disclosure, the electronic device can determine whether the point cloud corresponding to each target object overlaps with the point cloud corresponding to the plane associated with the stitching line. When the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, the stitching line is moved so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each target object.

[0066] Specifically, the stitching line is used to cut the first image so that the cut first image can be stitched together with other images at the stitching line. It should be noted that for each first image acquired at each image acquisition moment, when it is stitched with other images, the electronic device can determine whether its corresponding stitching line needs to be moved. The stitching line mentioned here is a preset stitching line, which may not be the actual stitching line used when it is finally stitched with other images. For the first image acquired at the first moment, its corresponding preset stitching line is the actual stitching line used when the first image acquired at the previous moment is stitched with other images.

[0067] Specifically, the straight line corresponding to the splicing line in physical space refers to the straight line containing the virtual object points in physical space corresponding to each point on the splicing line. Based on the conversion relationship between pixel coordinates and world coordinates, the coordinates of each point on the splicing line in the pixel coordinate system can be converted to coordinates in the world coordinate system to obtain the coordinates of the virtual object points corresponding to each point on the splicing line.

[0068] Specifically, the plane associated with the splicing line refers to the plane defined by the intersection of the straight line corresponding to the splicing line and the straight line perpendicular to the ground in physical space.

[0069] Specifically, overlap between the point cloud corresponding to the target object and the point cloud corresponding to the plane associated with the stitching line means that the point cloud corresponding to the target object and the point cloud corresponding to the plane associated with the stitching line include the same coordinates; similarly, non-overlap between the point cloud corresponding to the target object and the point cloud corresponding to the plane associated with the stitching line means that the point cloud corresponding to the target object and the point cloud corresponding to the plane associated with the stitching line do not include the same coordinates.

[0070] When the point cloud corresponding to the target object overlaps with the point cloud corresponding to the plane associated with the stitching line, it indicates that when the first image is cut by the stitching line, the stitching line will cut the target object. Thus, when the first image is stitched with other images, the complete image of the target object is formed by stitching the first image and other images, which may result in stitching misalignment. When the point cloud corresponding to the target object does not overlap with the point cloud corresponding to the plane associated with the stitching line, it indicates that when the stitching line cuts the first image, the stitching line will bypass the target object, thereby ensuring that the target object is completely located in the first image or other images that need to be stitched with the first image, avoiding the problem of target object stitching misalignment.

[0071] Specifically, the splicing line can be a straight line or a curve; there is no limitation here, as long as the splicing line bypasses the target object without cutting it. For example, the splicing line can be rotated around one of its endpoints at a certain angle to bypass the target object, or, based on the original splicing line, the part that does not cut the target object can be retained, while the part that cuts the target object can be bypassed by partially surrounding the target object, but it is not limited to these methods.

[0072] For example, Figure 3 This illustration shows a schematic diagram of a splicing line cutting target object according to an embodiment of the present disclosure. Figure 4 A schematic diagram illustrating a splicing line bypassing a target object, according to an embodiment of this disclosure, is shown. See also... Figure 3 The first image 310 is an image captured by the first image acquisition device at a first moment, and the second image 320 is an image captured by the second image acquisition device at a first moment. The first image 310 has an image region of interest 311, which includes a target object 312. When the electronic device detects that the point cloud corresponding to the target object 312 overlaps with the point cloud corresponding to the plane associated with the stitching line, it will not cut the first image with the stitching line 313, but will move the stitching line 313 so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to the target object 312. Figure 4 As shown, in the image region of interest 411, the moved stitching line 413 can bypass the target object 412. It is understandable that dynamically adjusting the stitching line can effectively prevent the stitching line from cutting the object, thereby improving the problem of misalignment in the target object stitching.

[0073] The stitching line moving method of this disclosure can acquire a first image and a second image; determine the point cloud corresponding to each target object in the region of interest using three-dimensional reconstruction technology; wherein the first image includes an overlapping image region with overlapping image content with the second image, and the overlapping image region includes the region of interest; when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, the stitching line is moved so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point clouds corresponding to each target object; wherein the plane associated with the stitching line is perpendicular to the ground and includes a straight line in physical space corresponding to the stitching line. In this way, the stitching line can avoid each target object in the region of interest, thereby preventing misalignment of target objects when the first image is stitched with other images.

[0074] In another embodiment of this disclosure, S120 may include: S1211, performing three-dimensional reconstruction on the region of interest to obtain the point cloud of the region of interest; S1212, performing image recognition on the region of interest to determine the image regions of each target object; S1213, for each target object image region, extracting the point cloud corresponding to the target object image region from the point cloud of the region of interest.

[0075] Specifically, S1211 may include: based on the transformation relationship between the pixel coordinate system and the world coordinate system, converting the pixel coordinates corresponding to each pixel in the region of interest to coordinates in the world coordinate system, that is, obtaining the coordinates of the object points corresponding to each pixel, thereby obtaining the point cloud of the region of interest corresponding to the region of interest.

[0076] In one example, the first image is an image captured by a first imaging device at a first moment, and the second image is an image captured by a second imaging device at a first moment. At this time, based on the information of the same object point when it is imaged in the first image and the information when it is imaged in the second image, the parameters in the transformation relationship between the pixel coordinate system and the world coordinate system are calibrated, thereby determining the transformation relationship between the pixel coordinate system and the world coordinate system.

[0077] In another example, the first image is an image captured by the first imaging device at a first moment, the second image is an image captured by the first imaging device at a second moment, and the electronic device can also acquire a third image captured by the second imaging device at the first moment. In this case, based on information about the same object point when it is imaged in the first image, information about it when it is imaged in the third image, and the motion information of the target vehicle (e.g., vehicle speed and deflection angle), the parameters in the transformation relationship between the pixel coordinate system and the world coordinate system are calibrated, thereby determining the transformation relationship between the pixel coordinate system and the world coordinate system.

[0078] In some embodiments, S1212 may specifically include: inputting the region of interest into the image recognition model, and receiving the category of each target object and the target object image region corresponding to each target object output by the image recognition model.

[0079] In some other embodiments, S1212 may further include: performing image recognition on the region of interest based on edge extraction technology to obtain the category of each target object and the target object image region corresponding to each target object.

[0080] It should be noted that the target object image area can include the pixels corresponding to the target object. When the extraction accuracy is poor, the pixels corresponding to the background near the target object can also be included. There is no limitation on this.

[0081] Specifically, S1213 may include: for each target object image region, extracting the coordinates of the object point corresponding to each pixel in the target object image region from the point cloud of the image region of interest.

[0082] It is understandable that extracting the point cloud corresponding to the target object's image region from the point cloud of the region of interest is a simple and easy-to-implement method that helps reduce the difficulty of determining the point cloud corresponding to the target object.

[0083] In another embodiment of this disclosure, S120 may include: S1221, performing image recognition on the region of interest to determine the target object image region where each target object is located; S1222, performing three-dimensional reconstruction on each target object image region to obtain the point cloud corresponding to the target object image region.

[0084] Specifically, S1221 is similar to S1212, and will not be described in detail here.

[0085] Specifically, S1212 may include: based on the transformation relationship between the pixel coordinate system and the world coordinate system, converting the pixel coordinates corresponding to each pixel in the target object image area into coordinates in the world coordinate system, that is, obtaining the coordinates of the object point corresponding to each pixel, thereby obtaining the point cloud of the target object image area corresponding to the target object image area.

[0086] It is understandable that by performing 3D reconstruction only on the target object image area to obtain the point cloud corresponding to the target object image area, it is possible to save the need to perform 3D reconstruction on image areas other than the target object image area, thereby improving the efficiency of determining the point cloud corresponding to the target object image area.

[0087] In another embodiment of this disclosure, before S130, the method may further include: S140, determining the coordinates of each projection point of each target object on the ground for the point cloud corresponding to each target object, to obtain a first projection coordinate set; S150, establishing a local coordinate system corresponding to the region of interest; S160, converting each coordinate in the first projection coordinate set into coordinates under the local coordinate system, to obtain a second projection coordinate set; wherein, S130 may include: moving the stitching line so that the coordinates of each point on the stitching line under the local coordinate system do not overlap with the second projection coordinate set.

[0088] Specifically, a ground coordinate system is pre-established with the center of the target vehicle as the origin, the ground as the X′O′Y′ plane, and the direction perpendicular to the ground upwards as the Z′ axis. The transformation relationship between the world coordinate system and the ground coordinate system is determined. In this case, S140 may include: S141, for each point cloud corresponding to the target object, based on the transformation relationship between the world coordinate system and the ground coordinate system, converting each coordinate in the point cloud corresponding to the target object from its world coordinate system to its ground coordinate system; S142, for each coordinate in the ground coordinate system, retaining its coordinate values ​​on the X′ and Y′ axes, and after deduplication, obtaining the first projected coordinate set. It should be noted that when the world coordinate system is the same as the ground coordinate system, S141 can be omitted.

[0089] Specifically, the local coordinate system is a two-dimensional coordinate system. The origin O″ of the local coordinate system, and the extension directions of the X″ and Y″ axes can be set by those skilled in the art according to the actual situation, and are not limited here.

[0090] Specifically, S160 may include: based on the transformation relationship between the local coordinate system and the ground coordinate system, each coordinate in the first projected coordinate set can be converted into a coordinate in the local coordinate system to obtain the second projected coordinates.

[0091] Specifically, based on the transformation relationship between the pixel coordinate system and the local coordinate system, the coordinates of each point on the splicing line in the pixel coordinate system can be converted to the coordinates in the local coordinate system. At this time, S130 may specifically include: moving the splicing line so that the second projection coordinate set does not contain the same coordinates as the coordinates of each point on the splicing line in the local coordinate system.

[0092] Understandably, by determining the second set of projection coordinates, the area that the stitching line should avoid can be accurately determined, and thus the area that the stitching line can move to can be accurately determined (i.e., the area in the image of interest excluding the area that needs to be avoided), which facilitates flexible movement of the stitching line.

[0093] In another embodiment of this disclosure, S150 may include: establishing a local coordinate system according to the principle that the origin of the local coordinate system coincides with the endpoint of the splicing line; before S130, the method may further include: S170, determining the maximum and minimum angles between the lines connecting each pixel in the image region of interest to the origin of the local coordinate system and the X″ axis of the local coordinate system; S180, based on each coordinate in the second projection coordinate set, determining the angle between the line connecting its corresponding pixel to the origin of the local coordinate system and the X″ axis of the local coordinate system, to obtain a first set of angles; wherein, S130 may include: moving the splicing line to a non-target object region, wherein the non-target object region is the region defined by the lines corresponding to the maximum and minimum angles, excluding the region defined by the lines corresponding to the set of angles.

[0094] Specifically, any endpoint of the stitching line can be used as the origin of the local coordinate system. The extension directions of the X″ and Y″ axes can be set by those skilled in the art according to the actual situation, and are not limited here. For example, the extension directions of the X″ and Y″ axes can be determined according to the principle that the region of interest is located in one of the quadrants, but it is not limited to this. For example, the region of interest can also span two quadrants in the local coordinate system.

[0095] Specifically, S170 may include: based on the transformation relationship between the pixel coordinate system and the local coordinate system, converting the coordinates of each pixel in the image region of interest from the pixel coordinate system to the local coordinate system; for each pixel in the image region of interest, based on the coordinates of the pixel in the local coordinate system, determining the angle between the line connecting the pixel to the origin of the local coordinate system and the X″ axis, obtaining a second set of angles; and selecting the maximum and minimum angles from the second set of angles.

[0096] Specifically, S180 may include: for each pixel in the target object image area, based on the coordinates of the pixel in the local coordinate system, determining the angle between the line connecting the pixel to the origin of the local coordinate system and the X″ axis, to obtain a first set of angles.

[0097] It should be noted that the angle between the line connecting the pixel to the origin of the local coordinate system and the X″ axis refers to the angle through which the line connecting the pixel to the origin of the local coordinate system rotates clockwise (or counterclockwise) to the X″ axis.

[0098] Specifically, S130 may include: moving the splicing line so that the angle between the splicing line and the X″ axis belongs to the second set of included angles and not to the first set of included angles.

[0099] For example, Figure 5 A schematic diagram of a local coordinate system provided in an embodiment of this disclosure is shown. See also Figure 5 In the image region of interest 511, a local coordinate system is established with one endpoint of the stitching line as the origin O″, and the horizontal direction to the left as the X″ axis and the forward direction as the Y″ axis. The maximum angle between the line connecting each pixel in the image region of interest 511 to the origin O″ of the local coordinate system and the X″ axis of the local coordinate system is determined to be 80°, and the minimum angle is determined to be 10°. Furthermore, based on each coordinate in the second projection coordinate set, the angle between the line connecting its corresponding pixel to the origin O″ of the local coordinate system and the X″ axis of the local coordinate system is determined, resulting in a first set of angles [20°, 40°] and [50°, 60°]. At this point, the stitching line can be rotated around the origin of the local coordinate system until the angle with the X″ axis is within the range of [10°, 20°), (40°, 50°), or (60°, 80°). Figure 5The example shows that the angle between the rotated splicing line 513 and the X″ axis is within the range of (60°, 80°), and the splicing line 513 always avoids each target object 512 within the range of (60°, 80°).

[0100] It is understandable that by determining the first set of included angles and the second set of included angles, it is possible to know the area that the splicing line can be rotated to (i.e., the non-target object area). Thus, it is only necessary to rotate the splicing line around its endpoints to the non-target object area, which simplifies the way the splicing line is moved.

[0101] In another embodiment of this disclosure, before moving the stitching line, the method may further include: for each target object, determining the maximum height H among the heights of each object point on the target object from the ground based on the point cloud corresponding to the target object; when the maximum height H is greater than a preset height threshold, determining the target object as an obstacle; wherein, moving the stitching line when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line includes: moving the stitching line when the point cloud corresponding to at least one obstacle overlaps with the point cloud corresponding to the plane associated with the stitching line.

[0102] Specifically, determining the height of each point on the target object from the ground can include: for the point cloud corresponding to the target object, based on the transformation relationship between the world coordinate system and the ground coordinate system, converting the coordinates of each point in the point cloud corresponding to the target object in the world coordinate system to coordinates in the ground coordinate system, thus obtaining the coordinates of each point on the target object in the ground coordinate system; for each point on the target object, based on the coordinates of that point in the ground coordinate system, determining the height of each point on the target object from the ground, that is, the coordinate value corresponding to the Z′ axis of each point in the ground coordinate system. It should be noted that when the world coordinate system is the same as the ground coordinate system, the step of converting the coordinates of each point in the point cloud corresponding to the target object in the world coordinate system to coordinates in the ground coordinate system can be omitted.

[0103] Specifically, the specific value of the preset height threshold can be set by those skilled in the art according to the actual situation and is not limited here. For example, the preset height threshold can be greater than or equal to 15 centimeters.

[0104] Understandably, some target objects, such as speed bumps, have a relatively minor impact on the driver's operation of the target vehicle. Drivers typically focus on obstacles that significantly affect driving safety, while paying less attention to target objects that have a smaller impact on the vehicle's operation. Therefore, when stitching the first image with other images, whether such target objects as speed bumps are misaligned has little impact on the driver's safe operation of the target vehicle. Thus, the cutting of these target objects by the stitching lines can be disregarded, thereby increasing the area where the stitching lines can be set.

[0105] In another embodiment of this disclosure, before moving the stitching line, the method may further include: for each target object, determining the maximum distance L among the distances between each object point on the target object and the target vehicle based on the point cloud corresponding to the target object; wherein the target vehicle is the vehicle where the first image acquisition device that acquired the first image is located; when the maximum distance L is greater than a preset distance threshold, determining the target object as an obstacle; wherein moving the stitching line when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line includes: moving the stitching line when the point cloud corresponding to at least one obstacle overlaps with the point cloud corresponding to the plane associated with the stitching line.

[0106] Specifically, determining the distance from each point on the target object to the target vehicle can include: for the point cloud corresponding to the target object, based on the transformation relationship between the world coordinate system and the ground coordinate system, converting the coordinates of each point in the point cloud corresponding to the target object in the world coordinate system to coordinates in the ground coordinate system, thus obtaining the coordinates of each point on the target object in the ground coordinate system; for each point on the target object, based on the coordinates of that point in the ground coordinate system and the coordinates of each point on the surface of the target vehicle in the ground coordinate system, determining the distance from that point to the target vehicle. It should be noted that when the world coordinate system is the same as the ground coordinate system, the step of converting the coordinates of each point in the point cloud corresponding to the target object in the world coordinate system to coordinates in the ground coordinate system can be omitted.

[0107] Specifically, the specific value of the preset distance threshold can be set by those skilled in the art according to the actual situation and is not limited here. For example, the preset distance threshold can be greater than or equal to 1.5 meters.

[0108] It is understandable that some target objects, when far from the target vehicle, have little impact on the driver's operation. Therefore, when stitching the first image with other images, whether target objects far from the target vehicle are misaligned has little impact on the driver's safe operation. Thus, the cutting effect of the stitching line on such target objects can be disregarded, thereby increasing the area where the stitching line can be set.

[0109] It should be noted that the electronic device can also determine the target object as an obstacle only when the maximum height H of the heights of all points on the target object from the ground is greater than a preset height threshold, and the maximum distance L of the distances between all points on the target object and the target vehicle is greater than a preset distance threshold. This further expands the area that can be set for the splicing lines.

[0110] Figure 6 A schematic diagram of the structure of a splicing line moving device provided in an embodiment of this disclosure is shown.

[0111] In some embodiments of this disclosure, Figure 6The splicing line moving device 600 shown can be applied to electronic devices such as vehicle controllers, without specific limitations.

[0112] like Figure 6 As shown, the splicing line moving device 600 may include: a first acquisition module 610, used to acquire a first image and a second image; a first determination module 620, used to determine the point cloud corresponding to each target object in the region of interest through three-dimensional reconstruction technology; wherein, the first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes the region of interest;

[0113] The first moving module 630 is used to move the stitching line when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each target object; wherein the plane associated with the stitching line is perpendicular to the ground and includes the straight line corresponding to the stitching line in physical space.

[0114] The stitching line moving device of this embodiment can acquire a first image and a second image; determine the point cloud corresponding to each target object in the region of interest using three-dimensional reconstruction technology; wherein the first image includes an overlapping image region with overlapping image content with the second image, and the overlapping image region includes the region of interest; when the point cloud corresponding to at least one target object overlaps with the point cloud corresponding to the plane associated with the stitching line, the stitching line is moved so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point clouds corresponding to each target object; wherein the plane associated with the stitching line is perpendicular to the ground and includes a straight line in physical space corresponding to the stitching line. In this way, the stitching line can avoid each target object in the region of interest, thereby preventing misalignment of target objects when the first image is stitched with other images.

[0115] In another embodiment of this disclosure, the first determining module 620 may include:

[0116] The first 3D reconstruction submodule is used to perform 3D reconstruction on the region of interest to obtain the point cloud of the region of interest.

[0117] The first image recognition submodule is used to perform image recognition on the region of interest and determine the image region of each target object.

[0118] The first extraction submodule is used to extract the point cloud corresponding to the target object image region from the point cloud of the image region of interest for each target object image region.

[0119] In yet another embodiment of this disclosure, the first determining module 620 may include:

[0120] The second image recognition submodule is used to perform image recognition on the region of interest and determine the image region of each target object.

[0121] The second 3D reconstruction submodule is used to perform 3D reconstruction on each target object image area to obtain the point cloud corresponding to the target object image area.

[0122] In another embodiment of this disclosure, the apparatus may further include:

[0123] The second determining module is used to determine the coordinates of each projection point of each target object on the ground for each target object's corresponding point cloud before moving the splicing line, so as to obtain the first set of projection coordinates.

[0124] The local coordinate system establishment module is used to establish a local coordinate system corresponding to the region of interest.

[0125] The first transformation module is used to transform each coordinate in the first projected coordinate set into coordinates in the local coordinate system to obtain the second projected coordinate set.

[0126] The first moving module 630 is specifically used to move the splicing line so that the coordinates of each point on the splicing line in the local coordinate system do not overlap with the second projected coordinate set.

[0127] In another embodiment of this disclosure, the local coordinate system establishment module is specifically used to establish a local coordinate system according to the principle that the origin of the local coordinate system coincides with the endpoint of the splicing line;

[0128] The device also includes a third determining module, used to determine the maximum and minimum angles between the lines connecting each pixel in the image region of interest to the origin of the local coordinate system and the X″ axis of the local coordinate system before moving the stitching line;

[0129] The fourth determining module is used to determine the angle between the line connecting the corresponding pixel point and the origin of the local coordinate system and the X″ axis of the local coordinate system based on each coordinate in the second projected coordinate set, so as to obtain the first set of angles.

[0130] Specifically, the first moving module 630 is used to move the splicing line to the non-target object area, wherein the non-target object area is the area defined by the line connecting the maximum and minimum included angles, excluding the area defined by the line connecting the set of included angles.

[0131] In another embodiment of this disclosure, the apparatus may further include:

[0132] The fourth determination module is used to determine the maximum height H of each object point on the target object from the ground, based on the point cloud corresponding to the target object, before moving the splicing line.

[0133] The fifth determination module is used to determine that the target object is an obstacle when the maximum height H is greater than a preset height threshold;

[0134] Specifically, the first moving module 630 is used to move the stitching line when the point cloud corresponding to at least one obstacle overlaps with the point cloud corresponding to the plane associated with the stitching line.

[0135] In another embodiment of this disclosure, the apparatus may further include:

[0136] The sixth determining module is used to determine the maximum distance L between each object point on the target object and the target vehicle based on the point cloud corresponding to the target object before moving the stitching line; where the target vehicle is the vehicle where the first image acquisition device that acquired the first image is located.

[0137] The seventh determination module is used to determine that the target object is an obstacle when the maximum distance L is greater than a preset distance threshold;

[0138] Specifically, the first moving module 630 is used to move the stitching line when the point cloud corresponding to at least one obstacle overlaps with the point cloud corresponding to the plane associated with the stitching line.

[0139] It should be noted that, Figure 6 The splicing line moving device 600 shown can perform the various steps in the above method embodiments and achieve the various processes and effects in the above method embodiments, which will not be elaborated here.

[0140] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown.

[0141] like Figure 7 As shown, the electronic device may include a controller 701 and a memory 702 storing computer program instructions.

[0142] Specifically, the controller 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0143] Memory 702 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway device. In a particular embodiment, memory 702 is a non-volatile solid-state memory. In a particular embodiment, memory 702 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0144] The controller 701 reads and executes the computer program instructions stored in the memory 702 to perform the steps of the splicing line moving method provided in the embodiments of this disclosure.

[0145] In one example, the electronic device may also include a transceiver 703 and a bus 704. Wherein, as... Figure 7 As shown, the controller 701, memory 702 and transceiver 703 are connected via bus 704 and communicate with each other.

[0146] Bus 704 includes hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 704 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0147] This disclosure also provides a computer-readable storage medium that can store a computer program. When the computer program is executed by a processor, the processor enables the splicing line movement method provided in this disclosure.

[0148] The aforementioned storage medium may, for example, include a memory 702 containing computer program instructions, which can be executed by a processor 701 of an electronic device to complete the splicing line movement method provided in this embodiment. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0149] This disclosure also provides a vehicle, including at least one of the following:

[0150] The splicing line moving device described in the above embodiments; the electronic device described in the above embodiments; the computer-readable storage medium described in the above embodiments.

[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0152] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for moving a splicing line, characterized in that, include: Acquire the first image and the second image; Using 3D reconstruction technology, the point cloud corresponding to each target object in the region of interest is determined; wherein, the first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes the region of interest; When the point cloud corresponding to at least one of the target objects overlaps with the point cloud corresponding to the plane associated with the stitching line, the stitching line is moved so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each of the target objects; wherein the plane associated with the stitching line is perpendicular to the ground and includes a straight line in physical space corresponding to the stitching line. Before moving the splicing line, the method further includes: For each target object's point cloud, determine the coordinates of each projection point of each target object on the ground to obtain a first set of projection coordinates; Establish a local coordinate system corresponding to the region of interest; Each coordinate in the first projected coordinate set is converted into a coordinate in the local coordinate system to obtain the second projected coordinate set. The moving of the splicing line includes: Move the splicing line so that the coordinates of each point on the splicing line in the local coordinate system do not overlap with the second projected coordinate set.

2. The method according to claim 1, characterized in that, The process of determining the point cloud corresponding to each target object in the region of interest using 3D reconstruction technology includes: Perform three-dimensional reconstruction on the region of interest to obtain the point cloud of the region of interest; Image recognition is performed on the region of interest to determine the image region of each target object. For each target object image region, the point cloud corresponding to the target object image region is extracted from the point cloud of the region of interest.

3. The method according to claim 1, characterized in that, The process of determining the point cloud corresponding to each target object in the region of interest using 3D reconstruction technology includes: Image recognition is performed on the region of interest to determine the image region of each target object. For each target object image region, a three-dimensional reconstruction is performed on the target object image region to obtain the point cloud corresponding to the target object image region.

4. The method according to claim 1, characterized in that, The step of establishing a local coordinate system corresponding to the region of interest includes: The local coordinate system is established according to the principle that the origin of the local coordinate system coincides with the endpoint of the splicing line; Before moving the splicing line, the method further includes: Determine the line connecting each pixel in the region of interest to the origin of the local coordinate system, and the line connecting the pixel to the origin of the local coordinate system. The maximum and minimum included angles of the axis; Based on each coordinate in the second set of projected coordinates, determine the line connecting its corresponding pixel point to the origin of the local coordinate system and the local coordinate system. The angle between the axes is used to obtain the first set of angles; The moving of the splicing line includes: Move the splicing line to the non-target object area, wherein the non-target object area is the area defined by the line connecting the maximum angle and the minimum angle, excluding the area defined by the line connecting the first set of angles.

5. The method according to claim 1, characterized in that, Before moving the splicing line, the method further includes: For each target object, based on the point cloud corresponding to the target object, determine the maximum height H among the heights of each object point on the target object from the ground. When the maximum height H is greater than a preset height threshold, the target object is determined to be an obstacle; The step of moving the stitching line when the point cloud corresponding to at least one of the target objects overlaps with the point cloud corresponding to the plane associated with the stitching line includes: The stitching line is moved when the point cloud corresponding to at least one of the obstacles overlaps with the point cloud corresponding to the plane associated with the stitching line.

6. The method according to claim 1, characterized in that, Before moving the splicing line, the method further includes: For each target object, based on the point cloud corresponding to the target object, determine the maximum distance L among the distances between each object point on the target object and the target vehicle; wherein, the target vehicle is the vehicle where the first image acquisition device that acquired the first image is located; When the maximum distance L is greater than a preset distance threshold, the target object is determined to be an obstacle; The step of moving the stitching line when the point cloud corresponding to at least one of the target objects overlaps with the point cloud corresponding to the plane associated with the stitching line includes: The stitching line is moved when the point cloud corresponding to at least one of the obstacles overlaps with the point cloud corresponding to the plane associated with the stitching line.

7. A splicing line moving device, characterized in that, include: The first acquisition module is used to acquire the first image and the second image; The first determining module is used to determine the point cloud corresponding to each target object in the region of interest through three-dimensional reconstruction technology; the first image includes an overlapping image region that has overlapping image content with the second image, and the overlapping image region includes the region of interest; A first moving module is configured to move the stitching line when the point cloud corresponding to at least one of the target objects overlaps with the point cloud corresponding to the plane associated with the stitching line, so that the point cloud corresponding to the plane associated with the moved stitching line does not overlap with the point cloud corresponding to each of the target objects; wherein the plane associated with the stitching line is perpendicular to the ground and includes a straight line in physical space corresponding to the stitching line. The splicing line moving device may further include: The second determining module is used to determine the coordinates of each projection point of each target object on the ground for each point cloud corresponding to each target object, and obtain a first set of projection coordinates. A local coordinate system establishment module is used to establish a local coordinate system corresponding to the region of interest. The first transformation module is used to transform each coordinate in the first projected coordinate set into coordinates in the local coordinate system to obtain the second projected coordinate set. The first moving module is used to move the splicing line so that the coordinates of each point on the splicing line in the local coordinate system do not overlap with the second projected coordinate set.

8. An electronic device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the splicing line moving method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the splicing line movement method according to any one of claims 1-6.

10. A vehicle, characterized in that, Includes at least one of the following: The splicing line moving device as described in claim 7 above; The electronic device according to claim 8; The computer-readable storage medium of claim 9 above.

Images