Image stitching apparatus, method, intelligent device, robot, and storage medium

Abstract

The application relates to an image splicing method and device, an intelligent device, a robot and a storage medium. The method comprises the following steps: acquiring to-be-processed images collected by different cameras; selecting a plurality of key points in each to-be-processed image, the key points one-to-one corresponding to a plurality of target points of a world coordinate system, and calculating a transformation matrix according to the key points and the target points; converting each to-be-processed image into a bird's-eye view by using the transformation matrix; and projecting each bird's-eye view to a preset position in the world coordinate system for splicing, so that a spliced image is obtained. The finally obtained spliced image has no splicing edge and has better imaging quality.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to an image stitching apparatus, method, smart device, robot, and storage medium. Background Technology

[0002] With the development of technology, the robotics industry has become increasingly mature, and robots are playing a significant role in various fields. Among these, robots can be used as food delivery robots in the service industry. During peak hours, the delivery efficiency of food delivery robots is crucial. If a food delivery robot has no blind spots, it can avoid obstacles in real time during operation, thus greatly improving delivery efficiency during peak periods. Therefore, surround-view perception is of great significance for improving the delivery efficiency of food delivery robots during peak hours.

[0003] In traditional technical solutions, panoramic image stitching is achieved by first performing an inverse perspective transformation (IPM) on the images and then stitching them together to form a panoramic stitched image. However, the stitching effect is not good, and the stitched image inevitably has obvious stitching edges. Summary of the Invention

[0004] Therefore, it is necessary to provide an image stitching device, method, smart device, robot, and storage medium to address the aforementioned technical problems.

[0005] An image stitching device includes a memory, a processor, and different cameras. The image stitching device has different image acquisition areas, and the different cameras are located in the different image acquisition areas. The memory stores computer-readable instructions that can be executed on the processor. When the processor executes the computer-readable instructions, it performs the following steps:

[0006] Acquire the images to be processed captured by the different cameras at the same time;

[0007] Multiple key points are selected in each image to be processed. Each key point corresponds to a target point in the world coordinate system. A transformation matrix is ​​calculated based on the key points and the target points.

[0008] The transformation matrix is ​​used to convert each image to be processed into an initial bird's-eye view, and the initial bird's-eye view collected in each image acquisition area is converted into a bird's-eye view to be processed.

[0009] Each of the bird's-eye view images to be processed is projected onto a preset position in the world coordinate system and then stitched together to obtain a stitched image.

[0010] In one embodiment, one of the image acquisition areas has a camera for capturing distant views and a camera for capturing close-up views, and the images to be processed include close-up images captured by the close-up camera and distant images captured by the distant camera.

[0011] The transformation matrix is ​​used to convert each image to be processed into an initial bird's-eye view, and the initial bird's-eye view acquired within each image acquisition area is converted into a bird's-eye view to be processed, including:

[0012] The transformation matrix is ​​used to convert the distant image and the close-up image into an initial distant bird's-eye view and an initial close-up bird's-eye view, respectively.

[0013] In the initial distant bird's-eye view and the initial close-up bird's-eye view, distant bird's-eye view image blocks and close-up bird's-eye view image blocks are selected according to the clarity of the initial distant bird's-eye view and the initial close-up bird's-eye view, respectively.

[0014] The distant bird's-eye view image block and the close-up bird's-eye view image block are stitched together to obtain the bird's-eye view image to be processed.

[0015] In one embodiment, the processor executing the computer-readable instructions further implements the following steps: before selecting a distant bird's-eye view image block and a close-up bird's-eye view image block based on the sharpness of the initial distant bird's-eye view and the initial close-up bird's-eye view, respectively, determining the sharpness of the initial distant bird's-eye view based on the pixels of the initial distant bird's-eye view and determining the sharpness of the initial close-up bird's-eye view based on the pixels of the initial close-up bird's-eye view.

[0016] In one embodiment, the processor executing the computer-readable instructions further performs the following steps: after acquiring images to be processed from different cameras, performing a distortion correction operation on each of the images to be processed.

[0017] In one embodiment, selecting multiple key points in each of the images to be processed specifically includes:

[0018] In response to the key point selection instruction, multiple key points specified by the key point selection instruction are selected in each of the images to be processed;

[0019] Alternatively, multiple key points in each of the images to be processed can be automatically detected.

[0020] In one embodiment, after projecting each of the bird's-eye view images to a preset position in the world coordinate system and stitching them together to obtain the stitched image, the method further includes:

[0021] When there are overlapping areas in the stitched image, the pixel values ​​in the overlapping areas of the stitched image are weighted to obtain weighted pixel values;

[0022] Calculate the average value of the weighted pixel values.

[0023] A smart device, the device comprising:

[0024] The image acquisition module is used to acquire images to be processed, captured by different cameras at the same time.

[0025] The calculation module is used to select multiple key points in each image to be processed, and the multiple key points correspond one-to-one with multiple target points in the world coordinate system, and calculate the transformation matrix based on the key points and the target points;

[0026] The transformation module is used to transform each image to be processed into an initial bird's-eye view through the transformation matrix, and to transform the initial bird's-eye view acquired in each image acquisition area into a bird's-eye view to be processed.

[0027] The projection module is used to project each of the bird's-eye views to be processed onto a preset position in the world coordinate system and stitch them together to obtain a stitched image.

[0028] A robot comprising the image stitching device as described above.

[0029] An image stitching method, the method comprising:

[0030] Acquire images to be processed from different cameras captured at the same time;

[0031] Multiple key points are selected in each of the images to be processed. Each key point corresponds to a target point in the world coordinate system. A transformation matrix is ​​calculated based on the key points and the target points.

[0032] The transformation matrix is ​​used to convert each image to be processed into an initial bird's-eye view, and the initial bird's-eye view collected in each image acquisition area is converted into a bird's-eye view to be processed.

[0033] Each of the bird's-eye view images to be processed is projected onto a preset position in the world coordinate system and then stitched together to obtain a stitched image.

[0034] A computer-readable storage medium stores a computer program that, when executed by a processor, implements an image stitching method, wherein the image stitching method is the image stitching method described above.

[0035] The aforementioned image stitching apparatus, method, intelligent device, robot, and storage medium acquire images to be processed, captured simultaneously by different cameras. Multiple key points are selected in each image to be processed, each key point corresponding one-to-one with multiple target points in the world coordinate system. A transformation matrix is ​​calculated based on the key points and target points. Using the transformation matrix, each image to be processed is converted into an initial bird's-eye view, and the initial bird's-eye views acquired within each image acquisition area are converted into bird's-eye views to be processed. Each bird's-eye view to be processed is projected onto a preset position in the world coordinate system and stitched together to obtain a stitched image. Through the transformation matrix, bird's-eye views to be processed in world coordinates are obtained. These bird's-eye views in world coordinates accurately correspond to different regions of the stitched image; therefore, the stitched image formed from these bird's-eye views has no stitching edges, thus improving the imaging quality of the stitched image. Attached Figure Description

[0036] Figure 1 This is an application environment diagram of the image stitching method in one embodiment;

[0037] Figure 2 This is a flowchart illustrating an image stitching method in one embodiment;

[0038] Figure 3 This is a schematic diagram of an image stitching method in one embodiment;

[0039] Figure 4 This is a flowchart illustrating the photometric optimization steps for a stitched image in one embodiment.

[0040] Figure 5 This is a structural block diagram of a smart device in one embodiment;

[0041] Figure 6 This is a block diagram of the smart device in another embodiment;

[0042] Figure 7 This is a diagram of the internal structure of a robot in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0045] The image stitching method provided in this application can be applied to, for example... Figure 1The application environment shown includes an image stitching device 102 and a cloud server 104.

[0046] Image stitching device 102 acquires images to be processed captured by different cameras at the same time. Multiple key points are selected in each image to be processed, and these key points correspond one-to-one with multiple target points in the world coordinate system. That is, one key point in each image to be processed corresponds to one target point in the world coordinate system. Based on the correspondence between key points and target points, a transformation matrix is ​​obtained. Using this transformation matrix, each image to be processed is converted into an initial bird's-eye view, and the initial bird's-eye view captured within each image acquisition area is converted into a bird's-eye view to be processed. That is, using the transformation matrix, points in each image to be processed are converted into points in the world coordinate system, resulting in a bird's-eye view in the world coordinate system. Each bird's-eye view to be processed is projected onto a preset position in the world coordinate system and stitched together to obtain a stitched image.

[0047] Among them, the cloud server 104 can be an independent physical server or a service node in the blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP).

[0048] In addition, cloud server 104 can also be a server cluster composed of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0049] The image stitching device 102 and the cloud server 104 can be connected via Bluetooth, USB (Universal Serial Bus) or network communication, and this application does not impose any restrictions on this.

[0050] In one embodiment, such as Figure 2 As shown, an image stitching device is provided, including a memory, a processor, and different cameras. The image stitching device has different image acquisition areas, and the different cameras are located in different image acquisition areas. The memory stores computer-readable instructions that can be executed by the processor. When the processor executes the computer-readable instructions, it includes the following steps:

[0051] S202: Acquire images to be processed from different cameras captured at the same time.

[0052] Different cameras are mounted on the image stitching device and located in different image acquisition areas of the device to capture images of the area surrounding the device. Each image acquisition area can have either a long-range or short-range camera, or both. It should be noted that the number of cameras in each image acquisition area can vary; one, two, or more cameras can be configured as needed.

[0053] In one embodiment, the camera can be a fisheye camera. For example, a fisheye camera with a short focal length and a wide field of view is installed in one image acquisition area of ​​the image stitching device to capture close-up images; and a fisheye camera with a long focal length and a narrow field of view is installed in the same image acquisition area of ​​the image stitching device to capture distant images.

[0054] In one embodiment, S202 may specifically include: the image to be processed may include a close-up image and a distant image; the image stitching device acquires distant and close-up images obtained by different cameras respectively capturing distant and close-up views.

[0055] For example, when an image acquisition area has a close-up camera and a distant camera, two cameras installed in the image acquisition area can be used to acquire distant and close-up images respectively. Specifically, a close-up image can be captured using a camera with a small focal length and a wide field of view installed in the image acquisition area; and a distant image can be captured using a camera with a large focal length and a narrow field of view installed in the same image acquisition area.

[0056] In one embodiment, when the accuracy requirement in the forward direction is high and the accuracy requirement in the side direction is low, the image stitching device can set up two cameras, one for close-up and one for distant view, in the image acquisition area located in the front to obtain close-up and distant view images. At the same time, the image stitching device can set up a close-up or distant view camera in the image acquisition area located on the side to obtain close-up or distant view images.

[0057] In one embodiment, the image stitching device can perform distortion correction on each image to be processed using either bilinear interpolation or linear interpolation. The distortion correction operation preserves all valid image information from the images acquired by different cameras, rather than cropping the maximum valid inscribed rectangle from the image after distortion correction, thus avoiding the loss of valid image information. Furthermore, the maximum valid inscribed rectangle refers to the largest inscribed rectangle of the region formed by the corresponding pixels of the original image and the distorted image. For example, when the image to be processed includes close-up and distant images of each image acquisition area, the image after distortion correction also includes close-up distorted images and distant distorted images from each shooting direction.

[0058] In one embodiment, the image stitching device performs denoising processing on the acquired image to be processed, performs brightness enhancement processing on the denoised image, and then executes S204.

[0059] In another embodiment, if the acquired image is blurry or has motion blur, the image stitching device can re-capture new images using different cameras.

[0060] S204. Select multiple key points in each image to be processed. The multiple key points correspond one-to-one with multiple target points in the world coordinate system. Calculate the transformation matrix based on the key points and target points.

[0061] In this system, multiple target points in the world coordinate system are several fixed points in the real world. By establishing the correspondence between these fixed points in the real world and the key points in the image, the image can be linked to the real world.

[0062] In one embodiment, multiple target points in the world coordinate system can be selected using a marker board, where the marker board refers to a large chessboard grid that establishes a unified world coordinate system with its origin located at the center of the grid. The multiple target points in the world coordinate system can refer to multiple pre-selected fixed points on the large chessboard grid. These fixed points all have known world coordinates, which are two-dimensional coordinates, such as [x...]. i ,y i Keypoints refer to the pixels that correspond to fixed points with known world coordinates, and the pixel coordinates of these pixels are also two-dimensional coordinates.

[0063] For example, there can be eight fixed points on a large chessboard grid. Four of these fixed points can be arranged side-by-side in front of the image stitching device, and the other four can be arranged side-by-side behind it. A camera located in front of the image stitching device can detect the four fixed points in front; a camera located behind the image stitching device can detect the four fixed points behind it; a camera located to the left of the image stitching device can detect the two fixed points to the left in front and the two fixed points to the left behind; a camera located to the right of the image stitching device can detect the two fixed points to the right in front and the two fixed points to the right behind; or, a camera to the left of the image stitching device can detect the four fixed points in front, and a camera to the right of the image stitching device can detect the four fixed points behind. Each camera can detect at least four fixed points.

[0064] In one embodiment, selecting multiple key points in each image to be processed in step S204 may specifically include: the image stitching device, in response to a key point selection instruction, selecting multiple key points specified by the key point selection instruction in each image to be processed. Here, the key points may be selected pixels.

[0065] For example, when a user clicks on an image to be processed, the image stitching device generates a key point selection instruction, and then selects the key point specified by the key point selection instruction in the image to be processed.

[0066] In another embodiment, selecting multiple key points in each image to be processed in S204 may specifically include: the image stitching device automatically detecting multiple key points in each image to be processed.

[0067] For example, image stitching devices can use the OpenCV open-source library to automatically detect multiple key points in each image to be processed. Specifically, the OpenCV library can utilize algorithms such as Moravec corner detection, Harris corner detection, and Shi-Tomas corner detection to automatically detect key points.

[0068] In one embodiment, calculating the transformation matrix based on key points and target points in S204 may specifically include: the image stitching device determining the mapping relationship between each key point and target point, and using the obtained mapping relationship as the transformation matrix.

[0069] S206, through the transformation matrix, transform each image to be processed into an initial bird's-eye view, and transform the initial bird's-eye view collected in each image acquisition area into a bird's-eye view to be processed.

[0070] A transformation matrix is ​​the process of transferring an image from the pixel coordinate system to the world coordinate system. Therefore, a transformation matrix can transform an image to be processed from a frontal view in the pixel coordinate system into a bird's-eye view from the top of the world coordinate system.

[0071] In one embodiment, an image acquisition area may have only a camera for capturing distant or close-up views, and the initial bird's-eye view may be directly used as the bird's-eye view to be processed.

[0072] In one embodiment, an image acquisition area has a camera for capturing distant views and a camera for capturing close-up views, and the images to be processed include close-up images captured by the close-up camera and distant images captured by the distant camera.

[0073] By using a transformation matrix, the distant and close-up images are converted into initial distant bird's-eye view and initial close-up bird's-eye view, respectively.

[0074] In the initial distant bird's-eye view and the initial close-up bird's-eye view, distant bird's-eye view image blocks and close-up bird's-eye view image blocks are selected according to their clarity, respectively; the distant bird's-eye view image blocks and close-up bird's-eye view image blocks are stitched together to obtain the bird's-eye view image to be processed.

[0075] Among them, the distant bird's-eye view image block is a relatively clear part of the initial distant bird's-eye view image, and the close-up bird's-eye view image block is a relatively clear part of the initial close-up bird's-eye view image.

[0076] Before selecting distant and close-up bird's-eye view image blocks based on their respective sharpness, the sharpness of the initial distant bird's-eye view can be determined based on the pixels of the initial distant bird's-eye view, and the sharpness of the initial close-up bird's-eye view can be determined based on the pixels of the initial close-up bird's-eye view.

[0077] In one embodiment, if the image to be processed is shifted after being converted into a bird's-eye view, making it impossible for the bird's-eye view to present the complete captured content, the image stitching device can adjust the pixel coordinates of the bird's-eye view to appropriately shift it until the complete captured content is presented.

[0078] S208 projects each bird's-eye view image to be processed onto a preset position in the world coordinate system and stitches them together to obtain a stitched image.

[0079] During the stitching process, two adjacent bird's-eye view images have two overlapping target points, which can be used to stitch them together.

[0080] Here we combine Figure 3 The above image stitching method will be further described, including the following:

[0081] Input the image captured by the camera; perform distortion correction on the image to obtain a distorted image; click the four target points in the image in sequence according to the distorted image order; calculate the transformation matrix; perform perspective transformation on the distorted image to obtain the bird's-eye view to be processed; project the bird's-eye view obtained in the previous step to the preset position in the world coordinate system, and perform photometric optimization on the stitched image.

[0082] The image stitching method described above involves acquiring images to be processed from different cameras; selecting multiple key points in each image, with each key point corresponding to multiple target points in the world coordinate system; calculating a transformation matrix based on the key points and target points; using the transformation matrix, converting each image into an initial bird's-eye view, and converting the initial bird's-eye view acquired within each image acquisition area into a bird's-eye view to be processed; and projecting each bird's-eye view to be processed onto a preset position in the world coordinate system for stitching to obtain a stitched image. By using the transformation matrix, bird's-eye views with known world coordinates are obtained. These bird's-eye views with known world coordinates can accurately correspond to different regions of the stitched image; therefore, the stitched image formed from these bird's-eye views has no stitching edges, improving the imaging quality of the stitched image.

[0083] In one embodiment, such as Figure 4 As shown, after S208, the method further includes:

[0084] S402, when there are overlapping areas in the stitched image, the pixel values ​​in the overlapping areas of the stitched image are weighted to obtain weighted pixel values.

[0085] S404 calculates the average of the weighted pixel values.

[0086] By weighting and averaging multiple pixels a and multiple pixels b, a stitched image is obtained that removes the brightness differences caused by overlapping image regions.

[0087] In the above embodiments, removing brightness differences caused by overlapping image regions helps improve the imaging quality of the stitched image.

[0088] It should be understood that, although Figure 2 , 4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , 4 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0089] In one embodiment, such as Figure 5 As shown, a smart device is provided, which can be a software module, a hardware module, or a combination of both as part of an image stitching device. Specifically, the device includes: an image acquisition module 502, a calculation module 504, a transformation module 506, and a projection module 508, wherein:

[0090] Image acquisition module 502 is used to acquire images to be processed captured by different cameras at the same time.

[0091] The calculation module 504 is used to select multiple key points in each image to be processed. The multiple key points correspond one-to-one with multiple target points in the world coordinate system. The transformation matrix is ​​calculated based on the key points and target points.

[0092] The transformation module 506 is used to convert each image to be processed into an initial bird's-eye view through the transformation matrix, and to convert the initial bird's-eye view collected in each image acquisition area into a bird's-eye view to be processed.

[0093] The projection module 508 is used to project each bird's-eye view to a preset position in the world coordinate system for stitching to obtain a stitched image.

[0094] In the above embodiments, images to be processed are acquired simultaneously by different cameras; multiple key points are selected in each image, and these key points correspond to multiple target points in the world coordinate system; a transformation matrix is ​​calculated based on the key points and target points; each image to be processed is converted into an initial bird's-eye view using the transformation matrix, and the initial bird's-eye view acquired within each image acquisition area is converted into a bird's-eye view to be processed; each bird's-eye view to be processed is projected onto a preset position in the world coordinate system and stitched together to obtain a stitched image. Through the transformation matrix, bird's-eye views with known world coordinates are obtained. These bird's-eye views with known world coordinates can accurately correspond to different regions of the stitched image; therefore, the stitched image formed by these bird's-eye views has no stitching edges, improving the imaging quality of the stitched image.

[0095] In one embodiment, the image to be processed includes a close-up image and / or a distant image; the smart device has different image acquisition areas, and different cameras are located in different image acquisition areas, one of which has cameras that capture distant and close-up views, and the image to be processed includes a close-up image captured by the close-up camera and a distant image captured by the distant camera.

[0096] The transformation module 506 is also used to transform the distant image and the close-up image into an initial distant bird's-eye view and an initial close-up bird's-eye view, respectively, through a transformation matrix;

[0097] In the initial distant bird's-eye view and the initial close-up bird's-eye view, select distant bird's-eye view image blocks and close-up bird's-eye view image blocks respectively based on the clarity of the initial distant bird's-eye view and the initial close-up bird's-eye view;

[0098] The bird's-eye view image is obtained by stitching together distant and close-up bird's-eye view image blocks.

[0099] In one embodiment, the transformation module 506 is further configured to determine the sharpness of the initial distant bird's-eye view based on the pixels of the initial distant bird's-eye view, and to determine the sharpness of the initial close-up bird's-eye view based on the pixels of the initial close-up bird's-eye view.

[0100] In one embodiment, such as Figure 6 The device also includes:

[0101] Image processing module 510 is used to perform distortion correction operation on each image to be processed;

[0102] The calculation module 504 is also used to select multiple key points in each image to be processed, which can be an image after distortion removal.

[0103] In one embodiment, the calculation module 504 is further configured to, in response to a key point selection instruction, select multiple key points specified by the key point selection instruction in each image to be processed; or, automatically detect multiple key points in each image to be processed.

[0104] In one embodiment, the projection module 508 is further configured to, when there is an overlapping area in the stitched image, weight the pixel values ​​in the overlapping area of ​​the stitched image to obtain weighted pixel values; and calculate the average value of the weighted pixel values.

[0105] For specific limitations regarding smart devices, please refer to the limitations on image stitching devices mentioned above, which will not be repeated here. Each module in the aforementioned smart device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the smart device in hardware form or independent of it, or they can be stored in the memory of the smart device in software form, so that the processor can call and execute the corresponding operations of each module.

[0106] In one embodiment, a robot is provided whose internal structure diagram can be as follows: Figure 7 As shown, the robot includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface allows for wired or wireless communication with external robots; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image stitching method. The robot's display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the robot's shell, or an external keyboard, touchpad, or mouse.

[0107] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the robot to which the present application is applied. A specific robot may include more or fewer parts than shown in the figure, or combine certain parts, or have different part arrangements.

[0108] In one embodiment, a robot is also provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps in the above method embodiments.

[0109] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0110] In one embodiment, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A robot's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the robot to perform the steps described in the method embodiments above.

[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0112] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image stitching device, comprising a memory, a processor, and different cameras, wherein the image stitching device has different image acquisition areas, the different cameras are located in the different image acquisition areas, and the memory stores computer-readable instructions executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it performs the following steps: Acquire images to be processed by the different cameras at the same time; wherein the image acquisition area has a camera for shooting distant scenes and a camera for shooting close-up scenes, and the images to be processed include close-up images taken by the close-up camera and distant images taken by the distant camera. A distortion correction operation is performed on each of the images to be processed; in the distortion correction operation, all valid image information on the images acquired by different cameras is retained, and the valid maximum inscribed rectangle in the image after the distortion correction operation is not truncated. Multiple key points are selected in the image to be processed after each distortion correction operation. These multiple key points correspond one-to-one with multiple target points in a unified world coordinate system. The multiple target points refer to multiple fixed points on a single large chessboard grid, which establishes a unified world coordinate system. The transformation matrix is ​​calculated based on the key points and the target points; The transformation matrix is ​​used to convert the distant image and the close-up image in the pixel coordinate system into an initial distant bird's-eye view and an initial close-up bird's-eye view in a unified world coordinate system, so that the world coordinates of the initial distant bird's-eye view and the initial close-up bird's-eye view are known. In the initial distant bird's-eye view and the initial close-up bird's-eye view, distant bird's-eye view image blocks and close-up bird's-eye view image blocks are selected according to their respective sharpness; wherein, the distant bird's-eye view image block is a relatively sharp part of the initial distant bird's-eye view; the close-up bird's-eye view image block is a relatively sharp part of the initial close-up bird's-eye view; The distant bird's-eye view image block and the close-up bird's-eye view image block are stitched together to obtain the bird's-eye view image to be processed; Each of the bird's-eye view images to be processed is projected onto a preset position in the world coordinate system and then stitched together to obtain a stitched image.

2. The image stitching device according to claim 1, characterized in that, The multiple key points are selected pixels.

3. The image stitching device according to claim 1, characterized in that, The processor, for executing the computer-readable instructions, also performs the following steps: before selecting a distant bird's-eye view image block and a close-up bird's-eye view image block respectively based on the sharpness of the initial distant bird's-eye view and the initial close-up bird's-eye view, determining the sharpness of the initial distant bird's-eye view based on the pixels of the initial distant bird's-eye view, and determining the sharpness of the initial close-up bird's-eye view based on the pixels of the initial close-up bird's-eye view.

4. The image stitching device according to claim 1, characterized in that, The camera in question is a fisheye camera.

5. The image stitching device according to claim 1, characterized in that, The selection of multiple key points in the image to be processed after each distortion correction operation specifically includes: In response to the key point selection instruction, multiple key points specified by the key point selection instruction are selected in the image to be processed after each distortion removal operation; Alternatively, multiple key points in the image to be processed can be automatically detected after each distortion correction operation.

6. The image stitching device according to claim 1, characterized in that, The step of projecting each of the bird's-eye views to be processed onto a preset position in the world coordinate system and stitching them together to obtain a stitched image further includes: When there are overlapping areas in the stitched image, the pixel values ​​in the overlapping areas of the stitched image are weighted to obtain weighted pixel values; Calculate the average value of the weighted pixel values.

7. A smart device, characterized in that, The device includes: An image acquisition module is used to acquire images to be processed captured by different cameras at the same time; wherein, the image acquisition area includes a camera for shooting distant scenes and a camera for shooting close-up scenes, and the images to be processed include close-up images captured by the close-up camera and distant images captured by the distant camera; a distortion correction operation is performed on each of the images to be processed; in the distortion correction operation, all valid image information on the images acquired by different cameras is retained, and the valid maximum inscribed rectangle in the image after the distortion correction operation is not truncated; The calculation module is used to select multiple key points in the image to be processed after each distortion correction operation. These multiple key points correspond one-to-one with multiple target points in a unified world coordinate system. The multiple target points refer to multiple fixed points on a single large chessboard grid, which establishes a unified world coordinate system. The transformation matrix is ​​calculated based on the key points and the target points. A transformation module is used to convert the distant image and the close-up image in the pixel coordinate system into an initial distant bird's-eye view and an initial close-up bird's-eye view in a unified world coordinate system, respectively, through the transformation matrix, so that the world coordinates of the initial distant bird's-eye view and the initial close-up bird's-eye view are known; in the initial distant bird's-eye view and the initial close-up bird's-eye view, distant bird's-eye view image blocks and close-up bird's-eye view image blocks are selected according to the sharpness of the initial distant bird's-eye view and the initial close-up bird's-eye view, respectively; wherein, the distant bird's-eye view image block is a relatively clear part of the initial distant bird's-eye view; the close-up bird's-eye view image block is a relatively clear part of the initial close-up bird's-eye view; the distant bird's-eye view image blocks and close-up bird's-eye view image blocks are stitched together to obtain a bird's-eye view image to be processed; a projection module is used to project each of the bird's-eye view images to be processed onto a preset position in the world coordinate system for stitching to obtain a stitched image.

8. A robot, characterized in that, The robot includes an image stitching device as described in any one of claims 1 to 6.

9. An image stitching method, characterized in that, The method includes: Acquire images to be processed by different cameras at the same time; wherein, the image acquisition area has a camera for shooting distant scenes and a camera for shooting close-up scenes, and the images to be processed include close-up images taken by the close-up camera and distant images taken by the distant camera. A distortion correction operation is performed on each of the images to be processed; in the distortion correction operation, all valid image information on the images acquired by different cameras is retained, and the valid maximum inscribed rectangle in the image after the distortion correction operation is not truncated. Multiple key points are selected in the image to be processed after each distortion correction operation. Each key point corresponds one-to-one with multiple target points in a unified world coordinate system. The multiple target points refer to multiple fixed points on a single large chessboard grid, and the large chessboard grid establishes a unified world coordinate system. The transformation matrix is ​​calculated based on the key points and the target points; The transformation matrix is ​​used to convert the distant image and the close-up image in the pixel coordinate system into an initial distant bird's-eye view and an initial close-up bird's-eye view in a unified world coordinate system, so that the world coordinates of the initial distant bird's-eye view and the initial close-up bird's-eye view are known. In the initial distant bird's-eye view and the initial close-up bird's-eye view, distant bird's-eye view image blocks and close-up bird's-eye view image blocks are selected according to their respective sharpness; wherein, the distant bird's-eye view image block is a relatively sharp part of the initial distant bird's-eye view; the close-up bird's-eye view image block is a relatively sharp part of the initial close-up bird's-eye view; The distant bird's-eye view image block and the close-up bird's-eye view image block are stitched together to obtain the bird's-eye view image to be processed; Each of the bird's-eye view images to be processed is projected onto a preset position in the world coordinate system and then stitched together to obtain a stitched image.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements an image stitching method, which is the image stitching method according to claim 9.

Images