Robot positioning method and device based on multi-view image fusion and electronic equipment

Abstract

According to the multi-view image fusion robot positioning method and device and the electronic equipment provided by the invention, the target images are firstly obtained, then the homography matrix and the corresponding inverse matrix of each target image are calculated, accurate mathematical support is provided for image splicing and view angle conversion, and based on the homography matrix of each target image, the multi-view image fusion robot positioning method and device are provided. The method comprises the steps of projecting pixel coordinates in a to-be-spliced image to a coordinate system of a reference image to obtain a spliced image, dividing the spliced image into four image areas as reference image sectors according to map angular point coordinates in the spliced image, establishing correlation between local and global angular points through sector division, and ensuring the orderliness of subsequent processing. And based on the inverse matrix, mapping the pixel point coordinates in each reference image sector to a top view coordinate system to obtain a top view corresponding to the target image, and based on the front end coordinates and the angular point mass center of the robot in the top view, determining the position and the course of the robot, thereby improving the accuracy and the robustness of robot positioning.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot localization method, apparatus and electronic device for multi-view image fusion. Background Technology

[0002] In smart manufacturing plants, robots are widely used in key processes such as material handling, quality inspection, and equipment maintenance. Their core capabilities rely on high-precision positioning systems. Robots need to perceive their own position and posture in real time in dynamic environments to ensure the accuracy and efficiency of task execution.

[0003] In related technologies, robot localization methods mainly rely on single-view image processing or multi-view stitching under a fixed camera layout. These methods have strict requirements on the camera installation angle and position, and the stitching results are easily affected by changes in lighting and occlusion.

[0004] Therefore, improving the accuracy of robot positioning has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a robot localization method, apparatus, and electronic device based on multi-view image fusion to address the problem of low accuracy in robot localization in related technologies.

[0006] In a first aspect, embodiments of this application provide a robot localization method using multi-view image fusion, including:

[0007] Acquire at least one target image; each target image includes images of the robot from different perspectives, wherein the image from the preset perspective is the reference image, and the other images besides the reference image are the images to be stitched together.

[0008] Calculate the homography matrix and the inverse matrix of each target image.

[0009] Based on the homography matrix corresponding to each target image, the pixel coordinates in the image to be stitched are projected onto the coordinate system of the reference image to obtain the stitched image;

[0010] Based on the coordinates of the map corner points in the stitched image, the stitched image is divided into four image regions, which serve as the reference image sectors;

[0011] Based on the inverse matrix, the pixel coordinates in each sector of the reference image are mapped to the top view coordinate system to obtain the top view corresponding to the target image;

[0012] Based on the robot's front-end coordinates and the centroids of its corners in the top view, the robot's position and heading are determined.

[0013] In one possible implementation, based on the homography matrix corresponding to each target image, the pixel coordinates in the image to be stitched are projected onto the coordinate system of the reference image to obtain the stitched image, including:

[0014] Geometric transformations are performed on each target image based on the homography matrix, and a stitching canvas covering the reference image and the images to be stitched is created in the coordinate system of the reference image.

[0015] The target stitching canvas is determined based on the overlap information between the reference image and the image to be stitched, as well as the stitching canvas.

[0016] Using the homography matrix, all corner points of the image to be stitched are mapped to the coordinate system of the reference image;

[0017] In the coordinate system of the reference image, the coordinates of the same corner point of the image to be stitched and the reference image are averaged to obtain the coordinates of the target corner point;

[0018] Project the coordinates of the target corner points onto the target splicing canvas to obtain the spliced ​​image.

[0019] In one possible implementation, determining the target stitching canvas based on the overlap information between the reference image and the image to be stitched, and the stitching canvas, includes:

[0020] A perspective transformation is performed on all images to be stitched using a homography matrix to generate an aligned image that is aligned with the viewpoint of the reference image.

[0021] The splicing canvas is filled based on the alignment image and overlap information until all pixels of the alignment image and the reference image are filled into the splicing canvas, thus obtaining the target splicing canvas.

[0022] In one possible implementation, filling the spliced ​​canvas based on aligned images and overlap information includes:

[0023] When the overlap information indicates the existence of an overlapping area, the pixels of the overlapping area between the aligned image and the reference image are mixed using a preset weight and filled into the splicing canvas.

[0024] When the overlap information indicates that there is no overlapping area, the pixels corresponding to the non-overlapping areas of the aligned image and the image to be stitched are respectively filled into the stitching canvas.

[0025] In one possible implementation, the stitched image is divided into four image regions as reference image sectors based on the coordinates of map corner points in the stitched image, including:

[0026] Obtain the coordinates of map corner points in the stitched image;

[0027] The centroid of the corner point is calculated based on the coordinates of the corner point on the map.

[0028] The stitched image is divided into four image regions centered on the centroid of the corner point to obtain the reference image sector.

[0029] In one possible implementation, based on the inverse matrix, the pixel coordinates in each reference image sector are mapped to the top view coordinate system to obtain the top view corresponding to the target image, including:

[0030] Obtain the pixel coordinates and preset top view of the sector in the base image; the preset top view is determined based on the coordinates of map corner points in the stitched image;

[0031] The target pixel is obtained by weighting the pixels around it using a bilinear interpolation algorithm; the target pixel is any pixel in the sector of the reference image.

[0032] Fill the target weighted pixels into the preset top view until all pixels of the stitched image are filled, and obtain the top view corresponding to the target image.

[0033] In one possible implementation, the robot's front-end coordinates are determined based on a preset mapping relationship between corner points and front-end coordinates; the robot's corner centroids are determined based on the coordinates of each corner point of the robot.

[0034] Secondly, embodiments of this application provide a robot localization device for multi-view image fusion, comprising:

[0035] The acquisition module is used to acquire at least one target image; each target image includes images of the robot from different perspectives, wherein the image from the preset perspective is the reference image, and the other images besides the reference image are images to be stitched together.

[0036] The calculation module is used to calculate the homography matrix and the inverse matrix corresponding to each target image, respectively.

[0037] The processing module is used to project the pixel coordinates in the image to be stitched onto the coordinate system of the reference image based on the homography matrix corresponding to each target image, so as to obtain the stitched image.

[0038] The processing module is also used to divide the stitched image into four image regions as reference image sectors based on the coordinates of map corner points in the stitched image;

[0039] The determination module is used to map the pixel coordinates in each sector of the reference image to the top view coordinate system based on the inverse matrix, so as to obtain the top view corresponding to the target image.

[0040] The determination module is also used to determine the robot's position and heading based on the robot's front-end coordinates and the centroids of its corners in the top view.

[0041] In one possible implementation, the processing module is specifically used for:

[0042] Geometric transformations are performed on each target image based on the homography matrix, and a stitching canvas covering the reference image and the images to be stitched is created in the coordinate system of the reference image.

[0043] The target stitching canvas is determined based on the overlap information between the reference image and the image to be stitched, as well as the stitching canvas.

[0044] Using the homography matrix, all corner points of the image to be stitched are mapped to the coordinate system of the reference image;

[0045] In the coordinate system of the reference image, the coordinates of the same corner point of the image to be stitched and the reference image are averaged to obtain the coordinates of the target corner point;

[0046] Project the coordinates of the target corner points onto the target splicing canvas to obtain the spliced ​​image.

[0047] In one possible implementation, the determining module is specifically used for:

[0048] A perspective transformation is performed on all images to be stitched using a homography matrix to generate an aligned image that is aligned with the viewpoint of the reference image.

[0049] The splicing canvas is filled based on the alignment image and overlap information until all pixels of the alignment image and the reference image are filled into the splicing canvas, thus obtaining the target splicing canvas.

[0050] In one possible implementation, the spliced ​​canvas is filled based on the aligned image and overlap information, and the determining module is further configured to:

[0051] When the overlap information indicates the existence of an overlapping area, the pixels of the overlapping area between the aligned image and the reference image are mixed using a preset weight and filled into the splicing canvas.

[0052] When the overlap information indicates that there is no overlapping area, the pixels corresponding to the non-overlapping areas of the aligned image and the image to be stitched are respectively filled into the stitching canvas.

[0053] In one possible implementation, the processing module is specifically used for:

[0054] Obtain the coordinates of map corner points in the stitched image;

[0055] The centroid of the corner point is calculated based on the coordinates of the corner point on the map.

[0056] The stitched image is divided into four image regions centered on the centroid of the corner point to obtain the reference image sector.

[0057] In one possible implementation, the determining module is specifically used for:

[0058] Obtain the pixel coordinates and preset top view of the sector in the base image; the preset top view is determined based on the coordinates of map corner points in the stitched image;

[0059] The target pixel is obtained by weighting the pixels around it using a bilinear interpolation algorithm; the target pixel is any pixel in the sector of the reference image.

[0060] Fill the target weighted pixels into the preset top view until all pixels of the stitched image are filled, and obtain the top view corresponding to the target image.

[0061] In one possible implementation, the robot's front-end coordinates are determined based on a preset mapping relationship between corner points and front-end coordinates; the robot's corner centroids are determined based on the coordinates of each corner point of the robot.

[0062] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0063] The memory stores the instructions that the computer executes;

[0064] The processor executes computer-executable instructions stored in memory to implement the method as described in the first aspect or any of the above.

[0065] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect or any of the above-mentioned methods.

[0066] Fifthly, embodiments of this application provide a computer program. The computer program product includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the methods described in the first aspect or any of the above-mentioned methods.

[0067] The robot localization method, apparatus, and electronic device for multi-view image fusion provided in this application first acquire at least one target image. Each target image includes images of the robot from different viewpoints, where the image from a preset viewpoint is the reference image, and the other images are images to be stitched. Then, the homography matrix and the inverse matrix corresponding to each target image are calculated respectively to provide accurate mathematical support for image stitching and viewpoint conversion. Based on the homography matrix corresponding to each target image, the pixel coordinates in the images to be stitched are projected onto the coordinate system of the reference image to obtain the stitched image. According to the map corner coordinates in the stitched image, the stitched image is divided into four image regions as reference image sectors. The sector division establishes the association between local and global corner points to ensure the orderliness of subsequent processing. Based on the inverse matrix, the pixel coordinates in each reference image sector are mapped to the top view coordinate system to obtain the top view corresponding to the target image. Based on the front-end coordinates of the robot and its corner centroids in the top view, the position and heading of the robot are determined, effectively canceling detection noise and improving the accuracy and robustness of robot localization, meeting the localization needs of tasks such as factory material transportation and quality inspection. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0069] Figure 1 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 1 ;

[0070] Figure 2 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 2 ;

[0071] Figure 3 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 3 ;

[0072] Figure 4 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 4 ;

[0073] Figure 5 A schematic diagram illustrating robot localization using multi-view image fusion as provided in an embodiment of this application;

[0074] Figure 6 A schematic diagram of the structure of the robot localization device for multi-view image fusion provided in an embodiment of this application;

[0075] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0076] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] Before introducing the embodiments of this application, the application background of the embodiments of this application will be explained first:

[0079] In smart manufacturing factories, robots are widely used in key areas such as material handling, quality inspection, and equipment maintenance. Their core capability relies on high-precision positioning systems. Factory workshops typically have a rectangular layout with clearly defined boundaries between production lines and work areas. Robots need to perceive their own position and posture in real time within a dynamic environment to ensure the accuracy and efficiency of task execution. However, factory environments often require multi-camera collaborative operation, with multiple cameras distributed in different locations capturing images from multiple perspectives to obtain information about the robot and its surrounding environment. Due to limitations in camera installation locations, significant differences in viewing angles, and environmental obstructions, traditional positioning methods struggle to achieve efficient fusion of multi-view images and accurate positioning within a unified coordinate system.

[0080] Existing technologies typically rely on manually configuring camera parameters and fixed layouts, leading to complex system deployments, poor adaptability, and the need for recalibration when camera positions are adjusted or the environment changes. This severely limits the flexibility and real-time performance requirements of robots in complex industrial scenarios. Furthermore, the fast pace of factory production necessitates rapid robot responses to task scheduling, placing higher demands on the computational efficiency and robustness of the localization system. Therefore, there is an urgent need for a universal localization method that does not strictly depend on camera configuration and can adapt to multi-view image fusion to improve the autonomous navigation capabilities of robots in dynamic industrial environments.

[0081] Furthermore, robot localization methods also rely on single-view image processing or multi-view stitching with a fixed camera layout. For example, traditional feature point matching-based image stitching methods require manually defining camera parameters and correcting distortion, and achieving image alignment by calculating the homography matrix. However, this method has strict requirements on camera installation angles and positions, and the stitching results are easily affected by changes in lighting and occlusion. In addition, some solutions use visual markers for localization, but this requires pre-fixing multiple markers on the map boundary and strictly defining the relative positional relationship between the camera and the markers, resulting in high system deployment costs and poor adaptability. For multi-camera scenarios, existing technologies typically achieve image fusion through fixed camera arrays and preset coordinate systems. However, such methods cannot adapt to dynamic adjustments in camera positions and suffer from high computational complexity and accumulated stitching errors when processing multi-view images. All of the above technical solutions have shortcomings such as insufficient versatility, complex configuration, and poor real-time performance, making it difficult to meet the requirements of modern intelligent manufacturing for the flexibility and reliability of robot localization systems.

[0082] In summary, improving the accuracy of robot positioning has become an urgent technical problem that needs to be solved.

[0083] To address the technical problems existing in related technologies, the inventors of this application propose the following solution: For the issue of low accuracy in robot localization, by selecting a preset viewpoint as a reference and calculating the homography matrix between each viewpoint image and the reference image, the pixel coordinates of multiple images to be stitched are precisely projected onto the same reference coordinate system, forming a stitched image with a wider coverage area. This fundamentally expands the perceptual field of view and achieves preliminary data fusion. Secondly, to avoid edge distortion caused by direct transformation, the stitched image is divided into multiple reference image sectors based on map corner points. Using the inverse of the homography matrix, the pixels of each sector are mapped to the top-view coordinate system, maximizing the preservation of the geometric accuracy of the original image. The generated distortion-free top view provides an ideal observation plane for subsequent localization. Finally, precise pose calculation is performed in the standardized top view. Under this coordinate system, the robot's geometric projection is accurately restored. Precise two-dimensional plane coordinates can be obtained by calculating the centroid of its corner points, and the accurate heading angle can be directly calculated based on the preset front-end coordinate definition. This invention transforms multi-view information into a unified, precise pose output, significantly improving the accuracy and reliability of robot localization in complex scenarios.

[0084] The parts not described in detail are disclosed in the following embodiments.

[0085] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0086] It is worth noting that the application fields of the methods, apparatus, electronic devices and storage media in this application are not limited.

[0087] The subject of this application is an electronic device, which may specifically be a server, terminal device, etc.

[0088] Figure 1 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, the method may include the following steps:

[0089] Step 11: Obtain at least one target image.

[0090] Each target image includes images of the robot from different perspectives. The image from the preset perspective is the reference image, and the other images are the images to be stitched together.

[0091] In this step, a sequence of images containing the target robot is captured synchronously or asynchronously using global cameras fixed around the robot's activity area. The target images must be able to show visual markers (such as AprilTag visual markers) on the robot from at least different angles.

[0092] To establish a unified coordinate reference, a viewpoint must be pre-specified as the reference image, and its image coordinate system will serve as the reference coordinate system for subsequent image stitching. Images captured from all other viewpoints are defined as images to be stitched. This step provides the basic input data for subsequent multi-view data fusion.

[0093] The robot's activity area is a rectangular or standard factory terrain, and the encoding of the visual tags is stored in the configuration dictionary.

[0094] For example, the image captured by the camera located due north of the site is the baseline image.

[0095] Step 12: Calculate the homography matrix and the inverse matrix corresponding to each target image.

[0096] In this step, a homography matrix is ​​calculated by identifying common feature points (usually visually marked corner points) between each target image and a pre-defined world plane model. The homography matrix of the image to be stitched defines the forward projection relationship from the coordinate system of the image to be stitched to the world plane. The homography matrix of the reference image defines the forward projection relationship from the coordinate system of the reference image to the world plane. Simultaneously, the inverse matrix of each homography matrix is ​​calculated and stored. This inverse matrix is ​​crucial for mapping back from the world plane to the respective image coordinate systems, providing the necessary transformation tools for generating perspective-distortion-free top views in subsequent steps.

[0097] The homography matrix is ​​a 3x3 transformation matrix that encapsulates the rotation and translation transformations caused by camera perspective projection, enabling it to linearly map the pixel coordinates in the current image to the corresponding coordinates on the world plane.

[0098] Step 13: Based on the homography matrix corresponding to each target image, project the pixel coordinates in the image to be stitched onto the coordinate system of the reference image to obtain the stitched image.

[0099] In this step, a stitching canvas large enough to accommodate all transformed images is created, with its coordinate system consistent with that of the reference image. Then, using the homography matrix of each image to be stitched, calculated in step 12, a perspective transformation is performed on the images to be stitched. The coordinates of all pixels in the images to be stitched are mapped to the coordinate system of the reference image through the homography matrix, generating an aligned image with the viewpoint of the reference image. Next, a corresponding mask is created for each aligned image. The mask identifies the overlapping areas (i.e., the same map corner or robot corner contained in both images) and non-overlapping areas between the aligned and reference images. Pixels from the aligned and reference images are filled into the stitching canvas, progressively filling all images into the same stitching canvas. Simultaneously, the coordinates of the detected map corners and robot corners in each image are fused. The average coordinate of the same corner after mapping in different images is calculated, and the fused corner coordinates are combined with the fused pixel results to obtain a stitched image covering the entire working area. This image has the same coordinate system as the reference image and contains scene information captured by all cameras.

[0100] For example, when two images detect the same corner point, the average coordinates of that corner point need to be taken. Corner point 1 is marked as d1 in the image to be stitched and d2 in the reference image. The symmetrical difference between corner point 1 in the image to be stitched and the reference image is d1∆d2. Then, all corner points in the stitched image are marked... It can be represented as follows:

[0101]

[0102] The positions of the aforementioned corner points can be represented by the following formula:

[0103]

[0104] In the formula, ( , ) represents the coordinates of corner point 1 in the image to be stitched. , () represents the coordinates of corner point 1 in the reference image. It is the set of x-axis coordinates of corner point 1 in the image to be stitched and the reference image. It is the set of y-axis coordinates of corner point 1 in the image to be stitched and the reference image. These are the x-axis and y-axis coordinates of corner point 1 in the stitched image, respectively.

[0105] Step 14: Based on the coordinates of the map corner points in the stitched image, divide the stitched image into four image regions as reference image sectors.

[0106] In this step, the coordinates of all map corner points (i.e., the coordinates of the points) are extracted from the stitched image. Figure 4 The pixel coordinates of the AprilTag markers in the stitched image are used to identify corners. If some map corners are not detected due to shooting angle limitations, the missing corners are filled in using an extrapolation algorithm based on the aspect ratio of the detected corner coordinates and the map corresponding to the robot's activity area. Next, the coordinates of all filled map corners are processed. If the corner orientations are inconsistent due to different camera orientations, the images are flipped to ensure consistent orientations for all corners. Then, the centroid of all map corner coordinates is calculated by averaging the x and y coordinates of all corners to obtain the centroid coordinates (cx, cy). Finally, two perpendicular dividing lines are drawn around this centroid, dividing the stitched image into four equal sectors. Each sector corresponds to a quadrant of the rectangular map, and each sector corresponds to a corner, thus establishing a connection between corners in the local view and global map corners.

[0107] The missing corner points are filled by extrapolation algorithm. For example, if the top left and bottom right corner points have been detected, the coordinates of the top right and bottom left corner points can be extrapolated according to the aspect ratio of the map.

[0108] For example, the upper left sector corresponds to the upper left corner of the map, and the lower right sector corresponds to the lower right corner of the map.

[0109] Step 15: Based on the inverse matrix, map the pixel coordinates in each sector of the reference image to the top view coordinate system to obtain the top view corresponding to the target image.

[0110] In this step, the stitched image is converted into a top view (i.e., a top-down perspective) through coordinate transformation and pixel reconstruction, providing an intuitive and unified viewpoint for robot localization.

[0111] The coordinate system of the top view is determined based on a preset ideal rectangular virtual projection. This ideal rectangle has the same aspect ratio as the map corresponding to the robot's activity area, and its four vertices correspond to the map's coordinate system. Figure 4The specific transformation process of the angle coordinates is as follows: First, for each pixel (x2, y2) in the sector of the reference image, its coordinates are expanded into three-dimensional homogeneous coordinates (x3, y3, 1); second, these homogeneous coordinates are multiplied by the inverse matrix H_left_inv of the homography matrix obtained in step 12 to obtain the mapped homogeneous coordinates (u, v, w), i.e. Then, the coordinates in the top-view coordinate system (k=u / w, h=v / w) are obtained through normalization. Finally, a bilinear interpolation algorithm is used to calculate the pixel value corresponding to each coordinate in the top view. If the mapped coordinate does not correspond to an integer pixel, the pixel value of the surrounding pixels is obtained by weighted averaging. After mapping the coordinates of all pixels and filling in the pixel values, a complete top view is obtained. This top view can fully present the boundaries of the map, the internal area, and the position of the robot, and the coordinates of all pixels have a fixed proportional relationship with the physical position of the actual map.

[0112] For example, if the actual map is 5m long and 3m wide, the ideal rectangle is set to 500 pixels long and 300 pixels wide, maintaining a 1:100 scale.

[0113] For example, if the coordinates of a pixel in the stitched image are (200, 300), after mapping with H_left_inv, the homogeneous coordinates are (400, 200, 1), and the normalized top view coordinates are (400, 200).

[0114] Step 16: Determine the robot's position and heading based on the robot's front-end coordinates and the centroids of its corners in the top view.

[0115] Specifically, the robot's front-end coordinates are determined based on the mapping relationship between preset corner points and front-end coordinates; the robot's corner centroids are determined based on the coordinates of each corner point of the robot.

[0116] In this step, the corner information of the robot is identified from the top view. The corner information includes the coordinates of the four corner points of the robot extracted based on the identifiable markers carried by the robot. Then, the centroid of the robot's corners (i.e., the geometric center of the robot) is calculated. Specifically, the average of the x-coordinates and y-coordinates of the four corner points is taken to obtain the coordinates of the robot's corner centroid (r). cx , r cy Simultaneously, the front-end coordinates (r) are determined from the robot's corner information based on a preset mapping relationship. fx , r fy Finally, based on the fixed proportional relationship between the top-view coordinate system and the actual map, the robot's centroid coordinates (r) are determined. cx , r cy Convert to actual physical coordinates according to this ratio (e.g., r).cx =400 pixels, r cy =200 pixels, corresponding to actual coordinates (4m, 2m)), which is the robot's actual position on the rectangular map.

[0117] Finally, based on the robot's front-end coordinates and center-of-mass coordinates, the heading angle is calculated using the arctangent function. The specific formula is as follows:

[0118] Heading angle θ = ( -arctan ((r fy -r cy ), (r fx -r cx )) + 2π ) mod 2π - π / 2;

[0119] In the formula, the arctan function calculates the initial angle based on the coordinate difference between the front end and the centroid. Adding 2π ensures that the angle is positive, taking the modulus of 2π restricts the angle range to [0, 2π), subtracting π / 2 makes the heading angle start counting from the north (i.e., directly above) of the map, and multiplying by -1 ensures that the angle increases clockwise.

[0120] Through the above calculations, the robot's precise position and heading in the actual working area are obtained, providing positioning support for tasks such as autonomous navigation and material transportation.

[0121] For example, the preset mapping relationship is defined during initialization as the association between the robot's front-end marker and the corner point. The marker on the robot's front end corresponds to a specific corner point, and the top view coordinates of that corner point are the front-end coordinates.

[0122] The robot localization method based on multi-view image fusion provided in this application first acquires at least one target image. Each target image includes images of the robot from different viewpoints, where the image from a preset viewpoint is the reference image, and the other images are images to be stitched. Then, the homography matrix and the inverse matrix corresponding to each target image are calculated respectively, providing accurate mathematical support for image stitching and viewpoint conversion. Based on the homography matrix corresponding to each target image, the pixel coordinates in the images to be stitched are projected onto the coordinate system of the reference image to obtain the stitched image. According to the map corner coordinates in the stitched image, the stitched image is divided into four image regions as reference image sectors. The sector division establishes the relationship between local and global corner points, ensuring the orderliness of subsequent processing. Based on the inverse matrix, the pixel coordinates in each reference image sector are mapped to the top view coordinate system to obtain the top view corresponding to the target image. Based on the front-end coordinates of the robot and its corner centroids in the top view, the position and heading of the robot are determined, effectively offsetting detection noise and improving the accuracy and robustness of robot localization, meeting the localization needs of tasks such as factory material transportation and quality inspection.

[0123] Based on the above embodiments, Figure 2 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, step 13 may include the following steps:

[0124] Step 21: Perform geometric transformations on each target image based on the homography matrix, and create a stitching canvas covering the reference image and the images to be stitched in the coordinate system of the reference image.

[0125] In this step, for each image to be stitched, a geometric transformation is performed using its corresponding homography matrix. This maps the pixel coordinates of the images to be stitched to the coordinate system of the reference image, eliminating geometric distortion caused by different viewpoints and ensuring that the scene viewpoints of the images to be stitched are consistent with those of the reference image. Subsequently, a stitching canvas is created in the coordinate system of the reference image. By calculating the dimensions of the reference image and the maximum boundary range of all geometrically transformed images to be stitched, the length and width of the canvas are determined. This ensures that the canvas can completely accommodate all pixel information of the reference image and all images to be stitched, providing a unified carrier for subsequent image fusion.

[0126] The homography matrix describes the perspective transformation relationship between the image to be stitched and the reference image.

[0127] Step 22: Determine the target stitching canvas based on the overlap information between the reference image and the image to be stitched, as well as the stitching canvas.

[0128] In this step, image masking technology is used to identify the overlap information between the geometrically transformed image to be stitched and the reference image, clarifying the boundaries of overlapping and non-overlapping areas. For overlapping areas, an equal-weighted fusion strategy is used to fuse the pixel values ​​of the two images, avoiding uneven brightness or residual contours in the overlapping parts. For non-overlapping areas, the original pixel values ​​of the reference image or the image to be stitched are directly filled into the corresponding positions on the stitching canvas to ensure that no scene information is lost. All images to be stitched are processed one by one according to the above rules, gradually filling their pixels into the stitching canvas, ultimately resulting in a target stitching canvas that covers the complete scene of the reference image and all images to be stitched, without any stitching defects.

[0129] For example, the weights of both images are set to 0.5.

[0130] Specifically, step 22 may include the following implementation methods:

[0131] Step 1: Use homography matrix to perform perspective transformation on all images to be stitched to generate an aligned image that is aligned with the viewpoint of the reference image.

[0132] In this implementation, for each image to be stitched, a perspective transformation operation is performed by calling its pre-solved homography matrix. This matrix maps the coordinates of all pixels in the image to be stitched to the coordinate system of the reference image, eliminating geometric distortions such as tilt, scaling, and offset caused by different camera shooting angles, ensuring that the scene perspective of the image to be stitched is completely consistent with that of the reference image.

[0133] For example, if the images to be stitched are tilted images taken by the right-side camera, after homography matrix perspective transformation, their scene layout and object proportions will be aligned with the reference image taken from the front, ultimately generating an aligned image corresponding to each image to be stitched.

[0134] Step 2: Fill the splicing canvas based on the alignment image and overlap information until all pixels of the alignment image and the reference image are filled into the splicing canvas to obtain the target splicing canvas.

[0135] In this implementation, image mask comparison and coordinate range verification are used to clarify the overlap information between each aligned image and the reference image, as well as between the aligned images. For overlapping areas, an equal-weight pixel fusion strategy is used to avoid contour ghosting or brightness differences in the overlapping parts, ensuring a natural fusion transition. For non-overlapping areas, the original pixel values ​​of the corresponding images are directly filled into the corresponding coordinate positions of the stitching canvas to ensure that no scene information is missed. Following the order of filling the reference image first, and then filling the aligned images one by one, all images are processed sequentially until the pixels of the reference image and all aligned images are completely filled into the stitching canvas, ultimately forming a target stitching canvas that covers the entire scene, has no stitching defects, and has a unified viewpoint.

[0136] The above-mentioned filling of the spliced ​​canvas based on aligned images and overlap information can be implemented in the following ways:

[0137] S1. When the overlap information indicates the existence of an overlapping area, the pixels of the overlapping area between the aligned image and the reference image are mixed using a preset weight and filled into the splicing canvas.

[0138] S2. When the overlap information indicates that there is no overlapping area, fill the non-overlapping areas of the aligned image and the image to be stitched into the stitching canvas.

[0139] Step 23: Using the homography matrix, map all corner points of the image to be stitched to the coordinate system of the reference image.

[0140] In this step, the perspective transformation of the pixel coordinates of all corner points in the image to be stitched is calculated using the homography matrix corresponding to the image to be stitched, and then mapped to the coordinate system of the reference image one by one to obtain the corresponding coordinates of each corner point in the reference coordinate system.

[0141] Among them, corner points include ground Figure 4 Visual positioning markers for corners (such as AprilTag markers) and corner markers for the robot itself.

[0142] For example, if the original coordinates of a map corner point in the image to be stitched are (x5, y5), after calculation using the homography matrix H_left, its mapped coordinates in the reference image coordinate system (x5', y5') are obtained, ensuring that all corner point information can be processed in the same coordinate system.

[0143] Step 24: In the coordinate system of the reference image, calculate the average value of the coordinates of the same corner point of the image to be stitched and the reference image to obtain the coordinates of the target corner point.

[0144] In this step, within the coordinate system of the reference image, the same corner points in the image to be stitched and the reference image are first selected based on the label encoding or spatial location features of each corner point. Then, the original coordinates of this same corner point in the reference image and its mapped coordinates in the image to be stitched are averaged. Specifically, the x-axis components and y-axis components of the two coordinates are averaged to obtain the target corner point coordinates. This processing method can cancel out the random noise in corner point detection within a single image, making the corner point coordinates closer to the actual physical location and providing an accurate reference for subsequent positioning calculations.

[0145] Step 25: Project the coordinates of the target corner points onto the target splicing canvas to obtain the spliced ​​image.

[0146] In this step, the coordinates of all target corner points are precisely projected onto the target stitching canvas determined in step 22, according to their corresponding positions in the reference image coordinate system, ensuring that the target corner point coordinates are accurately aligned with the scene pixels in the canvas. At this point, the target stitching canvas not only contains the complete scene pixels of the reference image and all images to be stitched, but also integrates the optimized corner point coordinate information, forming a stitched image that combines scene integrity and corner point accuracy, providing high-quality image data support for subsequent sector division, viewpoint transformation, and robot localization.

[0147] The robot localization method for multi-view image fusion provided in this application first performs geometric transformation on each target image based on homography matrix. A stitching canvas covering the reference image and the image to be stitched is created in the coordinate system of the reference image. Then, based on the overlap information between the reference image and the image to be stitched, and the stitching canvas, the target stitching canvas is determined. Image fusion is optimized based on the overlap information to avoid stitching artifacts and ensure the scene integrity and visual consistency of the stitching canvas. Using homography matrix, all corner points of the image to be stitched are mapped to the coordinate system of the reference image to eliminate distortion caused by camera viewpoint differences and unify them to the coordinate system of the reference image. In the coordinate system of the reference image, the coordinates of the same corner point of the image to be stitched and the reference image are averaged to obtain the target corner point coordinates. The target corner point coordinates are projected onto the target stitching canvas to obtain the stitched image. The stitched image has both full scene coverage and accurate corner point information, laying a reliable data foundation for subsequent viewpoint transformation and robot localization, thereby improving the accuracy of robot localization.

[0148] Based on the above embodiments, Figure 3 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 3 ,like Figure 3 As shown, step 14 may include the following steps:

[0149] Step 31: Obtain the coordinates of the map corner points in the stitched image.

[0150] In this step, the location of the target is accurately located from the fused stitched image using a visual marker recognition algorithm. Figure 4 Pre-defined visual positioning markers (i.e., map corner points) are used. Based on a pre-defined configuration dictionary, the unique code of each marker is matched to confirm the identity of the physical map corner point corresponding to each visual positioning marker (e.g., top left, top right, bottom left, bottom right). The pixel coordinates of each marker in the stitched image coordinate system are extracted to form a complete set of map corner point coordinates. If some corner points are not directly identified due to occlusion of the shooting view or detection errors, extrapolation can be performed based on the detected corner point coordinates and the aspect ratio constraints of the rectangular map to ensure the integrity of the map corner point coordinate set and provide accurate data for subsequent centroid calculation.

[0151] Step 32: Calculate the centroid of the corner point based on the coordinates of the corner point on the map.

[0152] In this step, the centroids of the corner points are calculated using the geometric centroid calculation method, based on the complete set of corner point coordinates.

[0153] The specific calculation method is as follows: First, take the arithmetic mean of the x-axis pixel coordinates of all map corner points to obtain the x-axis coordinate of the centroid. Then, take the arithmetic mean of the y-axis pixel coordinates of all map corner points to obtain the y-axis coordinate of the centroid. The final coordinate (c) is obtained. x , c y ) is the centroid of the corner point of the map.

[0154] The centroid of the corner point corresponds to the geometric center of the rectangular map region in the stitched image, providing a symmetrical reference point for subsequent sector division.

[0155] Step 33: Divide the stitched image into four image regions centered on the centroid of the corner point to obtain the reference image sector.

[0156] In this step, the centroid of the corner point (c) is used. x , c y Using the origin as the reference point, two perpendicular dividing lines are drawn in the stitched image. One line is parallel to the long side of the rectangular map, and the other is parallel to the short side. These two dividing lines evenly divide the corresponding map region in the stitched image into four independent image regions. Each region corresponds to a quadrant of the rectangular map (i.e., the upper left quadrant, upper right quadrant, lower left quadrant, and lower right quadrant). These four image regions are the base image sectors. Each sector corresponds one-to-one with a physical corner point of the map. This division establishes a connection between local image regions and global map corner points, providing a clear basis for subsequent corner point extrapolation and viewpoint transformation.

[0157] The robot localization method for multi-view image fusion provided in this application first obtains the coordinates of map corner points in the stitched image, then calculates the centroid of the corner points based on the coordinates of the map corner points, and finally divides the stitched image into four image regions with the centroid of the corner points as the center to obtain the reference image sector.

[0158] Based on the above embodiments, Figure 4 A flowchart illustrating the multi-view image fusion robot localization method provided in this application embodiment. Figure 4 ,like Figure 4 As shown, step 15 may include the following steps:

[0159] Step 41: Obtain the pixel coordinates and preset top view of the reference image sector.

[0160] The preset top view is determined based on the coordinates of the map corner points in the stitched image.

[0161] In this step, the four reference image sectors are traversed, and the pixel coordinates of all pixels in each sector are extracted in the stitched image coordinate system to form a complete set of sector pixel coordinates, ensuring that no pixels are missed. At the same time, a preset top view is constructed based on the determined map corner coordinates in the stitched image.

[0162] The preset top view is based on the actual physical size of the rectangular map, and the pixel scale of the top view is set according to the map. Figure 4 The coordinates of the corner point determine the boundary range of the top view, constructing a virtual top view frame with the same aspect ratio as the actual map and a top-down perspective. This preset top view provides a unified target coordinate system and carrier for subsequent pixel filling.

[0163] Step 42: Use bilinear interpolation algorithm to perform weighted averaging on the pixels around the target pixel to obtain the target weighted pixel.

[0164] The target pixel is any pixel in the sector of the reference image.

[0165] In this step, for each target pixel in the sector of the reference image, its coordinates in the stitched image coordinate system are first mapped to the preset top-view coordinate system using the inverse of the homography matrix, thus obtaining the corresponding coordinates of the target pixel in the top view. If the mapped coordinates are non-integer pixels, a bilinear interpolation algorithm is used to calculate the pixel value: taking the four integer pixels surrounding the non-integer coordinates as references, a weight is assigned based on the distance between each reference pixel and the target pixel (the closer the distance, the greater the weight), and the pixel values ​​of the four reference pixels are weighted and averaged to obtain the final weighted pixel value of the target pixel. This value can accurately fit the pixel information of the target pixel, avoiding pixel distortion or blank areas in the top view.

[0166] Step 43: Fill the target weighted pixels into the preset top view until all pixels of the stitched image are filled to obtain the top view corresponding to the target image.

[0167] In this step, each target weighted pixel is filled into its corresponding pixel position in the preset top-view coordinate system, one by one. Following a sector-by-sector, pixel-by-pixel filling order, pixels in all reference image sectors are processed sequentially to ensure that pixels in each stitched image are accurately mapped and filled into the top view. Once all pixels in the stitched image are filled, the preset top view will fully present the global scene of the rectangular map and the robot's position information, with a unified top-down perspective, ultimately forming the top view corresponding to the target image. This provides a clear and unified image reference for subsequent calculations of the robot's position and heading.

[0168] For example, Figure 5 This is a schematic diagram of robot localization using multi-view image fusion provided in an embodiment of this application, as shown below. Figure 5 As shown, the top left image is a picture of the robot taken by camera 1, the top right image is a picture of the robot taken by camera 2, and the middle image is an adjusted top view. The triangles in the image represent the robot, and A, B, C, and D are the map corner points of the target image.

[0169] The robot localization method based on multi-view image fusion provided in this application first obtains the pixel coordinates in a sector of a reference image and a preset top view. The preset top view is determined based on the coordinates of map corner points in the stitched image. Then, a bilinear interpolation algorithm is used to perform weighted averaging on the pixels surrounding the target pixel to obtain the target weighted pixel, ensuring the continuity and accuracy of pixel mapping and avoiding distortion of the top view. The target pixel is any pixel in a sector of the reference image. Finally, the target weighted pixel is filled into the preset top view until all pixels in the stitched image are filled, resulting in the top view corresponding to the target image. This method avoids view distortion in multi-view images, fully preserves the scene and corner information of the stitched image, and provides high-quality images for accurate robot localization.

[0170] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0171] Figure 6 This is a schematic diagram of the structure of a robot localization device for multi-view image fusion provided in an embodiment of this application. Figure 6 As shown, the device includes:

[0172] The acquisition module 61 is used to acquire at least one target image; each target image includes images of the robot from different perspectives, wherein the image from the preset perspective is the reference image, and the other images besides the reference image are images to be stitched together.

[0173] The calculation module 62 is used to calculate the homography matrix and the inverse matrix corresponding to each target image, respectively.

[0174] The processing module 63 is used to project the pixel coordinates in the image to be stitched onto the coordinate system of the reference image based on the homography matrix corresponding to each target image to obtain the stitched image.

[0175] The processing module 63 is also used to divide the stitched image into four image regions as reference image sectors based on the coordinates of map corner points in the stitched image;

[0176] The determination module 64 is used to map the pixel coordinates in each sector of the reference image to the top view coordinate system based on the inverse matrix, so as to obtain the top view corresponding to the target image.

[0177] The determination module 64 is also used to determine the position and heading of the robot based on the front-end coordinates of the robot and the centroids of its corners in the top view.

[0178] In one possible implementation, the processing module 63 is specifically used for:

[0179] Geometric transformations are performed on each target image based on the homography matrix, and a stitching canvas covering the reference image and the images to be stitched is created in the coordinate system of the reference image.

[0180] The target stitching canvas is determined based on the overlap information between the reference image and the image to be stitched, as well as the stitching canvas.

[0181] Using the homography matrix, all corner points of the image to be stitched are mapped to the coordinate system of the reference image;

[0182] In the coordinate system of the reference image, the coordinates of the same corner point of the image to be stitched and the reference image are averaged to obtain the coordinates of the target corner point;

[0183] Project the coordinates of the target corner points onto the target splicing canvas to obtain the spliced ​​image.

[0184] In one possible implementation, the determining module 64 is specifically used for:

[0185] A perspective transformation is performed on all images to be stitched using a homography matrix to generate an aligned image that is aligned with the viewpoint of the reference image.

[0186] The splicing canvas is filled based on the alignment image and overlap information until all pixels of the alignment image and the reference image are filled into the splicing canvas, thus obtaining the target splicing canvas.

[0187] In one possible implementation, the spliced ​​canvas is filled based on the aligned image and overlap information, and the determining module 64 is further configured to:

[0188] When the overlap information indicates the existence of an overlapping area, the pixels of the overlapping area between the aligned image and the reference image are mixed using a preset weight and filled into the splicing canvas.

[0189] When the overlap information indicates that there is no overlapping area, the pixels corresponding to the non-overlapping areas of the aligned image and the image to be stitched are respectively filled into the stitching canvas.

[0190] In one possible implementation, the processing module 63 is specifically used for:

[0191] Obtain the coordinates of map corner points in the stitched image;

[0192] The centroid of the corner point is calculated based on the coordinates of the corner point on the map.

[0193] The stitched image is divided into four image regions centered on the centroid of the corner point to obtain the reference image sector.

[0194] In one possible implementation, the determining module 64 is specifically used for:

[0195] Obtain the pixel coordinates and preset top view of the sector in the base image; the preset top view is determined based on the coordinates of map corner points in the stitched image;

[0196] The target pixel is obtained by weighting the pixels around it using a bilinear interpolation algorithm; the target pixel is any pixel in the sector of the reference image.

[0197] Fill the target weighted pixels into the preset top view until all pixels of the stitched image are filled, and obtain the top view corresponding to the target image.

[0198] In one possible implementation, the robot's front-end coordinates are determined based on a preset mapping relationship between corner points and front-end coordinates; the robot's corner centroids are determined based on the coordinates of each corner point of the robot.

[0199] The apparatus provided in this application embodiment can be used to execute the determination method in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0200] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0201] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device may include: a processor 71, a memory 72, and computer program instructions stored in the memory 72 and executable on the processor 71. When the processor 71 executes the computer program instructions, it implements the method provided in any of the foregoing embodiments.

[0202] Optionally, the various components of the electronic device can be connected via a system bus.

[0203] The memory 72 can be a separate memory unit or a memory unit integrated into the processor 71. The number of processors 71 can be one or more.

[0204] It should be understood that the processor 71 can be a Central Processing Unit (CPU), or other general-purpose processors 71, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor 71 can be a microprocessor 71, or any conventional processor 71. The steps of the method disclosed in this application can be directly manifested as being executed by the hardware processor 71, or being executed by a combination of hardware and software modules within the processor 71.

[0205] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Memory 72 may include random access memory (RAM) 72, and may also include non-volatile memory (NVM) 72, such as at least one disk storage device 72.

[0206] All or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory 72. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory 72 (storage medium) includes: read-only memory 72 (ROM), RAM, flash memory 72, hard disk, solid-state hard disk, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0207] The electronic device provided in this application embodiment can be used to execute the method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0208] This application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0209] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0210] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0211] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above-described method when executing the computer program.

[0212] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A robot localization method based on multi-view image fusion, characterized in that, include: Acquire at least one target image; each target image includes images of the robot from different perspectives, wherein the image from a preset perspective is the reference image, and the other images besides the reference image are images to be stitched together. Calculate the homography matrix and the inverse matrix corresponding to each of the target images; Based on the homography matrix corresponding to each target image, the pixel coordinates in the image to be stitched are projected onto the coordinate system of the reference image to obtain the stitched image; Based on the coordinates of the map corner points in the stitched image, the stitched image is divided into four image regions, which serve as reference image sectors. Based on the inverse matrix, the pixel coordinates in each of the reference image sectors are mapped to the top view coordinate system to obtain the top view corresponding to the target image; Based on the front-end coordinates of the robot and the centroids of its corners in the top view, the position and heading of the robot are determined.

2. The method according to claim 1, characterized in that, The step of projecting the pixel coordinates of the image to be stitched onto the coordinate system of the reference image based on the homography matrix corresponding to each of the target images to obtain the stitched image includes: Geometric transformations are performed on each of the target images based on the homography matrix, and a stitching canvas covering the reference image and the images to be stitched is created in the coordinate system of the reference image. The target splicing canvas is determined based on the overlap information between the reference image and the image to be spliced, as well as the splicing canvas. Using the homography matrix, all corner points of the image to be stitched are mapped to the coordinate system of the reference image; In the coordinate system of the reference image, the average coordinates of the same corner point of the image to be stitched and the reference image are calculated to obtain the target corner point coordinates; The coordinates of the target corner point are projected onto the target splicing canvas to obtain the spliced ​​image.

3. The method according to claim 2, characterized in that, The step of determining the target stitching canvas based on the overlap information between the reference image and the image to be stitched, and the stitching canvas, includes: A perspective transformation is performed on all images to be stitched using a homography matrix to generate an aligned image that is aligned with the viewpoint of the reference image. The splicing canvas is filled based on the alignment image and the overlap information until all pixels of the alignment image and the reference image are filled into the splicing canvas to obtain the target splicing canvas.

4. The method according to claim 3, characterized in that, The process of filling the spliced ​​canvas based on the aligned image and the overlap information includes: When the overlap information indicates the existence of an overlapping area, the pixels of the overlapping area between the aligned image and the reference image are mixed using a preset weight and filled into the splicing canvas; When the overlap information indicates that there is no overlapping area, the pixels corresponding to the non-overlapping areas of the aligned image and the image to be stitched are respectively filled into the stitching canvas.

5. The method according to claim 3, characterized in that, The step of dividing the stitched image into four image regions as reference image sectors based on the coordinates of map corner points in the stitched image includes: Obtain the coordinates of the map corner points in the stitched image; The centroid of the corner point is calculated based on the coordinates of the corner point on the map. With the centroid of the corner point as the center, the stitched image is divided into four image regions to obtain the reference image sector.

6. The method according to claim 5, characterized in that, The step of mapping the pixel coordinates in each of the reference image sectors to the top view coordinate system based on the inverse matrix to obtain the top view corresponding to the target image includes: Obtain the pixel coordinates and a preset top view in the sector of the reference image; the preset top view is determined based on the map corner coordinates in the stitched image; A weighted average of the pixels surrounding the target pixel is performed using a bilinear interpolation algorithm to obtain the target weighted pixel; the target pixel is any pixel in the sector of the reference image. The target weighted pixels are filled into the preset top view until all pixels of the stitched image are filled, thus obtaining the top view corresponding to the target image.

7. The method according to claim 1, characterized in that, The robot's front-end coordinates are determined based on a preset mapping relationship between corner points and front-end coordinates; the robot's corner centroids are determined based on the coordinates of each corner point of the robot.

8. A robot localization device for multi-view image fusion, characterized in that, include: An acquisition module is used to acquire at least one target image; each target image includes images of the robot from different perspectives, wherein the image from a preset perspective is a reference image, and the other images besides the reference image are images to be stitched together. The calculation module is used to calculate the homography matrix and the inverse matrix corresponding to each of the target images, respectively. The processing module is used to project the pixel coordinates in the image to be stitched onto the coordinate system of the reference image based on the homography matrix corresponding to each of the target images, so as to obtain the stitched image; The processing module is further configured to divide the stitched image into four image regions as reference image sectors based on the coordinates of map corner points in the stitched image. The determining module is used to map the pixel coordinates in each of the reference image sectors to the top view coordinate system based on the inverse matrix, so as to obtain the top view corresponding to the target image; The determining module is further configured to determine the position and heading of the robot based on the front-end coordinates and the centroids of its corner points in the top view.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images