Mapping relation determination method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a mapping relation determination method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: determining a first control coordinate marked in a visual image of a target scene and a second control coordinate marked in a map image; determining a first mapping matrix representing a mapping relationship between the visual image and the map image based on the first control coordinate and the second control coordinate; mapping each first control coordinate according to a first mapping matrix to obtain an intermediate pixel coordinate corresponding to each first control coordinate; determining a second mapping matrix representing a mapping relationship between the middle pixel coordinates and the pixel coordinates in the map image based on the middle pixel coordinates corresponding to the first control coordinates and the second control coordinates; and obtaining a nonlinear mapping relation between the visual image and the map image based on the first mapping matrix and the second mapping matrix. Therefore, the mapping relation between the visual image and the map image can be accurately established.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for determining a mapping relationship. Background Art

[0002] With the development of computer vision technology, the information that can be obtained from images captured by cameras (which can be called visual images) is becoming more and more abundant.

[0003] The visual image contains the environmental information of the actual scene collected by the camera. In addition, the map image of the actual scene can be obtained through the photoelectric sensor. In some actual scenes, a mapping relationship between the visual image and the map image can be established. For any object in the visual image, the pixel coordinates of the object in the visual image can be mapped according to the established mapping relationship to obtain the pixel coordinates of the object in the map image, and then, the route of the object in the map image can be determined.

[0004] However, how to accurately establish the mapping relationship between visual images and map images has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for determining a mapping relationship, so as to accurately establish a mapping relationship between a visual image and a map image. The specific technical solution is as follows:

[0006] In a first aspect of the present application, a method for determining a mapping relationship is provided, the method comprising:

[0007] Determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene;

[0008] Based on the current first control coordinates and the current second control coordinates, determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image as a current first mapping matrix;

[0009] Mapping the current first control coordinates respectively according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates;

[0010] Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, determining a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix;

[0011] Based on the current first mapping matrix and the current second mapping matrix, a nonlinear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image is obtained.

[0012] Optionally, the determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix includes:

[0013] Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined as the current second mapping matrix according to the thin plate spline interpolation algorithm.

[0014] Optionally, before determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, according to the thin plate spline interpolation algorithm, a nonlinear matrix representing a mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix, the method further includes:

[0015] For each current first control coordinate, based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate, a coordinate error value corresponding to the first control coordinate is calculated;

[0016] Based on the coordinate error value corresponding to the first control coordinate, determine the current error regularization term coefficient of the first control coordinate; wherein, when the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate;

[0017] The method of determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm as the current second mapping matrix, includes:

[0018] Based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates, as well as the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined according to the thin plate spline interpolation algorithm as the current second mapping matrix.

[0019] Optionally, the determining, based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates and the current second control coordinates, according to the thin plate spline interpolation algorithm, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix includes:

[0020] Based on the intermediate pixel coordinates corresponding to the current first control coordinates, the current first intermediate matrix is ​​determined according to the thin plate spline interpolation algorithm; wherein the current first intermediate matrix is ​​a symmetric matrix, and each row of the current first intermediate matrix corresponds to each current first control coordinate one by one; the elements in the current first intermediate matrix are calculated based on the following formula:

[0021]

[0022] U(r ij ) represents the element located at the i-th row and j-th column in the current first intermediate matrix, (x i ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the i-th row, (x j ,y j ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the jth column;

[0023] For each row in the current first intermediate matrix, the main diagonal elements in the row are replaced with the current error regularization term coefficients of the corresponding first control coordinates to obtain the current second intermediate matrix;

[0024] According to the thin plate spline interpolation algorithm, based on the current second intermediate matrix, the intermediate pixel coordinates corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is calculated as the current second mapping matrix.

[0025] Optionally, determining a current error regularization term coefficient of the first control coordinate based on the coordinate error value corresponding to the first control coordinate includes:

[0026] Calculate the ratio of the first threshold value to the coordinate error value corresponding to the first control coordinate;

[0027] The smaller value between the calculated ratio and the second threshold is determined as the current error regularization term coefficient of the first control coordinate; wherein the second threshold is greater than the first threshold, and both the first threshold and the second threshold are negatively correlated with the size of the map image.

[0028] Optionally, for each current first control coordinate, calculating the coordinate error value corresponding to the first control coordinate based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate includes:

[0029] For each current first control coordinate, convert the intermediate pixel coordinate corresponding to the first control coordinate into longitude and latitude coordinates to obtain first longitude and latitude coordinates, and convert the second control coordinate corresponding to the first control coordinate into longitude and latitude coordinates to obtain second longitude and latitude coordinates;

[0030] The distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates is calculated as the coordinate error value corresponding to the first control point.

[0031] Optionally, after determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix, the method further includes:

[0032] For each pixel coordinate in the visual image, mapping the pixel coordinate according to the current first mapping matrix and the second mapping matrix to obtain a mapping coordinate corresponding to the pixel coordinate;

[0033] Obtaining a current calibration image based on pixel values ​​corresponding to mapping coordinates corresponding to pixel coordinates in the visual image in the map image;

[0034] For a difference image area in the visual image that does not meet a preset matching condition with the current calibration image, the current first control coordinates are obtained based on the newly added first control coordinates in the difference image area, and the current second control coordinates are obtained based on the pixel coordinates corresponding to the newly added first control coordinates in the map image;

[0035] Return to the step of determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on the current first control coordinates and the current second control coordinates as the current first mapping matrix, until there is no difference image area in the visual image.

[0036] In a second aspect of the present application, a mapping relationship determination device is provided, the device comprising:

[0037] A control coordinate determination module, used to determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and to determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene;

[0038] A linear matrix determination module, used for determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on current first control coordinates and current second control coordinates, as a current first mapping matrix;

[0039] An intermediate pixel coordinate determination module is used to map the current first control coordinates according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates;

[0040] A nonlinear matrix determination module, for determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix;

[0041] The mapping relationship determination module is used to obtain a nonlinear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on a current first mapping matrix and a current second mapping matrix.

[0042] Optionally, the nonlinear matrix determination module includes:

[0043] The nonlinear matrix determination submodule is used to determine, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm, as the current second mapping matrix.

[0044] Optionally, the device further comprises:

[0045] A coordinate error value calculation module, used for determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm, as the current second mapping matrix, and for each current first control coordinate, calculating the coordinate error value corresponding to the first control coordinate based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate;

[0046] An error regularization term coefficient determination module is used to determine the current error regularization term coefficient of the first control coordinate based on the coordinate error value corresponding to the first control coordinate; wherein, when the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate;

[0047] The nonlinear matrix determination submodule includes:

[0048] A nonlinear matrix determination unit is used to determine, based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates, and the current second control coordinates, according to the thin plate spline interpolation algorithm, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix.

[0049] Optionally, the nonlinear matrix determination unit is specifically used to determine the current first intermediate matrix based on the intermediate pixel coordinates corresponding to the current first control coordinates according to the thin plate spline interpolation algorithm; wherein the current first intermediate matrix is ​​a symmetric matrix, and each row of the current first intermediate matrix corresponds to each current first control coordinate one by one; the elements in the current first intermediate matrix are calculated based on the following formula:

[0050]

[0051] U(r ij ) represents the element in the current first intermediate matrix located at the i-th row and j-th column, (x i ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the i-th row, (x j ,y j ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the jth column;

[0052] For each row in the current first intermediate matrix, the main diagonal elements in the row are replaced with the current error regularization term coefficients of the corresponding first control coordinates to obtain the current second intermediate matrix;

[0053] According to the thin plate spline interpolation algorithm, based on the current second intermediate matrix, the intermediate pixel coordinates corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is calculated as the current second mapping matrix.

[0054] Optionally, the error regularization term coefficient determination module is specifically used to calculate the ratio of the first threshold value to the coordinate error value corresponding to the first control coordinate;

[0055] The smaller value between the calculated ratio and the second threshold is determined as the current error regularization term coefficient of the first control coordinate; wherein the second threshold is greater than the first threshold, and both the first threshold and the second threshold are negatively correlated with the size of the map image.

[0056] Optionally, the coordinate error value calculation module is specifically used to convert the intermediate pixel coordinates corresponding to each current first control coordinate into longitude and latitude coordinates to obtain first longitude and latitude coordinates, and convert the second control coordinates corresponding to the first control coordinate into longitude and latitude coordinates to obtain second longitude and latitude coordinates; calculate the distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates as the coordinate error value corresponding to the first control point.

[0057] Optionally, the device further comprises:

[0058] A mapping coordinate determination module, for determining a nonlinear matrix representing a nonlinear mapping relationship between intermediate pixel coordinates and pixel coordinates in the map image based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates as the current second mapping matrix, and for each pixel coordinate in the visual image, mapping the pixel coordinate according to the current first mapping matrix and the second mapping matrix to obtain a mapping coordinate corresponding to the pixel coordinate;

[0059] A calibration image determination module, used to obtain a current calibration image based on the pixel values ​​corresponding to the mapping coordinates corresponding to the pixel coordinates in the visual image in the map image;

[0060] A control coordinate updating module is used to obtain the current first control coordinates based on the newly added first control coordinates in the difference image area that does not meet the preset matching conditions with the current calibration image, and obtain the current second control coordinates based on the pixel coordinates corresponding to the newly added first control coordinates in the map image; return to execute the step of determining a linear matrix representing the linear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image based on the current first control coordinates and the current second control coordinates as the current first mapping matrix, until there is no difference image area in the visual image.

[0061] According to a third aspect of the present application, there is also provided an electronic device, including:

[0062] Memory, used to store computer programs;

[0063] The processor is used to implement any of the above-mentioned mapping relationship determination methods when executing the program stored in the memory.

[0064] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned mapping relationship determination methods is implemented.

[0065] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the mapping relationship determination methods described above.

[0066] A mapping relationship determination method provided by an embodiment of the present application can determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond one-to-one to the current second control coordinates, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene; based on the current first control coordinates and the current second control coordinates, a linear matrix representing the linear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image is determined as the current first mapping matrix; the current first control coordinates are mapped respectively according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates; based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined as the current second mapping matrix; based on the current first mapping matrix and the current second mapping matrix, a nonlinear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image is obtained.

[0067] Based on the above processing, the pixel coordinates of the pixel points representing the same position in the visual image and the map image of the target scene can be determined as the first control coordinates and the second control coordinates. Then, according to the first control coordinates and the second control coordinates, a linear matrix (i.e., a first mapping matrix) that can represent the linear mapping relationship between the pixel coordinates in the visual image (which can be called visual pixel coordinates) and the pixel coordinates in the map image (which can be called map pixel coordinates) can be calculated. Since the target scene may be relatively complex, for example, there may be more slope roads and curved roads in the target scene, and the linear mapping relationship may not be able to effectively represent the mapping relationship between the visual pixel coordinates and the map pixel coordinates of the relatively complex target scene. Therefore, each first control coordinate can be mapped according to the determined first mapping matrix to obtain the intermediate pixel coordinates corresponding to each first control coordinate. Then, according to the intermediate pixel coordinates corresponding to each first control coordinate and each second control coordinate, a nonlinear matrix (i.e., a second mapping matrix) that can represent the nonlinear mapping relationship between the intermediate pixel coordinates and the map pixel coordinates can be calculated. Then, it is possible to combine the first mapping matrix and the second mapping matrix to obtain the mapping relationship between the visual pixel coordinates and the map pixel coordinates, that is, to obtain the mapping relationship between the visual image and the map image.

[0068] In this way, the mapping relationship between the visual image and the map image can be determined by combining the linear mapping and the nonlinear mapping. Accordingly, the mapping relationship between the visual image and the map image of a more complex target scene can be effectively represented, that is, the mapping relationship between the visual image and the map image corresponding to the target scene can be accurately established.

[0069] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0071] Figure 1 A first flow chart of a method for determining a mapping relationship provided in an embodiment of the present application;

[0072] Figure 2 A schematic diagram of a visual image provided by an embodiment of the present application;

[0073] Figure 3 A schematic diagram of a map image provided in an embodiment of the present application;

[0074] Figure 4 A second flow chart of the method for determining a mapping relationship provided in an embodiment of the present application;

[0075] Figure 5 A third flow chart of the mapping relationship determination method provided in an embodiment of the present application;

[0076] Figure 6 A fourth flow chart of the method for determining a mapping relationship provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of the structure of a mapping relationship determination device provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0080] With the development of computer vision technology, the information that can be obtained from images captured by cameras (which can be called visual images) is becoming more and more abundant.

[0081] The visual image contains the environmental information of the actual scene collected by the camera. In addition, the map image of the actual scene can be obtained through the photoelectric sensor. In some actual scenes, a mapping relationship between the visual image and the map image can be established. For any object in the visual image, the route of the object in the visual image (including multiple pixel coordinates) can be obtained, and then the pixel coordinates of the object in the visual image are mapped according to the established mapping relationship to obtain the pixel coordinates of the object in the map image, and then, the route of the object in the map image can be determined.

[0082] In order to accurately establish the mapping relationship between the visual image and the map image, the present application embodiment provides a method for determining the mapping relationship. Figure 1 , Figure 1 The first flowchart of the method for determining a mapping relationship provided in an embodiment of the present application may include the following steps:

[0083] Step S101: Determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates.

[0084] The current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene.

[0085] Step S102: Based on the current first control coordinates and the current second control coordinates, a linear matrix representing the linear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image is determined as the current first mapping matrix.

[0086] Step S103: mapping the current first control coordinates respectively according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates.

[0087] Step S104: Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined as the current second mapping matrix.

[0088] Step S105: Based on the current first mapping matrix and the current second mapping matrix, a nonlinear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image is obtained.

[0089] Based on the above processing, the pixel coordinates of the pixel points representing the same position in the visual image and the map image of the target scene can be determined as the first control coordinates and the second control coordinates. Then, according to the first control coordinates and the second control coordinates, a linear matrix (i.e., a first mapping matrix) that can represent the linear mapping relationship between the pixel coordinates in the visual image (which can be called visual pixel coordinates) and the pixel coordinates in the map image (which can be called map pixel coordinates) can be calculated. Since the target scene may be relatively complex, for example, there may be more slope roads and curved roads in the target scene, and the linear mapping relationship may not be able to effectively represent the mapping relationship between the visual pixel coordinates and the map pixel coordinates of the relatively complex target scene. Therefore, each first control coordinate can be mapped according to the determined first mapping matrix to obtain the intermediate pixel coordinates corresponding to each first control coordinate. Then, according to the intermediate pixel coordinates corresponding to each first control coordinate and each second control coordinate, a nonlinear matrix (i.e., a second mapping matrix) that can represent the nonlinear mapping relationship between the intermediate pixel coordinates and the map pixel coordinates can be calculated. Then, it is possible to combine the first mapping matrix and the second mapping matrix to obtain the mapping relationship between the visual pixel coordinates and the map pixel coordinates, that is, to obtain the mapping relationship between the visual image and the map image.

[0090] In this way, the mapping relationship between the visual image and the map image can be determined by combining the linear mapping and the nonlinear mapping. Accordingly, the mapping relationship between the visual image and the map image of a more complex target scene can be effectively represented, that is, the mapping relationship between the visual image and the map image corresponding to the target scene can be accurately established.

[0091] For step S101, the target scene is a scene for which the mapping relationship between the visual image and the map image needs to be determined. Both the visual image and the map image are two-dimensional images. For example, the target scene may be a traffic scene, such as a crossroads; or, the target scene may also be an indoor scene, such as an office or a shopping mall. The target scene may be photographed by an image acquisition device to obtain a visual image of the target scene. For example, the image acquisition device may be a camera set for the target scene. The map image of the target scene may represent: a projection image of the target scene on a plane perpendicular to the vertical direction. The target scene may be detected by an intelligent device for building a map carrying a photoelectric sensor to obtain a map image of the target scene, for example, the intelligent device may be a collection vehicle or a drone. Alternatively, an image of a high-precision map in a map navigation software may be directly obtained as a map image of the target scene. Visual images of the target scene collected from different angles may be obtained, and the multiple visual images may correspond to the same map image, that is, the multiple visual images may all be mapped to the same map image. For example, the target scene is an intersection, and the map image contains the complete scene of the intersection. Accordingly, cameras can be set in four directions of the intersection to collect visual images from four angles, and the visual images from the four angles all correspond to the map image of the intersection.

[0092] After obtaining the visual image and the map image of the target scene, a preset number of pixels can be selected from the visual image or the map image. Accordingly, in another image, the pixels representing the same position in the target scene as the selected preset number of pixels can be determined, and the determined pixels can be marked and the pixel coordinates of the determined pixels can be recorded. The pixels representing the same position in the target scene in the visual image and the map image are the same-named points. The pixel coordinates of the same-named points in the visual image and the map image can be recorded accordingly. The pixel points determined from the visual image can be called the first control point, or can also be called the camera visual image control point, and the visual pixel coordinates of the first control point are the first control coordinates; the pixel points determined from the map image can be called the second control point, or can also be called the map image control point, and the map pixel coordinates of the second control point are the second control coordinates. The preset number can be positively correlated with the size of the visual image or the map image, for example, the preset number can be any number between 35 and 45.

[0093] like Figure 2 As shown, Figure 2 A schematic diagram of a visual image provided in an embodiment of the present application. Figure 3 As shown, Figure 3 A schematic diagram of a map image provided in an embodiment of the present application. Figure 2 and Figure 3The target scenes are the same, which are all intersections. Figure 2 The road in the visual image shown is Figure 3 The right road in the map image shown represents the same road in the target scene, Figure 2 The pixel A1 in Figure 3 The pixel point A2 in is the same name point. Figure 2 The pixel B1 in Figure 3 The pixel point B2 in is the same name point.

[0094] Pixels with obvious features can be marked, and the marked pixels can be evenly distributed in the visual image and the map image, so that the marked points of the same name can effectively reflect the features of the target scene, that is, can effectively reflect the mapping relationship between the visual image and the map image of the target scene. For example, when the target scene is a traffic scene, the pixels corresponding to the vertices of the arrows indicating the lane lines, or the pixels at the corners of the lane lines can be selected for marking. In this way, the mapping relationship between the visual image and the map image of the target scene can be obtained based on the pixel coordinates of the marked pixels in the visual image and the map image of the target scene, thereby improving the accuracy of the obtained mapping relationship.

[0095] Furthermore, the marked visual image and map image can be obtained to determine the pixel coordinates of the marked pixel points in the visual image (ie, the first control coordinates) and the pixel coordinates of the marked pixel points in the map image of the target scene (ie, the second control coordinates).

[0096] For step S102, a linear matrix (i.e., a first mapping matrix) representing a linear mapping relationship between visual pixel coordinates and map pixel coordinates can be calculated based on each first control coordinate and each second control coordinate. The first mapping matrix can be a homography matrix, or can be other linear matrices, without specific limitation.

[0097] In one implementation, the linear mapping relationship between the visual pixel coordinates and the map pixel coordinates is shown in formula (1).

[0098]

[0099] Wherein, H is the first mapping matrix, for example, H can be a homography matrix, that is, The elements in H are all constants; (x, y) represents the map pixel coordinates, (u, v) represents the visual pixel coordinates; w = w ′ =1.

[0100] A specified number of first control coordinates are selected from a preset number of first control coordinates, and for each selected first control coordinate, the first control coordinate is substituted into formula (1) as (u, v), and the second control coordinate corresponding to the selected first control coordinate is substituted into formula (1) as (x, y), to obtain multiple equations about H. Then, H can be determined by combining the multiple equations about H, that is, the first mapping matrix is ​​obtained.

[0101] Since H has 8 degrees of freedom, H can be determined according to at least 4 first control coordinates and the corresponding second control coordinates. When the specified number is 4, in combination with 4 equations about H, H that satisfies the 4 equations can be solved. At this time, only when the selected 4 first control coordinates are mapped according to H, the mapping result obtained is the same as the second control coordinates corresponding to the 4 first control coordinates. When any other first control coordinates except the 4 first control coordinates are mapped according to H, the mapping result obtained may be different from the second control coordinates corresponding to the first control coordinate.

[0102] When the specified number is greater than 4, multiple equations about H are combined. At this time, the least square method can be used to determine the fitting solutions of the multiple equations about H as H, so that the sum of squares of the error values ​​between the mapping results obtained by mapping each first control coordinate according to the obtained H and each second control coordinate is minimized. Since the H determined at this time is an approximate value, when any first control coordinate is mapped according to H, the mapping result obtained may be different from the second control coordinate corresponding to the first control coordinate.

[0103] Therefore, when any first control coordinate is subsequently mapped according to the first mapping matrix, the intermediate pixel coordinate corresponding to the first control coordinate may be different from the corresponding second control coordinate.

[0104] For step S103 and step S104, each first control coordinate can be mapped according to the determined first mapping matrix to obtain the intermediate pixel coordinates corresponding to each first control coordinate. Since the intermediate pixel coordinates corresponding to each first control coordinate may be different from the corresponding second control coordinates, that is, the mapping relationship between the visual image and the map image cannot be effectively represented by the first mapping matrix, and the mapping relationship between the visual image and the map image is not necessarily linear. For example, when there is a slope in the target scene, the mapping relationship between the visual image and the map image is nonlinear. Therefore, the intermediate pixel coordinates corresponding to each first control coordinate can be further mapped according to the nonlinear matrix to effectively represent the nonlinear mapping relationship between the visual image and the map image, so as to obtain a more accurate mapping relationship between the visual image and the map image. According to the intermediate pixel coordinates corresponding to each first control coordinate and each second control coordinate, a nonlinear matrix (i.e., a second mapping matrix) that can represent the mapping relationship between the intermediate pixel coordinates and the map pixel coordinates can be calculated. Those skilled in the art can select different algorithms to calculate the second mapping matrix according to different target scenes. For example, when the target scene is a traffic scene, the second mapping matrix can be determined based on a thin plate splines (TPS) interpolation algorithm; or, the second mapping matrix can be determined based on polynomial transformation, Bezier surface, B-spline (Basis splines) interpolation method and other algorithms.

[0105] In one embodiment, see Figure 4 , Figure 4 The second flowchart of the mapping relationship determination method provided in the embodiment of the present application, step S104 includes:

[0106] Step S1041: Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined according to the thin plate spline interpolation algorithm as the current second mapping matrix.

[0107] With respect to step S1041, in one implementation, the coordinate error value between the intermediate pixel coordinates corresponding to each first control coordinate and the corresponding second control coordinate may be ignored, and the second mapping matrix may be directly calculated according to the thin plate spline interpolation algorithm. In this way, there is no need to calculate the coordinate error value between the intermediate pixel coordinates corresponding to each first control coordinate and the corresponding second control coordinate, and a nonlinear second mapping matrix can be determined to effectively represent the nonlinear mapping relationship between the visual image and the map image, thereby further improving the accuracy of the established mapping relationship between the visual image and the map image.

[0108] According to the thin plate spline interpolation algorithm, the mapping relationship between the intermediate pixel coordinates and the map pixel coordinates is shown in formula (2).

[0109]

[0110] Among them, f(x,y) represents the horizontal or vertical coordinate of the map pixel coordinate; (x,y) represents the middle pixel coordinate; a1, a x 、a y and w i are constant coefficients that need to be solved, and for the horizontal and vertical coordinates of the map pixel coordinates, a1, a x 、a y and w i Need to be solved separately; n represents the number of the current first control coordinates, that is, the preset number in the above embodiment; U(·) is a nonlinear function, as shown in formula (3):

[0111]

[0112] Where s = |P i -(x,y)|,P i represents the middle pixel coordinate corresponding to the i-th first control coordinate, (x, y) represents the middle pixel coordinate corresponding to any first control coordinate, and s represents P i The coordinate distance from (x, y).

[0113] In order to determine the mapping relationship between the intermediate pixel coordinates and the map pixel coordinates, it is necessary to solve the constant coefficients in formula (2). The horizontal and vertical coordinates of the map pixel coordinates have corresponding constant coefficients, which need to be solved separately. The process of solving the constant coefficient corresponding to the horizontal coordinate of the map pixel coordinate is used as an example to illustrate:

[0114] For the horizontal coordinates of the map pixel coordinates, we need to solve a1, a x 、a y and n w i , that is, n+3 parameters. For each first control coordinate, the middle pixel coordinate corresponding to the first control coordinate is taken as P i , and substitute the horizontal coordinate of the second control coordinate corresponding to the first control coordinate as f(x, y) into formula (2), and obtain n equations, and combine the three constraints shown in formulas (4), (5) and (6) to obtain n+3 equations to solve the above n+3 parameters.

[0115]

[0116]

[0117]

[0118] Among them, (x i ,y i ) represents the middle pixel coordinate corresponding to the i-th first control coordinate; n represents the number of the current first control coordinates, that is, the preset number in the above embodiment.

[0119] The obtained n+3 equations are expressed in matrix form, as shown in formula (7):

[0120]

[0121] in, The second mapping matrix corresponding to the horizontal coordinate is represented by L as shown in formula (8):

[0122]

[0123] Where K is a symmetric matrix, as shown in formula (9). And the main diagonal elements in K are all 0, that is, U(r 11 )、U(r 22 ),...,U(r nn ) is 0.

[0124]

[0125] P is as shown in formula (10), P T Represents the transposed matrix of P:

[0126]

[0127] O is a 3*3 zero matrix.

[0128] Similarly, the n+3 parameters corresponding to the ordinate can also be solved and the second mapping matrix corresponding to the ordinate can be obtained.

[0129] Correspondingly, the second mapping matrix includes: a second mapping matrix corresponding to the horizontal coordinate and a second mapping matrix corresponding to the vertical coordinate. Furthermore, for each visual pixel coordinate, the horizontal coordinate of the middle pixel coordinate corresponding to the visual pixel coordinate can be mapped according to the second mapping matrix corresponding to the horizontal coordinate to obtain the horizontal coordinate of the map pixel coordinate corresponding to the visual pixel coordinate; the vertical coordinate of the middle pixel coordinate corresponding to the visual pixel coordinate can be mapped according to the second mapping matrix corresponding to the vertical coordinate to obtain the vertical coordinate of the map pixel coordinate corresponding to the visual pixel coordinate.

[0130] With respect to step S1041, in another implementation, the coordinate error values ​​corresponding to the first control coordinates may be calculated, and the second mapping matrix may be calculated in combination with the coordinate error values. Figure 4 Based on Figure 5 , Figure 5 This is a third flow chart of the method for determining a mapping relationship provided in an embodiment of the present application. Before step S1041, the method further includes:

[0131] Step S106: for each current first control coordinate, based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate, calculate the coordinate error value corresponding to the first control coordinate.

[0132] Step S107: determining the current error regularization term coefficient of the first control coordinate based on the coordinate error value corresponding to the first control coordinate.

[0133] When the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate.

[0134] Step S1041 includes:

[0135] Step S10411: Based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined according to the thin plate spline interpolation algorithm as the current second mapping matrix.

[0136] In the embodiment of the present application, since each first control coordinate is mapped according to the determined first mapping matrix, there is an error between the intermediate pixel coordinate corresponding to each first control coordinate and the second control coordinate corresponding to each first control coordinate. Therefore, for each current first control point, the coordinate error value corresponding to the first control coordinate can be calculated based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate.

[0137] In one implementation, the distance between the middle pixel coordinates of the first control point and the map pixel coordinates of the corresponding second control point may be calculated as the coordinate error value corresponding to the first control point.

[0138] In another implementation, step S105 includes: for each current first control point, converting the intermediate pixel coordinates of the first control point into longitude and latitude coordinates to obtain first longitude and latitude coordinates, and converting the map pixel coordinates of the second control point corresponding to the first control point into longitude and latitude coordinates to obtain second longitude and latitude coordinates; calculating the distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates as the coordinate error value corresponding to the first control point.

[0139] In an embodiment of the present application, the longitude and latitude coordinates of an object can reflect the actual geographical location of the object in the target scene. The linear mapping relationship between the map pixel coordinates and the longitude and latitude coordinates can be predetermined. For each first control coordinate, the intermediate pixel coordinates corresponding to the first control coordinate can be mapped according to the linear mapping relationship to obtain the first longitude and latitude coordinates. And the second control coordinate corresponding to the first control coordinate can be mapped according to the linear mapping relationship to obtain the second longitude and latitude coordinates. Furthermore, the distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates can be calculated as the coordinate error value corresponding to the first control point. Since the longitude and latitude coordinates can reflect the actual geographical location in the target scene, the distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates can also intuitively reflect the coordinate error value corresponding to the first control coordinate.

[0140] For each first control coordinate, the current error regularization term coefficient of the first control coordinate can be determined according to the coordinate error value corresponding to the first control coordinate. When the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate. When the coordinate error value corresponding to any first control coordinate is not less than the preset threshold, the current error regularization term coefficient of the first control coordinate can be the preset threshold. For example, the preset threshold can be 0.2 or 0.25.

[0141] If the error regularization term coefficient is not used when calculating the second mapping matrix according to the intermediate pixel coordinates corresponding to each first control coordinate and each second control coordinate according to the thin plate spline interpolation algorithm, then for each first control coordinate, according to the second mapping matrix, the intermediate pixel coordinate corresponding to the first control coordinate will be forced to be mapped to the second control coordinate corresponding to the first control coordinate. That is, when calculating the second mapping matrix, for each first control coordinate, the error between the corresponding mapping result and the second control coordinate corresponding to the first control coordinate is strictly 0. For more complex target scenes, for example, there are slopes and curved surfaces in the target scene, if the intermediate pixel coordinate corresponding to the first control coordinate is forced to be mapped to the second control coordinate corresponding to the first control coordinate, it may cause local distortion of the mapping result obtained by mapping according to the determined mapping relationship.

[0142] In order to alleviate local distortion, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the map pixel coordinates can be determined based on the intermediate pixel coordinates and the error regularization term coefficients corresponding to the current first control coordinates, as well as the current second control coordinates, as the current second mapping matrix. In this way, the error regularization term coefficients of the first control point can be determined in combination with the coordinate error value. Accordingly, when determining the second mapping matrix, the mapping results of the intermediate pixel coordinates corresponding to the first control coordinates can be adjusted according to the error regularization term coefficients to obtain the second control coordinates corresponding to the first control coordinates, which can improve the robustness and smoothness of the mapping results obtained by mapping according to the determined mapping relationship, and further ensure that the mapping relationship between the visual image and the map image can be accurately determined.

[0143] In one implementation, step S107 includes: calculating the ratio of a first threshold value to the coordinate error value corresponding to the first control coordinate; determining the smaller value between the calculated ratio and the second threshold value as the current error regularization term coefficient of the first control coordinate; wherein the second threshold value is greater than the first threshold value, and both the first threshold value and the second threshold value are negatively correlated with the size of the map image.

[0144] In an embodiment of the present application, when the calculated ratio is equal to the second threshold, any one of the calculated ratio and the second threshold can be determined as the error regularization term coefficient of the first control point. The ratio of the first threshold and the second threshold can be the preset threshold in the above embodiment. The first threshold can be: the ratio of the first preset parameter to the parameter representing the size of the map image. The second threshold can be: the ratio of the second preset parameter to the parameter representing the size of the map image. The parameter representing the size of the map image can be the smaller value of the length and width of the map image. The first preset parameter is greater than the second preset parameter. For example, the first preset parameter can be 50 and the second preset parameter can be 10. As shown in formula (11):

[0145] λ=min(λ0 / d,λ max ) (11)

[0146] Wherein, λ represents the current error regularization term coefficient of the first control coordinate, λ0 represents the first threshold, d represents the coordinate error value corresponding to the first control coordinate, λ max Indicates the second threshold.

[0147] Based on the above processing, the current error regularization term coefficient of the first control coordinate can be determined in combination with the coordinate error value and the size of the map image. Accordingly, when the second mapping matrix is ​​subsequently determined according to the error regularization term coefficient, the second mapping matrix can be further reduced in the coordinate error value and adapted to the size of the map image, thereby ensuring that the second mapping matrix can be accurately determined, and further ensuring that the mapping relationship between the visual image and the map image can be accurately determined.

[0148] In one embodiment, step S10411 includes:

[0149] Step 1: Based on the intermediate pixel coordinates corresponding to the current first control coordinates, determine the current first intermediate matrix according to the thin plate spline interpolation algorithm.

[0150] Among them, the current first intermediate matrix is ​​a symmetric matrix, and each row of the current first intermediate matrix corresponds to each current first control coordinate one by one, and correspondingly, each column of the current first intermediate matrix also corresponds to each current first control coordinate one by one; the elements in the current first intermediate matrix are calculated based on formula (12):

[0151]

[0152] U(r ij ) represents the element in the current first intermediate matrix located at the i-th row and j-th column, (x i ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the i-th row, (x j ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the jth column.

[0153] Step 2: For each row in the current first intermediate matrix, replace the main diagonal elements in the row with the current error regularization term coefficients of the corresponding first control coordinates to obtain the current second intermediate matrix.

[0154] Step 3: According to the thin plate spline interpolation algorithm, based on the current second intermediate matrix, the intermediate pixel coordinates corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is calculated as the current second mapping matrix.

[0155] In the embodiment of the present application, the first intermediate matrix is ​​the matrix K, U (r ij ) is U(|P i -(x,y)|),(x i ,y i ) is P in the above embodimenti ,(x j ,y j ) is (x, y) in the above embodiment.

[0156] Since the error regularization term is not considered when calculating the second mapping matrix, the intermediate pixel coordinate corresponding to a first control coordinate will be forced to be mapped to the second control coordinate corresponding to the first control coordinate, which may cause the mapping result obtained by mapping according to the determined mapping relationship to be distorted. For example, for a road with a large slope, selecting inappropriate points of the same name and forcing the mapping calculation of the second mapping matrix may cause the final mapping result of the visual image to the map image to be significantly distorted.

[0157] In order to avoid the above situation, for each row in the current first intermediate matrix, the main diagonal elements in the row can be replaced with the current error regularization term coefficients of the corresponding first control point to obtain the current second intermediate matrix. That is, the main diagonal elements in the matrix K can be replaced with the error regularization term coefficients, and the second intermediate matrix is ​​shown in formula (13):

[0158]

[0159] Where K′ represents the current second intermediate matrix, λ1, λ2, ..., λ n Represents the current error regularization term coefficient of the first control coordinate corresponding to each row in the second intermediate matrix.

[0160] Ultimately, it is necessary to determine the mapping relationship between the visual image and the map image, that is, it is necessary to make the pixel coordinates representing the same position in the target scene in the visual image and the map image correspond one to one. However, since there are slopes and curved surfaces in the target scene, and for the local areas with slopes and curved surfaces, there may be large differences between the parts of the visual image and the map image corresponding to the local areas. Therefore, there may be local areas with large differences between the visual image and the map image. The number of first control coordinates and second control coordinates is limited. For local areas with large differences, only a few first control coordinates and second control coordinates may be used for forced alignment, which will cause distortion of local areas in the final mapping result of the visual image to the map image. Therefore, in the process of mapping the intermediate pixel coordinates corresponding to a first control coordinate to the second control coordinate corresponding to the first control coordinate, the mapping result can be compensated by the error regularization term coefficient. In this way, by combining the error regularization term coefficient and calculating the second mapping matrix, it is possible to avoid the forced alignment of local areas with large differences in the visual image and the map image through a small number of first control coordinates and second control coordinates, which leads to distortion of local areas in the final mapping result of the visual image to the map image. This reduces the distortion of local areas and improves the robustness of the obtained second mapping matrix, further improving the robustness of the mapping relationship between the final visual image and the map image.

[0161] In one embodiment, after step S103, the method further includes:

[0162] Step 1: For each pixel coordinate in the visual image, the pixel coordinate is mapped according to the current first mapping matrix and the second mapping matrix to obtain the mapping coordinate corresponding to the pixel coordinate.

[0163] Step 2: Based on the pixel values ​​corresponding to the mapping coordinates of each pixel coordinate in the visual image in the map image, the current calibration image is obtained.

[0164] Step three: For the difference image area in the visual image that does not meet the preset matching conditions with the current calibration image, obtain the current first control coordinates based on the newly added first control coordinates in the difference image area, and obtain the current second control coordinates based on the pixel coordinates corresponding to the newly added first control coordinates in the map image; return to execute step S102 until there is no difference image area in the visual image.

[0165] In an embodiment of the present application, after determining the first mapping matrix and the second mapping matrix, for each pixel point (which can be called a visual pixel point) in the visual image, the visual pixel coordinates of the visual pixel point can be mapped according to the first mapping matrix to obtain the intermediate pixel coordinates of the visual pixel point, and the intermediate pixel coordinates of the visual pixel point can be mapped according to the second mapping matrix to obtain the mapping coordinates of the visual pixel point.

[0166] In a map image, different objects may be set with different pixel values. Therefore, the pixel value of the pixel point at the mapping coordinates of each visual pixel point determined can be obtained from the map image as the target pixel value corresponding to the visual pixel point, and then, the calibration image can be obtained according to the pixel coordinates of each visual pixel point in the visual image and the corresponding target pixel value. In other words, the calibration image is consistent with the size of the visual image, and the calibration image can reflect the position of the object in the image obtained by mapping the visual image. Then, the calibration image and the visual image can be compared. For example, the difference image area in the visual image that does not meet the preset matching condition with the calibration image can be determined by manual detection. Alternatively, the visual image and the calibration image can be edge detected, and accordingly, the edges of each object in the two images can be determined. Then, the matching degree of the edges of the objects determined at the corresponding positions in the two images can be compared. If there is an object whose matching degree is lower than the specified matching degree threshold, the area where the object is located is the difference image area that does not meet the preset matching condition.

[0167] If the calculated nonlinear mapping relationship between the visual pixel coordinates and the map pixel coordinates has a high accuracy, the calibration image and the visual image are highly consistent. When there is no difference image area in the visual image, the mapping relationship between the visual image and the map image can be obtained based on the latest first mapping matrix and the second mapping matrix.

[0168] On the contrary, if there is an image area in the visual image that is inconsistent with the calibration image and has a large deviation (i.e., a difference image area), it means that the accuracy of the calculated nonlinear mapping relationship is not high. Therefore, for each difference image area, a new first control point can be added to the difference image area, and the second control point corresponding to the new first control point can be marked in the map image. Then, the latest first control coordinates and the corresponding second control coordinates can be determined, and the first mapping matrix and the second mapping matrix can be determined according to the above steps S102-S105 based on the latest first control coordinates and the corresponding second control coordinates.

[0169] Based on the above processing, when an image area with a large error is determined, a new first control point can be added to the image area to optimize and adjust the first mapping matrix and the second mapping matrix, thereby reducing the error in the mapping relationship between the obtained visual image and the map image, further ensuring that the mapping relationship between the visual image and the map image can be accurately established, and improving the robustness of the control point selection. For areas with poor calibration effects, there is no need to screen the existing control points, only to add new control points in the area, which can improve the efficiency of establishing the mapping relationship between the visual image and the map image.

[0170] For step S105, based on the current first mapping matrix and the current second mapping matrix, a mapping relationship between the visual image and the map image is obtained. That is, the pixel coordinates in the visual image can be mapped by the first mapping matrix to obtain a mapping result, and the obtained mapping result can be further mapped by the second mapping matrix to obtain the corresponding pixel coordinates in the map image.

[0171] For more complex target scenes, the mapping relationship determination method provided in the embodiment of the present application does not need to divide the target scene into regions, and can obtain the mapping relationship between the visual image and the map image by directly mapping the linear matrix and the nonlinear matrix cascade. In this way, it is possible to ensure that the mapping result determined according to the mapping relationship is smooth, and avoid the situation where the multi-plane connection part jumps.

[0172] In addition, based on formulas (1)-(13) in the above embodiments, it can be seen that in the mapping relationship determination method provided in the embodiment of the present application, the mapping relationship between the visual image and the map image can be determined based on the first control coordinate and the second control coordinate, without the need to calibrate the internal and external parameters of the image acquisition device. The mapping relationship between the visual image and the map image can be directly digitally simulated, which can improve the efficiency of mapping relationship determination and the applicability of the mapping relationship determination method provided in the embodiment of the present application.

[0173] Furthermore, according to the linear relationship between the map image and the longitude and latitude map, each pixel in the map image can be mapped to obtain the longitude and latitude coordinates of the pixel. In this way, it is possible to solve the spatial projection relationship between the image space of the visual image and the BEV (Bird's Eye View) space of the longitude and latitude map. The determination of the mapping relationship between the visual image collected by the camera and the longitude and latitude map can be called VMAP (visual image-map image) calibration.

[0174] In one embodiment, see Figure 6 , Figure 6 A fourth flow chart of a method for determining a mapping relationship provided in an embodiment of the present application, the method comprising:

[0175] Step S601: calibrate control points, that is, step S101 in the above embodiment.

[0176] Step S602: Solve the homography matrix. That is, step S102 in the above embodiment. The homography matrix is ​​the first mapping matrix in the above embodiment. The process of solving the homography matrix can be called rough calibration.

[0177] Step S603: Calculate the linear error, that is, step S106 in the above embodiment. The linear error is the coordinate error value corresponding to the first control point in the above embodiment.

[0178] Step S604: Calculate the linear mapping result of the visual image. Map the visual image according to the calculated homography matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates, that is, step S103 in the above embodiment.

[0179] Step S605: Solving the nonlinear transformation matrix. Calculate the nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the map pixel coordinates, that is, step S1041 in the above embodiment. The process of solving the nonlinear transformation matrix can be called fine calibration.

[0180] Step S606: Visualize the calibration result. Determine the calibration image, that is, step 1 and step 2 in the above embodiment.

[0181] Step S607: Determine whether there is an area with a large error. If there is an area with a large error, the area is the difference image area that does not meet the preset matching conditions in the above embodiment, and then execute step S608; if there is no area with a large error, execute step S609.

[0182] Step S608: Determine the latest first control coordinates and the corresponding second control coordinates, and return to execute step S602. That is, for each difference image area, after adding a new first control point in the difference image area and marking the second control point corresponding to the new first control point in the map image, the latest first control coordinates and the corresponding second control coordinates can be determined. And according to the latest first control coordinates and the corresponding second control coordinates, calculate the first mapping matrix. That is, step three in the above embodiment.

[0183] Step S609: Based on the current first mapping matrix and the current second mapping matrix, a mapping relationship between the visual image and the map image is obtained.

[0184] In an embodiment of the present application, a linear matrix and a nonlinear matrix can be cascaded to determine the mapping relationship between the visual image and the map image, and the rough calibration result mapped according to the linear matrix can be further mapped according to the nonlinear matrix to obtain a precise calibration result. And the regularization term of the nonlinear matrix can be determined according to the coordinate error value between the rough calibration result and the second control coordinate, so that it is possible to avoid the local areas with large differences in the visual image and the map image being forced to align through a small number of first control coordinates and second control coordinates, resulting in distortion of the local areas in the final mapping result of the visual image to the map image, thereby reducing the distortion of the local areas. In this way, compared with the method of determining the mapping relationship between the visual image and the map image only through a linear homography matrix, a more accurate mapping relationship between the visual image and the map image can be obtained.

[0185] Based on the same inventive concept, the present application also provides a mapping relationship determination device, see Figure 7 , Figure 7 A schematic diagram of a mapping relationship determination device provided in an embodiment of the present application. The device comprises:

[0186] The control coordinate determination module 701 is used to determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and to determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene;

[0187] A linear matrix determination module 702 is used to determine, based on each current first control coordinate and each current second control coordinate, a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image as a current first mapping matrix;

[0188] The intermediate pixel coordinate determination module 703 is used to map the current first control coordinates according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates;

[0189] A nonlinear matrix determination module 704 is used to determine, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix;

[0190] The mapping relationship determination module 705 is used to obtain a nonlinear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on the current first mapping matrix and the current second mapping matrix.

[0191] In one embodiment, the nonlinear matrix determination module 704 includes:

[0192] The nonlinear matrix determination submodule is used to determine, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm, as the current second mapping matrix.

[0193] In one embodiment, the apparatus further comprises:

[0194] A coordinate error value calculation module, used for determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm, as the current second mapping matrix, and for each current first control coordinate, calculating the coordinate error value corresponding to the first control coordinate based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate;

[0195] An error regularization term coefficient determination module is used to determine the current error regularization term coefficient of the first control coordinate based on the coordinate error value corresponding to the first control coordinate; wherein, when the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate;

[0196] The nonlinear matrix determination submodule includes:

[0197] A nonlinear matrix determination unit is used to determine, based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates, and the current second control coordinates, according to the thin plate spline interpolation algorithm, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix.

[0198] In one embodiment, the nonlinear matrix determination unit is specifically used to determine the current first intermediate matrix based on the intermediate pixel coordinates corresponding to the current first control coordinates according to the thin plate spline interpolation algorithm; wherein the current first intermediate matrix is ​​a symmetric matrix, and each row of the current first intermediate matrix corresponds to each current first control coordinate one by one; the elements in the current first intermediate matrix are calculated based on the following formula:

[0199]

[0200] U(r ij ) represents the element located at the i-th row and j-th column in the current first intermediate matrix, (x i ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the i-th row, (x j ,y j ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the jth column;

[0201] For each row in the current first intermediate matrix, the main diagonal elements in the row are replaced with the current error regularization term coefficients of the corresponding first control coordinates to obtain the current second intermediate matrix;

[0202] According to the thin plate spline interpolation algorithm, based on the current second intermediate matrix, the intermediate pixel coordinates corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is calculated as the current second mapping matrix.

[0203] In one embodiment, the error regularization term coefficient determination module is specifically used to calculate the ratio of the first threshold value to the coordinate error value corresponding to the first control coordinate;

[0204] The smaller value between the calculated ratio and the second threshold is determined as the current error regularization term coefficient of the first control coordinate; wherein the second threshold is greater than the first threshold, and both the first threshold and the second threshold are negatively correlated with the size of the map image.

[0205] In one embodiment, the coordinate error value calculation module is specifically used to convert the intermediate pixel coordinates corresponding to each current first control coordinate into longitude and latitude coordinates to obtain first longitude and latitude coordinates, and convert the second control coordinates corresponding to the first control coordinate into longitude and latitude coordinates to obtain second longitude and latitude coordinates; calculate the distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates as the coordinate error value corresponding to the first control point.

[0206] In one embodiment, the apparatus further comprises:

[0207] A mapping coordinate determination module, for determining a nonlinear matrix representing a nonlinear mapping relationship between intermediate pixel coordinates and pixel coordinates in the map image based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates as the current second mapping matrix, and for each pixel coordinate in the visual image, mapping the pixel coordinate according to the current first mapping matrix and the second mapping matrix to obtain a mapping coordinate corresponding to the pixel coordinate;

[0208] A calibration image determination module, used to obtain a current calibration image based on the pixel values ​​corresponding to the mapping coordinates corresponding to the pixel coordinates in the visual image in the map image;

[0209] A control coordinate updating module is used to obtain the current first control coordinates based on the newly added first control coordinates in the difference image area that does not meet the preset matching conditions with the current calibration image, and obtain the current second control coordinates based on the pixel coordinates corresponding to the newly added first control coordinates in the map image; return to execute the step of determining a linear matrix representing the linear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image based on the current first control coordinates and the current second control coordinates as the current first mapping matrix, until there is no difference image area in the visual image.

[0210] The present application also provides an electronic device, such as Figure 8 As shown, including:

[0211] Memory 801, used for storing computer programs;

[0212] The processor 802 is used to execute the program stored in the memory 801, and implements the following steps:

[0213] Determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene;

[0214] Based on the current first control coordinates and the current second control coordinates, determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image as a current first mapping matrix;

[0215] Mapping the current first control coordinates respectively according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates;

[0216] Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, determining a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix;

[0217] Based on the current first mapping matrix and the current second mapping matrix, a nonlinear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image is obtained.

[0218] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.

[0219] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0220] The communication interface is used for communication between the above electronic device and other devices.

[0221] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0222] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0223] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned mapping relationship determination methods are implemented.

[0224] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any mapping relationship determination method in the above embodiments.

[0225] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0226] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0227] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0228] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for determining a mapping relationship, characterized in that: The method comprises: Determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene; Based on the current first control coordinates and the current second control coordinates, determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image as a current first mapping matrix; Mapping the current first control coordinates respectively according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates; Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, determining a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix; Based on the current first mapping matrix and the current second mapping matrix, a nonlinear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image is obtained.

2. The method according to claim 1, characterized in that The method of determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix, comprises: Based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined as the current second mapping matrix according to the thin plate spline interpolation algorithm.

3. The method according to claim 2, characterized in that Before determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to a thin plate spline interpolation algorithm as the current second mapping matrix, the method further includes: For each current first control coordinate, based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate, a coordinate error value corresponding to the first control coordinate is calculated; Based on the coordinate error value corresponding to the first control coordinate, determine the current error regularization term coefficient of the first control coordinate; wherein, when the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate; The method of determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm as the current second mapping matrix, includes: Based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates, as well as the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is determined according to the thin plate spline interpolation algorithm as the current second mapping matrix.

4. The method according to claim 3, characterized in that The method comprises: determining a nonlinear matrix representing a mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to a thin plate spline interpolation algorithm based on the intermediate pixel coordinates and the error regularization term coefficients corresponding to the current first control coordinates and the current second control coordinates, as the current second mapping matrix, including: Based on the intermediate pixel coordinates corresponding to the current first control coordinates, the current first intermediate matrix is ​​determined according to the thin plate spline interpolation algorithm; wherein the current first intermediate matrix is ​​a symmetric matrix, and each row of the current first intermediate matrix corresponds to each current first control coordinate one by one; the elements in the current first intermediate matrix are calculated based on the following formula: U(r ij ) represents the element located at the i-th row and j-th column in the current first intermediate matrix, (x i ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the i-th row, (x j ,y j ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the jth column; For each row in the current first intermediate matrix, the main diagonal elements in the row are replaced with the current error regularization term coefficients of the corresponding first control coordinates to obtain the current second intermediate matrix; According to the thin plate spline interpolation algorithm, based on the current second intermediate matrix, the intermediate pixel coordinates corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is calculated as the current second mapping matrix.

5. The method according to claim 3, characterized in that: The step of determining the current error regularization term coefficient of the first control coordinate based on the coordinate error value corresponding to the first control coordinate includes: Calculate the ratio of the first threshold value to the coordinate error value corresponding to the first control coordinate; The smaller value between the calculated ratio and the second threshold is determined as the current error regularization term coefficient of the first control coordinate; wherein the second threshold is greater than the first threshold, and both the first threshold and the second threshold are negatively correlated with the size of the map image.

6. The method according to claim 3, characterized in that The step of calculating, for each current first control coordinate, a coordinate error value corresponding to the first control coordinate based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate, includes: For each current first control coordinate, convert the intermediate pixel coordinate corresponding to the first control coordinate into longitude and latitude coordinates to obtain first longitude and latitude coordinates, and convert the second control coordinate corresponding to the first control coordinate into longitude and latitude coordinates to obtain second longitude and latitude coordinates; The distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates is calculated as the coordinate error value corresponding to the first control point.

7. The method according to claim 1, characterized in that After determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix, the method further includes: For each pixel coordinate in the visual image, mapping the pixel coordinate according to the current first mapping matrix and the second mapping matrix to obtain a mapping coordinate corresponding to the pixel coordinate; Obtaining a current calibration image based on pixel values ​​corresponding to mapping coordinates corresponding to pixel coordinates in the visual image in the map image; For a difference image area in the visual image that does not meet a preset matching condition with the current calibration image, the current first control coordinates are obtained based on the newly added first control coordinates in the difference image area, and the current second control coordinates are obtained based on the pixel coordinates corresponding to the newly added first control coordinates in the map image; Return to the step of determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on the current first control coordinates and the current second control coordinates as the current first mapping matrix, until there is no difference image area in the visual image.

8. A mapping relationship determination device, characterized in that: The device comprises: A control coordinate determination module, used to determine the pixel coordinates marked in the visual image of the target scene as the current first control coordinates, and to determine the pixel coordinates marked in the map image of the target scene as the current second control coordinates; wherein the current first control coordinates correspond to the current second control coordinates one by one, and any current first control coordinate and the corresponding second control coordinate represent the same position in the target scene; A linear matrix determination module, used for determining a linear matrix representing a linear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on current first control coordinates and current second control coordinates, as a current first mapping matrix; An intermediate pixel coordinate determination module is used to map the current first control coordinates according to the current first mapping matrix to obtain the intermediate pixel coordinates corresponding to the current first control coordinates; A nonlinear matrix determination module, for determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image as the current second mapping matrix; The mapping relationship determination module is used to obtain a nonlinear mapping relationship between pixel coordinates in the visual image and pixel coordinates in the map image based on a current first mapping matrix and a current second mapping matrix.

9. The device according to claim 8, characterized in that The nonlinear matrix determination module comprises: A nonlinear matrix determination submodule, for determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to a thin plate spline interpolation algorithm, as the current second mapping matrix; The device also includes: A coordinate error value calculation module, used for determining, based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing the mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to the thin plate spline interpolation algorithm, as the current second mapping matrix, and for each current first control coordinate, calculating the coordinate error value corresponding to the first control coordinate based on the intermediate pixel coordinate corresponding to the first control coordinate and the corresponding second control coordinate; An error regularization term coefficient determination module is used to determine the current error regularization term coefficient of the first control coordinate based on the coordinate error value corresponding to the first control coordinate; wherein, when the coordinate error value corresponding to any first control coordinate is less than a preset threshold, the current error regularization term coefficient of the first control coordinate is negatively correlated with the coordinate error value corresponding to the first control coordinate; The nonlinear matrix determination submodule includes: A nonlinear matrix determination unit, for determining, based on the intermediate pixel coordinates and error regularization term coefficients corresponding to the current first control coordinates and the current second control coordinates, a nonlinear matrix representing a nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image according to a thin plate spline interpolation algorithm, as a current second mapping matrix; The nonlinear matrix determination unit is specifically used to determine the current first intermediate matrix based on the intermediate pixel coordinates corresponding to the current first control coordinates according to the thin plate spline interpolation algorithm; wherein the current first intermediate matrix is ​​a symmetric matrix, and each row of the current first intermediate matrix corresponds to each current first control coordinate one by one; the elements in the current first intermediate matrix are calculated based on the following formula: U(r ij ) represents the element located at the i-th row and j-th column in the current first intermediate matrix, (x i ,y i ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the i-th row, (x j ,y j ) represents the middle pixel coordinate corresponding to the first control coordinate corresponding to the jth column; For each row in the current first intermediate matrix, the main diagonal elements in the row are replaced with the current error regularization term coefficients of the corresponding first control coordinates to obtain the current second intermediate matrix; According to the thin plate spline interpolation algorithm, based on the current second intermediate matrix, the intermediate pixel coordinates corresponding to the current first control coordinates, and the current second control coordinates, a nonlinear matrix representing the nonlinear mapping relationship between the intermediate pixel coordinates and the pixel coordinates in the map image is calculated as the current second mapping matrix; The error regularization term coefficient determination module is specifically used to calculate the ratio of the first threshold value to the coordinate error value corresponding to the first control coordinate; The smaller value between the calculated ratio and the second threshold is determined as the current error regularization term coefficient of the first control coordinate; wherein the second threshold is greater than the first threshold, and both the first threshold and the second threshold are negatively correlated with the size of the map image; The coordinate error value calculation module is specifically used to convert the intermediate pixel coordinates corresponding to each current first control coordinate into longitude and latitude coordinates to obtain first longitude and latitude coordinates, and convert the second control coordinates corresponding to the first control coordinate into longitude and latitude coordinates to obtain second longitude and latitude coordinates; calculate the distance between the first longitude and latitude coordinates and the second longitude and latitude coordinates as the coordinate error value corresponding to the first control point; The device also includes: A mapping coordinate determination module, for determining a nonlinear matrix representing a nonlinear mapping relationship between intermediate pixel coordinates and pixel coordinates in the map image based on the intermediate pixel coordinates corresponding to the current first control coordinates and the current second control coordinates as the current second mapping matrix, and for each pixel coordinate in the visual image, mapping the pixel coordinate according to the current first mapping matrix and the second mapping matrix to obtain a mapping coordinate corresponding to the pixel coordinate; A calibration image determination module, used to obtain a current calibration image based on the pixel values ​​corresponding to the mapping coordinates corresponding to the pixel coordinates in the visual image in the map image; A control coordinate updating module is used to obtain the current first control coordinates based on the newly added first control coordinates in the difference image area that does not meet the preset matching conditions with the current calibration image, and obtain the current second control coordinates based on the pixel coordinates corresponding to the newly added first control coordinates in the map image; return to execute the step of determining a linear matrix representing the linear mapping relationship between the pixel coordinates in the visual image and the pixel coordinates in the map image based on the current first control coordinates and the current second control coordinates as the current first mapping matrix, until there is no difference image area in the visual image.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images