Image alignment method and device, equipment and storage medium

Abstract

The invention discloses an image alignment method and device, equipment and a storage medium, and relates to the technical field of computers. According to the method, the initial trajectory can be automatically aligned to the planar graph to generate the target trajectory, and the features of the at least three trajectory points specified by the user are comprehensively considered, so that the accuracy of image alignment is improved. The method comprises the following steps: acquiring an initial track, wherein the initial track comprises a plurality of track points; then displaying a target plane map of the target scene and the initial track; in response to a position change operation of a user on at least three track points in the plurality of track points, displaying a target track, the target track comprising respective target positions of the plurality of track points in the target planar map, the target positions of the plurality of track points in the target planar map being obtained based on an optimal transformation matrix; and storing the target trajectory in response to a confirmation operation of the user on the target trajectory.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on October 9, 2023, with application number 202311303743.1 and application name “A trajectory adjustment method, device and system”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an image alignment method, device, equipment and storage medium. Background Art

[0003] In industrial construction scenarios, data collectors can manually or through mobile robots and other equipment to shoot construction sites during inspections, collect inspection track image data of the inspection sites, and record the construction process and existing problems in the construction sites. In order to facilitate the determination of which parts of the scene the images captured during the inspection correspond to, the data collectors can mark the corresponding position of the image in the plane design after taking the relevant pictures, and record the position as the actual position of the image in the scene, so as to align it with the plane design.

[0004] However, in the above method, the collector needs to mark the position of each picture in the plane design drawing every time he takes it, which causes a lot of workload for the collector, and he cannot quickly determine the corresponding position of each picture in the plane design drawing, so the efficiency is low. Summary of the invention

[0005] The present application provides an image alignment method, device, equipment and storage medium, which solves the technical problem in the related art that the collection personnel need to mark the position of each picture taken in the plane design drawing once, and cannot quickly determine the corresponding position of each picture in the plane design drawing.

[0006] In a first aspect, an image alignment method is provided, the method comprising: first, obtaining an initial trajectory, the initial trajectory comprising a plurality of trajectory points, the plurality of trajectory points being trajectory points at which a user collects inspection data in a target scene; then displaying a target plane map of the target scene and the initial trajectory, the plurality of trajectory points being located at respective initial positions on the target plane map; thereafter, in response to the user's position change operation on at least three of the plurality of trajectory points, displaying a target trajectory, the target trajectory comprising respective target positions of the plurality of trajectory points in the target plane map, the target positions of the plurality of trajectory points in the target plane map being obtained based on an optimized transformation matrix, the The optimized transformation matrix is ​​used to characterize that the target distance reaches a minimum value, and the target distance includes the distance between the target position of each trajectory point of the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map, wherein the target position of the target trajectory point of the at least three trajectory points in the target plane map is obtained based on the initial position of the target trajectory point in the target plane map and the optimized transformation matrix, and the transformed position of the target trajectory point of the at least three trajectory points in the target plane map is determined according to the position change operation; finally, in response to the user's confirmation operation on the target trajectory, the target trajectory is saved.

[0007] It can be understood that the method described in the first aspect can be executed by a computing device, which can be a terminal, an apparatus including a terminal, or a chip in a terminal, or the computing device can also be a network device, an apparatus including a network device, or a chip in a network device. For ease of understanding, the following description will be given by taking the execution of a computing device as an example.

[0008] In the present application, the computing device can first obtain an initial trajectory, which includes multiple trajectory points; after that, the computing device can display the target plane map of the target scene and the initial trajectory, specifically displaying the initial positions of each of the multiple trajectory points in the target plane map. Then, in response to the user's position change operation on at least three of the multiple trajectory points, the computing device can display the target trajectory, which includes the target positions of each of the multiple trajectory points in the target plane map. Finally, in response to the user's confirmation operation on the target trajectory, it is indicated that the target trajectory (specifically the target positions of multiple trajectory points in the target plane map) meets the user's needs, and the computing device can save the target trajectory for subsequent use. In the present application, for at least three trajectory points actively adjusted by the user, the distance between the target position obtained after the optimization transformation matrix and the position specified by the user (i.e., the transformed position obtained after performing the position change operation) is the shortest (or the similarity is high), and at this time, the computing device can obtain the relevant and correct target positions of the multiple trajectory points based on the optimization transformation matrix. That is, the positions of the multiple trajectory points can be automatically adjusted, and the target trajectory can be automatically generated and displayed. The features of at least three trajectory points specified by the user (or actively adjusted) are comprehensively considered to improve the accuracy of image alignment.

[0009] Optionally, the method also includes: determining the derivative of the objective function, which is used to characterize the target distance; performing gradient descent on the objective function based on the derivative of the objective function to obtain a target parameter, which is a parameter that minimizes the result of the objective function; and determining the transformation matrix corresponding to the target parameter as the optimized transformation matrix.

[0010] In the present application, since the optimal transformation matrix is ​​used to characterize the minimum value of the target distance, the objective function is used to characterize the target distance. In this way, the computing device can obtain the parameter that makes the result of the objective function reach the minimum value by performing gradient descent on the objective function based on the derivative of the objective function, which is the target parameter (which can also be understood as the minimum value of the target distance). Thus, the computing device can determine the transformation matrix corresponding to the target parameter as the optimal transformation matrix, and can accurately and effectively determine the optimal transformation matrix. Moreover, since the determination process of the optimal transformation matrix comprehensively considers the positions of at least three trajectory points actively adjusted by the user, it is possible to avoid slight adjustments to some trajectory points that will affect the entire trajectory, thereby improving the convenience of user operation.

[0011] Optionally, the above-mentioned displaying the target trajectory in response to the user's position changing operation on at least three trajectory points among the multiple trajectory points specifically includes: obtaining the transformed positions of the at least three trajectory points in the target plane map in response to the user's position changing operation on the at least three trajectory points; obtaining the optimized transformation matrix based on the transformed positions of the at least three trajectory points in the target plane map; determining the target positions of the multiple trajectory points in the target plane map based on the initial positions of the multiple trajectory points in the target plane map and the optimized transformation matrix; and generating and displaying the target trajectory based on the target positions of the multiple trajectory points in the target plane map.

[0012] In the present application, after the user performs a position change operation on at least three trajectory points, the computing device can obtain the transformed positions of the at least three trajectory points in the target plane map. Since the similarity between the target position obtained based on the optimized transformation matrix and the transformed position specified by the user is high, the computing device can automatically adjust the positions of the multiple trajectory points in the target plane map to the correct target positions based on the initial positions of the multiple trajectory points in the target plane map and the optimized transformation matrix. Thus, the computing device can generate and display the target trajectory based on the correct positions of the multiple trajectory points in the target plane map. It is possible to automatically generate and display a trajectory that is more in line with the user's expectations.

[0013] Optionally, the above-mentioned obtaining of the initial trajectory specifically includes: obtaining inspection data collected by the user in the target scene; generating a target three-dimensional trajectory based on the inspection data collected by the user in the target scene, the target three-dimensional trajectory including the three-dimensional coordinates of the multiple trajectory points in the target scene; performing plane fitting processing on the target three-dimensional trajectory to obtain a target two-dimensional trajectory; and obtaining the initial trajectory based on the target two-dimensional trajectory and the target plane map.

[0014] In the present application, after obtaining the inspection data collected by the user in the target scene, the computing device can generate a target three-dimensional trajectory based on the inspection data collected by the user in the target scene. The computing device can then perform plane fitting processing on the target three-dimensional trajectory to obtain a target two-dimensional trajectory. Finally, the computing device can obtain an initial trajectory based on the target two-dimensional trajectory and the target plane map. The computing device can conveniently and quickly determine the initial trajectory based on the inspection data collected by the user in the target scene, thereby improving the efficiency of image alignment.

[0015] Optionally, the inspection data includes one or a combination of the following data: image data, laser data, and depth data.

[0016] In the present application, one or more of the image data, laser data, and depth data included in the target scene can accurately and effectively characterize the actual situation in the target scene. And because the target plane map is a plane map or a plane design map in the target scene. Based on these data, the computing device can accurately and effectively determine the positions of multiple trajectory points in the target plane map, thereby improving the accuracy of image alignment.

[0017] Optionally, the inspection data includes image data, and the image data collected by the user at the first trajectory point and the image data collected by the user at the second trajectory point have the same object, the first trajectory point is any one of the multiple trajectory points, and the second trajectory point is a trajectory point among the multiple trajectory points adjacent to the first trajectory point.

[0018] In the present application, since the image data collected from adjacent trajectory points in the initial trajectory corresponding to the user contain the same object, it is ensured that the computing device can collect continuous image data, thereby effectively obtaining and displaying a continuous trajectory related to the user.

[0019] Optionally, the above-mentioned saving of the target trajectory specifically includes: obtaining inspection data collected by the user at the multiple trajectory points; and saving the corresponding relationship between the inspection data collected by the user at the multiple trajectory points and the target positions of the multiple trajectory points in the target plane map.

[0020] In the present application, for any of the above-mentioned multiple track points, the computing device can determine that there is a corresponding relationship between the inspection data collected by the user at the track point and the target position of the track point in the target plane map. Thus, the target position of the track point in the target plane map can be determined (or recorded) as the actual position of the track point in the target scene. The accuracy of image alignment is further improved.

[0021] Optionally, the position change operation includes one or a combination of the following operations: a drag operation, a click operation, and a value input operation.

[0022] In the present application, the user can drag, select, and input values ​​on at least three track points to make the at least three track points reach the user-specified position, that is, the transformation position of the at least three track points can be accurately and effectively determined. Thus, the target track can be accurately and effectively displayed and saved, ensuring the effectiveness of image alignment.

[0023] Optionally, the method further includes: generating a target map based on the inspection data collected by the user at the multiple trajectory points and the target positions of the multiple trajectory points in the target plane map, wherein the target map is used to characterize the semantic information of the multiple trajectory points in the target scene.

[0024] In the present application, since there is a corresponding relationship between the target positions of the multiple trajectory points included in the target trajectory in the target plane map and the inspection data collected by the user at the multiple trajectory points, the computing device can accurately and effectively determine the semantic information of each of the multiple trajectory points in the target scene based on the inspection data collected by the user at the multiple trajectory points and the target positions of the multiple trajectory points in the target plane map, thereby generating a target map that can accurately characterize the semantic information of each trajectory point in the target scene, thereby improving the richness of the map.

[0025] Optionally, the method also includes: when there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, generating a VR scene corresponding to the preset position based on the inspection data collected by the user at the first trajectory point, the first trajectory point is any one of the multiple trajectory points, and the preset position is any position included in the target plane map.

[0026] In the present application, when there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, it means that the preset position is the position of the user when the inspection data is collected at the first trajectory point, and specifically, the inspection data collected by the user at the first trajectory point is the inspection data related to the preset position. At this time, the computing device can accurately and effectively generate a VR scene corresponding to the preset position based on the inspection data collected by the user at the first trajectory point, thereby helping to construct a panoramic roaming scene and enhance the user experience.

[0027] In a second aspect, an image alignment device is provided. The device includes an acquisition module, a display module, and a storage module; the acquisition module is used to acquire an initial trajectory, the initial trajectory includes multiple trajectory points, and the multiple trajectory points are trajectory points for the user to collect inspection data in the target scene; the display module is used to display a target plane map of the target scene and the initial trajectory, and the multiple trajectory points are located at their respective initial positions on the target plane map; the display module is also used to display a target trajectory in response to the user's position change operation on at least three of the multiple trajectory points, the target trajectory includes the target positions of the multiple trajectory points in the target plane map, and the target positions of the multiple trajectory points in the target plane map are based on the optimized transformation matrix The optimized transformation matrix is ​​used to characterize that the target distance reaches a minimum value, and the target distance includes the distance between the target position of each trajectory point among the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map, wherein the target position of the target trajectory point among the at least three trajectory points in the target plane map is obtained based on the initial position of the target trajectory point in the target plane map and the optimized transformation matrix, and the transformed position of the target trajectory point among the at least three trajectory points in the target plane map is determined according to the position change operation; the saving module is used to save the target trajectory in response to the user's confirmation operation on the target trajectory.

[0028] In addition, the technical effects of the image alignment device described in the second aspect can refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0029] In a third aspect, an image alignment device is provided. The image alignment device includes: a module for executing the method described in the first aspect, such as a transceiver module and a processing module. For example, the transceiver module is used to execute the transceiver function of the image alignment device, and the processing module is used to execute functions of the image alignment device other than the transceiver function.

[0030] Optionally, the transceiver module may include a sending module and a receiving module, wherein the sending module is used to implement the sending function of the image alignment device described in the second aspect, and the receiving module is used to implement the receiving function of the image alignment device described in the second aspect.

[0031] Optionally, the image alignment device described in the third aspect may further include a storage module, wherein the storage module stores a program or instruction. When the processing module executes the program or instruction, the image alignment device may execute the method described in the first aspect.

[0032] It can be understood that the image alignment device described in the third aspect can be a terminal or a network device, or a chip (system) or other parts or components that can be set in a terminal or a network device, or a device that includes a terminal or a network device, and this application does not limit this.

[0033] In addition, the technical effects of the image alignment device described in the third aspect can refer to the technical effects of the other aspects mentioned above, and will not be repeated here.

[0034] In a fourth aspect, an image alignment device is provided, wherein the image alignment device comprises: a processor, wherein the processor is configured to execute the method described in the first aspect.

[0035] In a possible design solution, the image alignment device described in the fourth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the image alignment device described in the fourth aspect to communicate with other devices.

[0036] In a possible design scheme, the image alignment device described in the fourth aspect may also include a memory. The memory may be integrated with the processor or may be separately provided. The memory may be used to store the computer program and / or data involved in the method described in the first aspect.

[0037] In an embodiment of the present application, the communication device described in the fourth aspect may be the terminal or network device described in the first aspect, or a chip (system) or other parts or components that may be arranged in the terminal or network device, or a device including the terminal or network device.

[0038] In addition, the technical effects of the image alignment device described in the fourth aspect can refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0039] In a fifth aspect, an image alignment device is provided. The communication device includes: a processor, the processor is coupled to a memory, and the processor is used to execute a computer program stored in the memory, so that the image alignment device executes the method described in the first aspect.

[0040] In a possible design solution, the image alignment device described in the fifth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the image alignment device described in the fifth aspect to communicate with other devices.

[0041] In an embodiment of the present application, the image alignment device described in the fifth aspect may be the terminal or network device described in the first aspect, or a chip (system) or other parts or components that may be arranged in the terminal or network device, or a device including the terminal or network device.

[0042] In addition, the technical effects of the image alignment device described in the fifth aspect can refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0043] In a sixth aspect, an image alignment device is provided, comprising: a processor and a memory; the memory is used to store a computer program, and when the processor executes the computer program, the image alignment device executes the method described in the first aspect.

[0044] In a possible design solution, the image alignment device described in the sixth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the image alignment device described in the sixth aspect to communicate with other devices.

[0045] In an embodiment of the present application, the image alignment device described in the sixth aspect may be the terminal or network device described in the first aspect, or a chip (system) or other parts or components that may be arranged in the terminal or network device, or a device including the terminal or network device.

[0046] In addition, the technical effects of the image alignment device described in the sixth aspect can refer to the technical effects of the method described in the first aspect, and will not be repeated here.

[0047] In the seventh aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect.

[0048] In an eighth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect.

[0049] In a ninth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the internal structure of a computing device provided in an embodiment of the present application is shown;

[0051] Figure 2 A schematic diagram of a process of an image alignment method provided in an embodiment of the present application is shown;

[0052] Figure 3 A schematic diagram of a process flow of another image alignment method provided in an embodiment of the present application is shown;

[0053] Figure 4 A schematic diagram showing the process principle of an image alignment method provided by an embodiment of the present application is shown;

[0054] Figure 5 A schematic diagram of the structure of an image alignment device provided in an embodiment of the present application is shown;

[0055] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown;

[0056] Figure 7 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application is shown;

[0057] Figure 8 A schematic diagram of a network architecture of a computing device cluster provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] To facilitate understanding of the solutions provided by the embodiments of the present application, some concepts involved in the present application are explained before introducing the solutions provided by the embodiments of the present application.

[0059] Simultaneous localization and mapping (SLAM) is a technology for autonomous mapping and positioning of robots. Specifically, it refers to the process in which a moving object calculates its own position based on information from sensors while building a map of the environment. Its uses include the positioning of the sensor itself and subsequent path planning, scene understanding, etc. In the embodiment of the present application, the computing device can generate a three-dimensional trajectory of the target based on the SLAM algorithm and the inspection data collected by the user in the target scene.

[0060] Point of interest (POI) map is a geographical location-based information service that can help users find nearby POIs, such as stores, restaurants, etc. These POIs can be presented in the form of markers on the map, and users can click on the markers to obtain more information about the POI, such as address, business hours, etc. In an embodiment of the present application, the computing device can generate (or construct) a POI map based on the inspection data collected by the user at multiple track points and the target positions of the multiple track points in the target plane map.

[0061] At present, aligning image sequences with graphic designs is a common task in the field of computer vision and is of great value in many fields. For example, in industrial construction scenes, data collectors often need to patrol and shoot production and construction sites to record the construction process and existing problems in the construction sites. In order to facilitate the determination of which parts of the scene the images (or inspection data) captured during the inspection process correspond to, the data collector can associate and align them with the graphic design after taking the relevant pictures (or video frames). If the captured images (or inspection data) cannot be associated and aligned with the graphic design, they are just isolated data that cannot be matched with global information and cannot actually play a role.

[0062] Specifically, aligning the image sequence with the graphic design can include the following three steps:

[0063] 1. Data collection: Specifically, data collectors can use ordinary cameras, panoramic cameras, laser scanners, etc. to inspect the site and record the actual situation in the site.

[0064] Second, mark the collected data in the graphic design, that is, mark the corresponding position of the collected data in the scene. Specifically, after taking the relevant pictures, the collector marks the corresponding position of the picture in the graphic design, and records the position as the actual position of the picture in the scene. The second step is performed alternately with the first step, that is, each time data is collected (or a picture is taken), it is marked on the graphic design once.

[0065] 3. Compare and analyze the data marked on the plane design. Specifically, you can compare the inspection data of the same location on different dates, query and analyze the data within a specific range, etc., so that the data can really play a role.

[0066] In summary, in order for the data to play a greater role, it needs to be associated with the scene's floor plan (or global map), but the process of marking the collected data in the global map is very cumbersome and time-consuming. For example, for a panoramic inspection of a construction site, it is necessary to associate a series of panoramic pictures taken with the floor plan and mark the position of each photo, so that the photos of the corresponding location can be found when needed, so that the panoramic images of the same location on different dates can be compared, and finally the construction progress on different dates can be analyzed and the source of the problem can be traced back. However, the collector needs to mark the location on the floor plan every time he takes a picture. The workload of the collector is large, and it is impossible to quickly determine the corresponding position of each picture in the floor plan.

[0067] The above solution has the following difficulties:

[0068] First, due to the large amount of image data, the workload of manually marking each image in the graphic design drawing is very large.

[0069] Second, the automatic alignment between images and graphic designs is cross-modal and it is difficult to guarantee the success rate of alignment. Errors often occur and manual adjustments are still required by users.

[0070] 3. During manual adjustment, you can usually only drag the start point and / or the end point. When the distance between the end point and the start point is relatively close, even a small movement may cause the track to be over-adjusted and not meet expectations, resulting in a poor user experience.

[0071] In one implementation, the user can manually drag the start and end points in the track, and the inspection data of the remaining positions will be automatically synchronized to correspond, which can reduce the workload of the data collector. This implementation works well for simple straight inspection tracks, but for complex tracks, especially when the distance between the start and end points is relatively close, a slight adjustment of the start and / or end points will have a significant impact on the entire track. This makes it difficult for users to adjust the track to the correct position, and it takes a lot of time to adjust.

[0072] In another possible implementation, after calculating the trajectory pose, the trajectory can be automatically mapped to the plane design based on the inspection start position specified by the user. This method can reduce the user's workload, but in difficult scenarios, there are often cases where the trajectory is automatically mapped incorrectly, and the user still needs to manually adjust it.

[0073] Based on this, an embodiment of the present application provides an image alignment method. For at least three trajectory points actively adjusted by the user, the distance between the target position obtained based on the optimized transformation matrix and the position specified by the user (that is, the transformed position obtained after performing the position change operation) is the shortest (or the similarity is high). At this time, the computing device can obtain the correct target position related to the multiple trajectory points based on the optimized transformation matrix. That is, the positions of the multiple trajectory points can be automatically adjusted, and the target trajectory can be automatically generated and displayed. The characteristics of at least three trajectory points specified by the user (or actively adjusted) are comprehensively considered, and the entire trajectory is adjusted to the best global level, thereby improving the accuracy of image alignment.

[0074] The image alignment method, device, equipment and storage medium provided in the embodiments of the present application are applied to image and plane map alignment scenarios (including but not limited to constructing virtual reality (VR) scenarios, map building scenarios and industrial building scenarios). After the computing device obtains the initial trajectory, the positions of multiple trajectory points included in the initial trajectory can be updated based on the image alignment method provided in the embodiments of the present application to obtain and save the target trajectory.

[0075] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0076] It is understood that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0077] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0078] In one example, the computing device that executes the image alignment method provided in an embodiment of the present application can be a terminal, which can also be called user equipment (UE), access terminal, subscriber unit (subscriber unit), user station, mobile station (MS), mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device. The terminal in the embodiments of the present application can be a mobile phone, a cellular phone, a smart phone, a tablet computer, a wireless data card, a personal digital assistant (PDA), a wireless modem, a handset, a laptop computer, a machine type communication (MTC) terminal, a computer with a wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a smart home device (for example, a refrigerator, a television, an air conditioner, an electric meter, etc.), an intelligent robot, a robotic arm, a workshop equipment, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a vehicle-mounted terminal, a roadside unit with a terminal function, and a wireless terminal in a smart city. unit, RSU), etc., flying equipment (e.g., intelligent robots, hot air balloons, drones, airplanes), etc. The terminal of the present application may also be an on-board module, on-board module, on-board component, on-board chip or on-board unit built into the vehicle as one or more components or units. The terminal may also be other devices with terminal functions, for example, the terminal may also be a device that functions as a terminal in device to device (D2D) communication.

[0079] The embodiments of the present application do not limit the device form of the terminal. The device for realizing the function of the terminal can be a terminal; it can also be a device that can support the terminal to realize the function, such as a chip system. The device can be installed in the terminal, or used in combination with the terminal. In the embodiments of the present application, the chip system can be composed of chips, or it can include chips and other discrete devices.

[0080] In another example, the computing device may be a server, which may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (content delivery network, CDN), as well as big data and artificial intelligence platforms.

[0081] In one implementation of the embodiment of the present application, Figure 1 As shown, the computing device may include a data acquisition module 101 , a trajectory calculation module 102 , a front-end interaction module 103 , a trajectory adjustment module 104 and a data storage and analysis module 105 .

[0082] Specifically, the data collection module 101 is used to obtain inspection data collected by the user at multiple track points.

[0083] The trajectory calculation module 102 is used to generate a three-dimensional trajectory of the target based on the inspection data collected by the user in the target scene, and then obtain an initial trajectory.

[0084] The front-end interaction module 103 is used for triggering, by the user, a position change operation on at least three of the multiple track points, so as to realize interaction between the user and the computing device.

[0085] The trajectory adjustment module 104 is used to determine the optimal transformation matrix, and obtain the target position of each of the plurality of trajectory points in the target plane map based on the optimal transformation matrix, thereby generating a target trajectory.

[0086] The data storage and analysis module 105 is used to save the target trajectory, and to construct a target map and related VR scenes based on the target trajectory and the inspection data collected by the user at multiple trajectory points.

[0087] The image alignment method provided in the embodiment of the present application can be applied to the above computing device to implement the image alignment process. Figure 2 As shown, the image alignment method provided in the embodiment of the present application may include S101-S104.

[0088] S101: A computing device obtains an initial trajectory.

[0089] The initial trajectory includes a plurality of trajectory points, which are trajectory points at which the user collects inspection data in the target scene.

[0090] It should be understood that the user can collect inspection data at the multiple track points, and one track point corresponds to one inspection data.

[0091] In an optional implementation, the inspection data includes one or a combination of the following data: image data, laser data, and depth data.

[0092] In another optional implementation, the above-mentioned inspection data includes image data, and the image data collected by the user at the first trajectory point and the image data collected by the user at the second trajectory point have the same object, the first trajectory point is any one of the above-mentioned multiple trajectory points, and the second trajectory point is a trajectory point adjacent to the first trajectory point among the multiple trajectory points.

[0093] Optionally, the above-mentioned same object may be the same object or the same area.

[0094] In the embodiment of the present application, since the image data collected by adjacent trajectory points in the initial trajectory corresponding to the user contain the same object, it can be ensured that the computing device can collect continuous image data, so that the continuous trajectory related to the user can be effectively obtained and displayed.

[0095] In an implementation manner of the embodiment of the present application, the computing device acquiring the initial trajectory may specifically include S1011-S1014.

[0096] S1011. The computing device obtains the inspection data collected by the user in the target scene.

[0097] Specifically, the user can collect inspection data at each of the multiple track points.

[0098] Optionally, users can use sensors such as cameras and lidar to collect inspection data.

[0099] S1012: The computing device generates a three-dimensional trajectory of the target based on the inspection data collected by the user in the target scene.

[0100] The target three-dimensional trajectory includes the three-dimensional coordinates of the above-mentioned multiple trajectory points in the target scene.

[0101] Specifically, the computing device can determine the three-dimensional trajectory of the target based on the inspection data collected by the user in the target scene and positioning algorithms such as color map (colmap) and SLAM.

[0102] In the embodiment of the present application, the target three-dimensional trajectory can also be understood as the three-dimensional trajectory of the user in the target scene. The three-dimensional trajectory of the user in the target scene is specifically the three-dimensional trajectory of the user when collecting inspection data in the target scene.

[0103] S1013: The computing device performs plane fitting processing on the target three-dimensional trajectory to obtain the target two-dimensional trajectory.

[0104] Specifically, the computing device can fit the plane where the target three-dimensional trajectory is located based on a plane fitting algorithm. Then the target three-dimensional trajectory is rotated to a position parallel to the plane, and finally only the motion trajectory in the direction parallel to the plane is retained, that is, the target two-dimensional trajectory can be obtained. The target two-dimensional trajectory can also be understood as the two-dimensional trajectory of the user in the target scene.

[0105] S1014: The computing device obtains an initial trajectory based on the target two-dimensional trajectory and the target plane map.

[0106] It should be understood that the target plane map is a plane map (or plane design map) related to (or corresponding to) the target scene.

[0107] Specifically, the computing device may place (or draw) the target two-dimensional trajectory to any position in the target plane map to obtain the initial trajectory.

[0108] In an embodiment of the present application, after obtaining the inspection data collected by the user in the target scene, the computing device can generate a target three-dimensional trajectory based on the inspection data collected by the user in the target scene. The computing device can then perform plane fitting processing on the target three-dimensional trajectory to obtain a target two-dimensional trajectory. Finally, the computing device can obtain an initial trajectory based on the target two-dimensional trajectory and the target plane map. The computing device can conveniently and quickly determine the initial trajectory based on the inspection data collected by the user in the target scene, thereby improving the efficiency of image alignment.

[0109] S102: The computing device displays a target plane map and an initial trajectory of the target scene.

[0110] The multiple track points are located at their respective initial positions on the target plane map. In an embodiment provided by the present application, the initial position can be roughly aligned with the target plane map through a simple initialization algorithm, or can be centered in the center of the plane map. The present application does not limit the initialization positions of the multiple track points.

[0111] S103: In response to the user's operation of changing the positions of at least three of the multiple track points, the computing device displays the target track.

[0112] Wherein, the target trajectory includes the target positions of each of the multiple trajectory points in the above-mentioned target plane map, the target positions of the multiple trajectory points in the target plane map are obtained based on the optimized transformation matrix, the optimized transformation matrix is ​​used to characterize the minimum value of the target distance, the target distance includes the distance between the target position of each trajectory point of the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map, wherein the target position of the target trajectory point of the at least three trajectory points in the target plane map is obtained based on the initial position of the target trajectory point in the target plane map and the optimized transformation matrix, and the transformed position of the target trajectory point of the at least three trajectory points in the target plane map is determined according to the position change operation.

[0113] It should be understood that after each of the at least three trajectory points performs the position change operation, the computing device can determine the transformed position of each trajectory point in the target plane map. Then, after the computing device determines the optimized transformation matrix based on the transformed position of each trajectory point in the target plane map, the target position of the multiple trajectory points in the target plane map can be obtained based on the initial positions of the multiple trajectory points in the target plane map and the optimized transformation matrix. In this way, the target trajectory can be obtained, so that the computing device can display the target trajectory.

[0114] It can be understood that since the optimized transformation matrix is ​​used to characterize the minimum value of the target distance, and the target distance includes the distance between the target position of each of the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map. That is, for the at least three trajectory points, the distance between the target position obtained based on the optimized transformation matrix and the transformed position specified by the user (that is, the position obtained after performing the position change operation) is the shortest, and it can also be understood that the similarity between the target position obtained based on the optimized transformation matrix and the transformed position specified by the user is high. At this time, the computing device can obtain the correct target position related to multiple trajectory points based on the optimized transformation matrix. That is, the positions of multiple trajectory points can be automatically adjusted, and the target trajectory can be automatically generated and displayed.

[0115] Specifically, the computing device can use the Levenberg-Marquardt algorithm to minimize the target distance based on an optimization method, and solve the two-dimensional similarity transformation from the initial position of each trajectory point in at least three trajectory points to the transformed position of each trajectory point. After reaching the optimized convergence condition, the two-dimensional similarity transformation (that is, the optimized transformation matrix) is applied to other trajectory points to obtain the target trajectory.

[0116] It is understood that, in an optional implementation, the trajectory (including the initial trajectory and the target trajectory) in the embodiment of the present application is used to characterize discrete points. In an optional manner, the discrete points characterized by the trajectory (including the initial trajectory and the target trajectory) in the embodiment of the present application can be connected in sequence and presented in the form of a line.

[0117] In an optional implementation, the position change operation includes one or a combination of the following operations: a drag operation, a point selection operation, and a numerical input operation.

[0118] Specifically, for a certain track point (e.g., a target track point) among the at least three track points mentioned above, the user can perform a drag operation on the target track point, specifically dragging the target track point from its initial position in the target plane map to the transformed position of the target track point in the target plane map. The user can also select the at least three track points and click on the corresponding position, so that each of the at least three track points reaches its respective transformed position in the target plane map. The user can also input the identifier of each of the at least three track points and the relevant position information of each track point (which can be embodied in the form of a numerical value), so that each track point can reach its transformed position in the target plane map.

[0119] In the embodiment of the present application, the user can drag, select, and input a value on at least three track points to make the at least three track points reach the position specified by the user, that is, the transformation position of the at least three track points can be accurately and effectively determined. Therefore, the target track can be accurately and effectively displayed and saved, ensuring the effectiveness of image alignment.

[0120] In an implementation of the embodiment of the present application, in response to the user's operation of changing the position of at least three of the multiple track points, the computing device displays the target track, which may specifically include S1031-S1034.

[0121] S1031. In response to the user's operation of changing the positions of at least three track points, the computing device obtains the transformed positions of the at least three track points in the target plane map.

[0122] It should be understood that the transformed positions of the at least three track points in the target plane map are positions obtained (or determined) after the user performs a position change operation on the at least three track points (specifically, the initial positions of the at least three track points in the target plane map). The transformed positions of the at least three track points in the target plane map can also be understood as the positions specified by the user for the at least three track points, and the at least three track points are track points specified by the user.

[0123] S1032: The computing device obtains an optimized transformation matrix based on the transformed positions of at least three trajectory points in the target plane map.

[0124] In combination with the description of the above embodiments, it should be understood that the computing device can determine the optimal transformation matrix based on the target distance, which target distance includes the distance between the target position of each of the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map.

[0125] S1033: The computing device determines target positions of the multiple trajectory points in the target plane map based on the initial positions of the multiple trajectory points in the target plane map and the optimized transformation matrix.

[0126] In an optional implementation, the computing device may determine the product of the initial positions of the multiple trajectory points in the target plane map and the optimized transformation matrix as the target positions of the multiple trajectory points in the target plane map.

[0127] S1034: The computing device generates and displays a target trajectory based on the target positions of the multiple trajectory points in the target plane map.

[0128] Optionally, after determining the target positions of the multiple trajectory points in the target plane map, the computing device may connect the target positions of the multiple trajectory points in the target plane map in a sequence of the multiple trajectory points, so as to obtain the target trajectory.

[0129] In an embodiment of the present application, after the user performs a position change operation on at least three trajectory points, the computing device can obtain the transformed positions of the at least three trajectory points in the target plane map. Since the similarity between the target position obtained based on the optimized transformation matrix and the transformed position specified by the user is high, the computing device can automatically adjust the positions of the multiple trajectory points in the target plane map to the correct target positions based on the initial positions of the multiple trajectory points in the target plane map and the optimized transformation matrix. Thus, the computing device can generate and display the target trajectory based on the correct positions of the multiple trajectory points in the target plane map. It is possible to automatically generate and display a trajectory that is more in line with the user's expectations.

[0130] S104: In response to the user's confirmation operation on the target trajectory, the computing device saves the target trajectory.

[0131] It is understandable that after the computing device displays the target trajectory, the user can identify and judge the target trajectory. When the target trajectory (specifically, the target position of multiple trajectory points in the target plane map) meets the user's needs, the user can perform a confirmation operation on the target trajectory. Finally, the computing device can save the target trajectory for subsequent use.

[0132] In an embodiment of the present application, the computing device may first obtain an initial trajectory, which includes a plurality of trajectory points; thereafter, the computing device may display a target plane map of the target scene and the initial trajectory, specifically displaying the initial positions of each of the plurality of trajectory points in the target plane map. Then, in response to the user's position change operation on at least three of the plurality of trajectory points, the computing device may display a target trajectory, which includes the target positions of each of the plurality of trajectory points in the target plane map. Finally, in response to the user's confirmation operation on the target trajectory, it is indicated that the target trajectory (specifically the target positions of the plurality of trajectory points in the target plane map) meets the user's needs, and the computing device may save the target trajectory for subsequent use. In an embodiment of the present application, for at least three trajectory points actively adjusted by the user, the distance between the target position obtained after the optimization transformation matrix and the position specified by the user (i.e., the transformed position obtained after the position change operation is performed) is the shortest (or the similarity is high), and at this time, the computing device may obtain the relevant and correct target positions of the plurality of trajectory points based on the optimization transformation matrix. That is, the positions of the plurality of trajectory points can be automatically adjusted, and the target trajectory can be automatically generated and displayed. The features of at least three trajectory points specified by the user (or actively adjusted) are comprehensively considered to improve the accuracy of image alignment.

[0133] In an optional implementation, the computing device saves the target trajectory, which may specifically include S1041 - S1042 .

[0134] S1041. The computing device obtains inspection data collected by the user at multiple track points.

[0135] In combination with the description of the above embodiment, it should be understood that the computing device can collect inspection data of the user in the target scene at each of the multiple track points.

[0136] S1042: The computing device saves the inspection data collected by the user at multiple track points and the corresponding relationship between the target positions of the multiple track points in the target plane map.

[0137] It is understandable that, for any of the above multiple track points, the computing device can determine that there is a corresponding relationship between the inspection data collected by the user at the track point and the target position of the track point in the target plane map. Thus, the target position of the track point in the target plane map can be determined (or recorded) as the actual position of the track point in the target scene. The accuracy of image alignment is further improved.

[0138] In the embodiment of the present application, since the target positions of the above-mentioned multiple trajectory points in the target plane map are obtained based on the optimized transformation matrix, the optimized transformation matrix can be pre-stored in the computing device before use, or can be determined (or calculated) by the computing device. Figure 2 ,like Figure 3 The image alignment method provided in the embodiment of the present application may further include S105-S107.

[0139] S105. The computing device determines the derivative of the objective function.

[0140] The objective function is used to characterize the above target distance.

[0141] S106. The computing device performs gradient descent on the objective function based on the derivative of the objective function to obtain the target parameter.

[0142] The target parameter is a parameter that minimizes the result of the target function.

[0143] S107. The computing device determines the transformation matrix corresponding to the target parameter as the optimal transformation matrix.

[0144] It should be understood that since the optimal transformation matrix is ​​used to characterize the minimum value of the target distance, the objective function is used to characterize the target distance. In this way, the computing device can obtain the parameter that makes the result of the objective function reach the minimum value by performing gradient descent on the objective function based on the derivative of the objective function, which is the target parameter (which can also be understood as the minimum value of the target distance). Therefore, the computing device can determine the transformation matrix corresponding to the target parameter as the optimal transformation matrix, and can accurately and effectively determine the optimal transformation matrix. In addition, since the determination process of the optimal transformation matrix comprehensively considers the positions of at least three trajectory points actively adjusted by the user, it is possible to avoid slight adjustments to some trajectory points that will affect the entire trajectory, thereby improving the convenience of user operation.

[0145] In one implementation of an embodiment of the present application, the computing device can also generate a target map based on the inspection data collected by the user at the above-mentioned multiple trajectory points and the target positions of the multiple trajectory points in the target plane map. The target map is used to characterize the semantic information of the multiple trajectory points in the target scene.

[0146] Specifically, for a track point, the computing device can identify the semantic information (or type) of the track point in the target scene based on the inspection data collected by the track point. Afterwards, the computing device can record (or mark) the semantic information of the track point in the target scene to the target position of the track point in the target plane map.

[0147] Optionally, the semantic information (or type) of a trajectory point in the target scene may be accessories, clothing, firefighting points, special equipment, and the like.

[0148] Exemplarily, the target map may be a POI map.

[0149] In an embodiment of the present application, since there is a corresponding relationship between the target positions of the multiple trajectory points included in the target trajectory in the target plane map and the inspection data collected by the user at the multiple trajectory points, the computing device can accurately and effectively determine the semantic information of each of the multiple trajectory points in the target scene based on the inspection data collected by the user at the multiple trajectory points and the target positions of the multiple trajectory points in the target plane map, thereby generating a target map that can accurately characterize the semantic information of each trajectory point in the target scene, thereby improving the richness of the map.

[0150] In another implementation of the embodiment of the present application, when there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, the computing device can generate a VR scene corresponding to the preset position based on the inspection data collected by the user at the first trajectory point.

[0151] The first trajectory point is any one of the above-mentioned multiple trajectory points, and the preset position is any position included in the target plane map.

[0152] It should be understood that when there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, it means that the preset position is the position when the user collected the inspection data at the first trajectory point, and specifically, the inspection data collected by the user at the first trajectory point is the inspection data related to the preset position. At this time, the computing device can accurately and effectively generate a VR scene corresponding to the preset position based on the inspection data collected by the user at the first trajectory point, which helps to construct a panoramic roaming scene and enhance the user experience.

[0153] The following is an overall example to illustrate the image alignment method provided in the embodiment of the present application.

[0154] like Figure 4 As shown, the image alignment method provided in the embodiment of the present application may include three processes, namely, a data acquisition and trajectory calculation process, a trajectory position adjustment process, and a data storage and processing process.

[0155] The data collection and trajectory calculation process specifically includes data collection, three-dimensional trajectory calculation and two-dimensional trajectory calculation.

[0156] Specifically, data collection is to obtain the inspection data collected by the user in the target scene, three-dimensional trajectory calculation is to calculate the three-dimensional trajectory of the target, and two-dimensional trajectory calculation is to calculate the two-dimensional trajectory of the target.

[0157] The trajectory position adjustment process includes displaying the two-dimensional trajectory, interactively inputting the trajectory points to be adjusted, calculating the trajectory adjustment parameters, and generating the adjusted two-dimensional trajectory.

[0158] Specifically, displaying the two-dimensional trajectory is to display the initial trajectory. The interactive input of the trajectory point to be adjusted is specifically to perform a position change operation on at least three of the multiple trajectory points. The trajectory adjustment parameter calculation is specifically to determine the optimal transformation matrix, and determine the target positions of the multiple trajectory points in the target plane map based on the optimal transformation matrix. Generating the adjusted two-dimensional trajectory is to obtain the target trajectory.

[0159] The data storage and processing process includes the associated storage of inspection data and trajectory data as well as advanced analysis functions.

[0160] Specifically, the inspection data and the trajectory data are associated and saved to save the target trajectory, specifically to save the inspection data collected by the user at multiple trajectory points and the corresponding relationship between the target positions of each of the multiple trajectory points in the target plane map. Advanced analysis functions include the above-mentioned generation of target maps and generation of VR scenes.

[0161] It is understandable that in order to implement the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0162] Combined with the above Figures 1 to 4 , describes in detail the image alignment method provided by this embodiment, and will now be combined with Figure 5 , describing the image alignment device provided according to this embodiment.

[0163] The present application also provides an image alignment device, such as Figure 5 As shown, the image alignment device 50 includes: an acquisition module 501 , a display module 502 and a storage module 503 .

[0164] The acquisition module 501 is used to acquire an initial trajectory, which includes a plurality of trajectory points, and the plurality of trajectory points are trajectory points at which a user collects inspection data in a target scene.

[0165] The display module 502 is used to display the target plane map of the target scene and the initial trajectory, and the multiple trajectory points are located at their respective initial positions on the target plane map.

[0166] The display module 502 is further used to display a target trajectory in response to the user's position change operation on at least three of the multiple trajectory points, wherein the target trajectory includes the target positions of each of the multiple trajectory points in the target plane map, the target positions of the multiple trajectory points in the target plane map are obtained based on an optimized transformation matrix, the optimized transformation matrix is ​​used to characterize that the target distance reaches a minimum value, the target distance includes the distance between the target position of each of the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map, wherein the target position of the target trajectory point among the at least three trajectory points in the target plane map is obtained based on the initial position of the target trajectory point in the target plane map and the optimized transformation matrix, and the transformed position of the target trajectory point among the at least three trajectory points in the target plane map is determined according to the position change operation.

[0167] The saving module 503 is used to save the target trajectory in response to the user's confirmation operation on the target trajectory.

[0168] Optionally, the image alignment device 50 further includes a determination module 504 and a processing module 505 .

[0169] The determination module 504 is used to determine the derivative of the target function, where the target function is used to characterize the target distance.

[0170] The processing module 505 is used to perform gradient descent on the objective function based on the derivative of the objective function to obtain a target parameter, where the target parameter is a parameter that minimizes the result of the objective function.

[0171] The determination module 504 is further configured to determine the transformation matrix corresponding to the target parameter as the optimized transformation matrix.

[0172] Optionally, the processing module 505 is used to obtain the transformed positions of the at least three track points in the target plane map in response to the user's position change operation on the at least three track points.

[0173] The processing module 505 is further configured to obtain the optimized transformation matrix based on the transformed positions of the at least three trajectory points in the target plane map.

[0174] The determination module 504 is used to determine the target positions of the multiple track points in the target plane map based on the initial positions of the multiple track points in the target plane map and the optimized transformation matrix.

[0175] The processing module 505 is further configured to generate the target trajectory based on the target positions of the multiple trajectory points in the target plane map.

[0176] The display module 502 is specifically used to display the generated target trajectory.

[0177] Optionally, the acquisition module 501 is specifically used to acquire the inspection data collected by the user in the target scene.

[0178] The processing module 505 is used to generate a target three-dimensional trajectory based on the inspection data collected by the user in the target scene, and the target three-dimensional trajectory includes the three-dimensional coordinates of the multiple trajectory points in the target scene.

[0179] The processing module 505 is further used to perform plane fitting processing on the target three-dimensional trajectory to obtain the target two-dimensional trajectory.

[0180] The processing module 505 is further configured to obtain the initial trajectory based on the target two-dimensional trajectory and the target plane map.

[0181] Optionally, the inspection data includes one or a combination of the following data: image data, laser data, and depth data.

[0182] Optionally, the inspection data includes image data, and the image data collected by the user at the first trajectory point and the image data collected by the user at the second trajectory point have the same object, the first trajectory point is any one of the multiple trajectory points, and the second trajectory point is a trajectory point among the multiple trajectory points adjacent to the first trajectory point.

[0183] Optionally, the acquisition module 501 is further used to acquire the inspection data collected by the user at the multiple track points.

[0184] The saving module 503 is specifically used to save the inspection data collected by the user at the multiple track points and the corresponding relationship between the target positions of the multiple track points in the target plane map.

[0185] Optionally, the position change operation includes one or a combination of the following operations: a drag operation, a click operation, and a value input operation.

[0186] Optionally, the processing module 505 is used to generate a target map based on the inspection data collected by the user at the multiple track points and the target positions of the multiple track points in the target plane map, wherein the target map is used to represent the semantic information of the multiple track points in the target scene.

[0187] Optionally, the processing module 505 is used to generate a VR scene corresponding to the preset position based on the inspection data collected by the user at the first trajectory point when there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, the first trajectory point is any one of the multiple trajectory points, and the preset position is any position included in the target plane map.

[0188] Among them, the acquisition module 501, the display module 502, the storage module 503, the determination module 504 and the processing module 505 can all be implemented by software, or can be implemented by hardware. Exemplarily, the display module 502 is taken as an example to introduce the implementation of the display module 502. Similarly, the implementation of the acquisition module 501, the storage module 503, the determination module 504 and the processing module 505 can refer to the implementation of the display module 502.

[0189] As an example of a software functional unit, the display module 502 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the display module 502 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0190] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0191] As an example of a hardware functional unit, the display module 502 may include at least one computing device, such as a server, etc. Alternatively, the display module 502 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0192] The multiple computing devices included in the display module 502 can be distributed in the same region or in different regions. The multiple computing devices included in the display module 502 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the display module 502 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0193] It should be noted that, in other embodiments, the display module 502 can be used to execute any step in the image alignment method, the acquisition module 501 can be used to execute any step in the image alignment method, the storage module 503 can be used to execute any step in the image alignment method, the determination module 504 can be used to execute any step in the image alignment method, and the processing module 505 can also be used to execute any step in the image alignment method. The steps that the acquisition module 501, the display module 502, the storage module 503, the determination module 504 and the processing module 505 are responsible for implementing can be specified as needed, and the acquisition module 501, the display module 502, the storage module 503, the determination module 504 and the processing module 505 respectively implement different steps in the image alignment method to realize the full functions of the image alignment device.

[0194] The present application also provides a computing device 600. Figure 6 As shown, computing device 600 includes: bus 602, processor 604, memory 606 and communication interface 608. Processor 604, memory 606 and communication interface 608 communicate through bus 602. Computing device 600 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 600.

[0195] The bus 602 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus 604 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 604 may include a path for transmitting information between various components of the computing device 600 (eg, the memory 606, the processor 604, and the communication interface 608).

[0196] The processor 604 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0197] The memory 606 may include a volatile memory, such as a random access memory (RAM). The processor 604 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0198] The memory 606 stores executable program codes, and the processor 604 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module, display module, storage module, determination module, and processing module, thereby implementing the image alignment method. That is, the memory 606 stores instructions for executing the image alignment method.

[0199] The communication interface 603 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or a communication network.

[0200] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0201] like Figure 7 As shown, the computing device cluster includes at least one computing device 600. The memory 606 in one or more computing devices 600 in the computing device cluster may store the same instructions for executing the image alignment method.

[0202] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the image alignment method. In other words, the combination of one or more computing devices 600 may jointly execute instructions for executing the image alignment method.

[0203] It should be noted that the memory 606 in different computing devices 600 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the image alignment device. That is, the instructions stored in the memory 606 in different computing devices 600 can implement the functions of one or more modules among the acquisition module, display module, storage module, determination module and processing module.

[0204] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 8 A possible implementation is shown. Figure 8 As shown, two computing devices 600A and 600B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 606 in the computing device 600A stores instructions for executing the functions of the acquisition module, the display module, and the storage module. At the same time, the memory 606 in the computing device 600B stores instructions for executing the functions of the determination module and the processing module.

[0205] Figure 8 The connection mode between the computing device clusters shown may be considered to be that the image method provided by the present application requires a large number of determination processes and processing processes, so it is considered that the functions implemented by the determination module and the processing module are handed over to the computing device 600B for execution.

[0206] It should be understood that Figure 8The functions of the computing device 600A shown in FIG. 6A may also be completed by multiple computing devices 600. Similarly, the functions of the computing device 600B may also be completed by multiple computing devices 600.

[0207] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 7 and Figure 8 The connection mode of the computing device cluster is different in that the memory 606 in one or more computing devices 600 in the computing device cluster may store the same instructions for executing the image alignment method.

[0208] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the image alignment method. In other words, the combination of one or more computing devices 600 may jointly execute instructions for executing the image alignment method.

[0209] The embodiment of the present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device performs the image alignment method.

[0210] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to perform the image alignment method.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image alignment method, characterized in that: The method comprises: Acquire an initial trajectory, where the initial trajectory includes a plurality of trajectory points, where the plurality of trajectory points are trajectory points at which a user collects inspection data in a target scene; Displaying a target plane map of the target scene and the initial trajectory, wherein the plurality of trajectory points are located at respective initial positions on the target plane map; In response to the user's position change operation on at least three of the multiple track points, displaying a target track, the target track includes the target positions of each of the multiple track points in the target plane map, the target positions of the multiple track points in the target plane map are obtained based on an optimized transformation matrix, the optimized transformation matrix is ​​used to characterize that the target distance reaches a minimum value, the target distance includes the distance between the target position of each of the at least three track points in the target plane map and the transformed position of each track point in the target plane map, wherein the target position of the target track point in the at least three track points in the target plane map is obtained based on the initial position of the target track point in the target plane map and the optimized transformation matrix, and the transformed position of the target track point in the at least three track points in the target plane map is determined according to the position change operation; In response to the user's confirmation operation on the target trajectory, the target trajectory is saved.

2. The image alignment method according to claim 1, characterized in that: The method further comprises: determining a derivative of an objective function, the objective function being used to characterize the target distance; Performing gradient descent on the objective function based on the derivative of the objective function to obtain an objective parameter, wherein the objective parameter is a parameter that minimizes the result of the objective function; The transformation matrix corresponding to the target parameter is determined as the optimized transformation matrix.

3. The image alignment method according to claim 1 or 2, characterized in that: The step of displaying a target track in response to the user's operation of changing the position of at least three track points among the plurality of track points comprises: In response to the user's operation of changing the positions of the at least three track points, obtaining the transformed positions of the at least three track points in the target plane map; Obtaining the optimized transformation matrix based on the transformed positions of the at least three trajectory points in the target plane map; Determining target positions of the plurality of trajectory points in the target plane map based on initial positions of the plurality of trajectory points in the target plane map and the optimized transformation matrix; The target trajectory is generated and displayed based on the target positions of the multiple trajectory points in the target plane map.

4. The image alignment method according to any one of claims 1 to 3, characterized in that: The obtaining of the initial trajectory comprises: Acquire inspection data collected by the user in the target scene; Generate a target three-dimensional trajectory based on the inspection data collected by the user in the target scene, wherein the target three-dimensional trajectory includes the three-dimensional coordinates of the plurality of trajectory points in the target scene; Performing plane fitting processing on the target three-dimensional trajectory to obtain a target two-dimensional trajectory; The initial trajectory is obtained based on the target two-dimensional trajectory and the target plane map.

5. The image alignment method according to any one of claims 1 to 4, characterized in that: The inspection data includes one or a combination of the following data: image data, laser data, and depth data.

6. The image alignment method according to any one of claims 1 to 5, characterized in that: The inspection data includes image data, the image data collected by the user at the first track point and the image data collected by the user at the second track point have the same object, the first track point is any one of the multiple track points, and the second track point is a track point adjacent to the first track point among the multiple track points.

7. The image alignment method according to any one of claims 1 to 6, characterized in that: The step of saving the target trajectory comprises: Acquire inspection data collected by the user at the multiple track points; The corresponding relationship between the inspection data collected by the user at the multiple track points and the target positions of the multiple track points in the target plane map is saved.

8. The image alignment method according to any one of claims 1 to 7, characterized in that: The position change operation includes one or a combination of the following operations: a drag operation, a click operation, and a value input operation.

9. The image alignment method according to any one of claims 1 to 8, characterized in that: The method further comprises: Based on the inspection data collected by the user at the multiple track points and the target positions of the multiple track points in the target plane map, a target map is generated, where the target map is used to represent the semantic information of the multiple track points in the target scene.

10. The image alignment method according to any one of claims 1 to 9, characterized in that: The method further comprises: When there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, a virtual reality VR scene corresponding to the preset position is generated based on the inspection data collected by the user at the first trajectory point, the first trajectory point is any one of the multiple trajectory points, and the preset position is any position included in the target plane map.

11. An image alignment device, characterized in that: The device comprises: an acquisition module, a display module and a storage module; The acquisition module is used to acquire an initial trajectory, wherein the initial trajectory includes a plurality of trajectory points, and the plurality of trajectory points are trajectory points at which the user collects inspection data in the target scene; The display module is used to display the target plane map of the target scene and the initial trajectory, and the multiple trajectory points are located at respective initial positions on the target plane map; The display module is further used to display a target trajectory in response to the user's position change operation on at least three of the multiple trajectory points, wherein the target trajectory includes the target positions of each of the multiple trajectory points in the target plane map, the target positions of the multiple trajectory points in the target plane map are obtained based on an optimized transformation matrix, the optimized transformation matrix is ​​used to characterize that the target distance reaches a minimum value, the target distance includes the distance between the target position of each of the at least three trajectory points in the target plane map and the transformed position of each trajectory point in the target plane map, wherein the target position of the target trajectory point in the at least three trajectory points in the target plane map is obtained based on the initial position of the target trajectory point in the target plane map and the optimized transformation matrix, and the transformed position of the target trajectory point in the at least three trajectory points in the target plane map is determined according to the position change operation; The saving module is used to save the target trajectory in response to the user's confirmation operation on the target trajectory.

12. The image alignment device according to claim 11, characterized in that: The device also includes a determination module and a processing module; The determination module is used to determine a derivative of an objective function, wherein the objective function is used to characterize the target distance; The processing module is used to perform gradient descent on the objective function based on the derivative of the objective function to obtain a target parameter, where the target parameter is a parameter that makes the result of the objective function reach a minimum value; The determination module is further used to determine the transformation matrix corresponding to the target parameter as the optimized transformation matrix.

13. The image alignment device according to claim 11 or 12, characterized in that: The device also includes a determination module and a processing module; The processing module is used to obtain the transformed positions of the at least three track points in the target plane map in response to the user's position change operation on the at least three track points; The processing module is further used to obtain the optimized transformation matrix based on the transformed positions of the at least three trajectory points in the target plane map; The determination module is used to determine the target positions of the multiple track points in the target plane map based on the initial positions of the multiple track points in the target plane map and the optimized transformation matrix; The processing module is further used to generate the target trajectory based on the target positions of the multiple trajectory points in the target plane map; The display module is specifically used to display the generated target trajectory.

14. The image alignment device according to any one of claims 11 to 13, characterized in that: The device also includes a processing module; The acquisition module is specifically used to acquire the inspection data collected by the user in the target scene; The processing module is used to generate a target three-dimensional trajectory based on the inspection data collected by the user in the target scene, and the target three-dimensional trajectory includes the three-dimensional coordinates of the multiple trajectory points in the target scene; The processing module is further used to perform plane fitting processing on the target three-dimensional trajectory to obtain a target two-dimensional trajectory; The processing module is further used to obtain the initial trajectory based on the target two-dimensional trajectory and the target plane map.

15. The image alignment device according to any one of claims 11 to 14, characterized in that: The inspection data includes one or a combination of the following data: image data, laser data, and depth data.

16. The image alignment device according to any one of claims 11 to 15, characterized in that: The inspection data includes image data, the image data collected by the user at the first track point and the image data collected by the user at the second track point have the same object, the first track point is any one of the multiple track points, and the second track point is a track point adjacent to the first track point among the multiple track points.

17. The image alignment device according to any one of claims 11 to 16, characterized in that: The acquisition module is further used to acquire the inspection data collected by the user at the multiple track points; The saving module is specifically used to save the inspection data collected by the user at the multiple track points and the corresponding relationship between the target positions of the multiple track points in the target plane map.

18. The image alignment device according to any one of claims 11 to 17, characterized in that: The position change operation includes one or a combination of the following operations: a drag operation, a click operation, and a value input operation.

19. The image alignment device according to any one of claims 11 to 18, characterized in that: The device also includes a processing module; The processing module is used to generate a target map based on the inspection data collected by the user at the multiple track points and the target positions of the multiple track points in the target plane map, and the target map is used to represent the semantic information of the multiple track points in the target scene.

20. The image alignment device according to any one of claims 11 to 19, characterized in that: The device also includes a processing module; The processing module is used to generate a virtual reality VR scene corresponding to the preset position based on the inspection data collected by the user at the first trajectory point when there is a corresponding relationship between the inspection data collected by the user at the first trajectory point and the preset position, the first trajectory point is any one of the multiple trajectory points, and the preset position is any position included in the target plane map.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the image alignment method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the image alignment method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the image alignment method according to any one of claims 1 to 10.

Images